Evaluation
ts-agents now includes a deterministic internal benchmark that compares the raw tool surface against the structured discovery, skills, and workflow layers. No LLM is called, and the command sequences (including deliberate failures) are fixed; it is to make the repo’s own agent-facing contract measurable and easy to rerun.
What the benchmark covers
The harness in ts_agents.evals.refactor_benchmark compares four assist levels:
plain_model— no runnable repo contractplain_tools— direct tool calls with no discovery or workflow guidancestructured_discovery—tool search/tool showbefore executionskills_workflows—skills showplusworkflow run
It runs three representative tasks:
- inspect an unknown univariate series
- compare forecasting baselines on a deterministic series
- perform labeled-stream window-size selection and evaluation
Metrics:
- task success rate
- parse / schema failure rate
- invalid tool calls
- artifact completeness
- latency
- retries / recovery
Latest checked-in snapshot
Regenerated for 0.4.1 on 2026-10-25, after the tool-input and result-contract changes; the previous snapshot dated from 2026-04-01.
Rerun command:
uv run python -m ts_agents.evals.refactor_benchmark \
--output-dir benchmarks/results/latestChecked-in outputs:
benchmarks/results/latest/results.jsonbenchmarks/results/latest/summary.md
Results from the current snapshot:
| Assist level | Success rate | Parse failure rate | Invalid tool calls | Avg artifact completeness | Avg duration (ms) | Retries | Recovery rate |
|---|---|---|---|---|---|---|---|
plain_model |
0.00 | 1.00 | 0 | 0.00 | 0.0 | 0 | 0.00 |
plain_tools |
0.67 | 0.00 | 1 | 0.00 | 291.4 | 1 | 0.33 |
structured_discovery |
1.00 | 0.00 | 0 | 0.00 | 63.8 | 0 | 0.00 |
skills_workflows |
1.00 | 0.00 | 0 | 1.00 | 134.6 | 0 | 0.00 |
How to read it
plain_modelfails every scenario because it produces no machine-runnable contract.plain_toolscan recover sometimes, but it still guesses wrong once and produces no artifact bundles.structured_discoveryreaches full task success, but it still leaves artifact completeness at0.00because raw tool calls do not write the workflow bundle.skills_workflowsreaches full task success and full artifact completeness, demonstrating the artifact guarantees of these scripted commands. It does not measure whether an LLM chooses better commands.
What it does not cover
The scenarios exercise inspection, forecasting, and activity recognition only. Anomaly detection, change-point segmentation, and spectral analysis are covered by known-answer unit tests (synthetic signals with planted periods, shifts, and events), not by this benchmark, and no scenario measures an LLM agent’s answer quality. Agent-in-the-loop evaluations with labeled tasks per family are on the roadmap.
External Context
The checked-in internal benchmark is contract evidence, not an accuracy leaderboard. External forecasting claims should be compared against benchmark surfaces such as GIFT-Eval before being presented as model-performance evidence. The repo now tracks that context in:
benchmarks/external/summary.md
TimeCopilot is treated as a comparator and interoperability target because it has a broad foundation-model forecasting surface. That is intentionally different from ts-agents, whose durable surface is CLI contracts, workflow manifests, artifact bundles, sandboxes, and reusable skills. The scoped foundation-smoke loop runs pinned Darts zero-shot checkpoints (default chronos2_small; --models also accepts chronos2, timesfm2p5 and patchtst_fm, and only the selected checkpoints download) through the same artifact contract. It is a smoke check, not a model hub or an accuracy leaderboard: M4 may overlap FM pretraining data. A weekly CI job (darts-fm-smoke) runs it unmocked, so Darts API drift is caught in CI. The former foundation-chronos-smoke name remains a deprecated alias.
Why this matters
This benchmark closes the loop on the refactor:
- workflows make artifact production deterministic
skills showexposes the actionable policy layer
If you change the workflow contract, skill metadata, or tool discovery surface, rerun the benchmark and update the checked-in snapshot.