Skip to main content

Evaluation

ts-agents now includes a deterministic internal benchmark that compares the raw tool surface against the structured discovery, skills, and workflow layers. No LLM is called, and the command sequences (including deliberate failures) are fixed; it is to make the repo’s own agent-facing contract measurable and easy to rerun.

What the benchmark covers

The harness in ts_agents.evals.refactor_benchmark compares four assist levels:

  1. plain_model — no runnable repo contract
  2. plain_tools — direct tool calls with no discovery or workflow guidance
  3. structured_discovery — tool search / tool show before execution
  4. skills_workflows — skills show plus workflow run

It runs three representative tasks:

  • inspect an unknown univariate series
  • compare forecasting baselines on a deterministic series
  • perform labeled-stream window-size selection and evaluation

Metrics:

  • task success rate
  • parse / schema failure rate
  • invalid tool calls
  • artifact completeness
  • latency
  • retries / recovery

Latest checked-in snapshot

Regenerated for 0.4.1 on 2026-10-25, after the tool-input and result-contract changes; the previous snapshot dated from 2026-04-01.

Rerun command:

uv run python -m ts_agents.evals.refactor_benchmark \
  --output-dir benchmarks/results/latest

Checked-in outputs:

  • benchmarks/results/latest/results.json
  • benchmarks/results/latest/summary.md

Results from the current snapshot:

Assist level Success rate Parse failure rate Invalid tool calls Avg artifact completeness Avg duration (ms) Retries Recovery rate
plain_model 0.00 1.00 0 0.00 0.0 0 0.00
plain_tools 0.67 0.00 1 0.00 291.4 1 0.33
structured_discovery 1.00 0.00 0 0.00 63.8 0 0.00
skills_workflows 1.00 0.00 0 1.00 134.6 0 0.00

How to read it

  • plain_model fails every scenario because it produces no machine-runnable contract.
  • plain_tools can recover sometimes, but it still guesses wrong once and produces no artifact bundles.
  • structured_discovery reaches full task success, but it still leaves artifact completeness at 0.00 because raw tool calls do not write the workflow bundle.
  • skills_workflows reaches full task success and full artifact completeness, demonstrating the artifact guarantees of these scripted commands. It does not measure whether an LLM chooses better commands.

What it does not cover

The scenarios exercise inspection, forecasting, and activity recognition only. Anomaly detection, change-point segmentation, and spectral analysis are covered by known-answer unit tests (synthetic signals with planted periods, shifts, and events), not by this benchmark, and no scenario measures an LLM agent’s answer quality. Agent-in-the-loop evaluations with labeled tasks per family are on the roadmap.

External Context

The checked-in internal benchmark is contract evidence, not an accuracy leaderboard. External forecasting claims should be compared against benchmark surfaces such as GIFT-Eval before being presented as model-performance evidence. The repo now tracks that context in:

  • benchmarks/external/summary.md

TimeCopilot is treated as a comparator and interoperability target because it has a broad foundation-model forecasting surface. That is intentionally different from ts-agents, whose durable surface is CLI contracts, workflow manifests, artifact bundles, sandboxes, and reusable skills. The scoped foundation-smoke loop runs pinned Darts zero-shot checkpoints (default chronos2_small; --models also accepts chronos2, timesfm2p5 and patchtst_fm, and only the selected checkpoints download) through the same artifact contract. It is a smoke check, not a model hub or an accuracy leaderboard: M4 may overlap FM pretraining data. A weekly CI job (darts-fm-smoke) runs it unmocked, so Darts API drift is caught in CI. The former foundation-chronos-smoke name remains a deprecated alias.

Why this matters

This benchmark closes the loop on the refactor:

  • workflows make artifact production deterministic
  • skills show exposes the actionable policy layer

If you change the workflow contract, skill metadata, or tool discovery surface, rerun the benchmark and update the checked-in snapshot.