feat(evals): add agent evaluation suite

[sudoKrishna]

Summary

Adds a deterministic, CI-runnable evaluation layer for the agent harness, plus
opt-in live tooling. Scripted scenarios drive the real code — the OpenAI-compatible
streaming tool loop and the DAGExecutor — and score tool selection, planning,
retrieval, recovery, and context assembly. Live runs compare models and support
record/replay. No provider key is required for CI.

This is the consolidated branch: it folds the incremental stack into one PR.

What's included

Deterministic (CI, no key)

• agent-tool-use/ — 13 tool-loop scenarios (reliability + adversarial) and 4
  executor scenarios (Start→Agent, variable resolution, block retry, model fallback)
• agent-context/ — provider-request assembly with conversation memory
• replay.ts — record live transcripts once, replay through the real loop;
  replay.test.ts covers chunk round-trip and fixture I/O
• judge.ts — LLM-as-judge scoring with a weighted rubric; judge.test.ts covers
  parsing, weighting, clamping, and malformed responses
• report.ts — JSON + Markdown reports, including the model-comparison matrix

Opt-in live (EVAL_LIVE=1, never in CI)

• live.ts — real model transport (any OpenAI-compatible provider)
• model comparison across EVAL_MODELS with pass rates, latency, tokens
• judge.live.test.ts — a grounded answer outscores an invented one

How to run

cd apps/sim
bun run test:evals            # tool-use + executor + judge + report + replay (32 tests)
bun run test:evals:context    # context assembly

Live (needs a key):

DEEPSEEK_API_KEY=... bun run test:evals:live
EVAL_MODELS=deepseek:deepseek-chat,deepseek:deepseek-reasoner DEEPSEEK_API_KEY=... bun run test:evals:compare
DEEPSEEK_API_KEY=... bun run test:evals:judge

The scripted suite is collected by the normal bun run test, so a regression
fails CI without the dedicated commands.

Real results from live runs

• Tool use on DeepSeek: 100% (33/33) across 11 scenarios, including
  no-tool-needed, near-duplicate tools, empty results, and a four-tool chain.
• Model comparison: deepseek-chat 100% / deepseek-reasoner 89%, reasoner
  ~27% more tokens — a counterintuitive result the suite surfaced.
• Every live failure so far was a test-design bug (case-sensitive phrasing,
  an over-specific id, a missing input a tool required), not a model failure.
  The adversarial cases are what exposed them.

Test plan

• bun run test:evals → 30/30 (1 replay suite skips with no fixtures)
• bun run test:evals:context → 2/2
• Live runs and a key-free model-comparison report validated
• Negative checks: broke the executor-retry, model-fallback, and
  context-isolation expectations and confirmed each failed, then reverted
• bun run check:test-patterns passes
• Eval modules type-check against the real loop/executor signatures
• Full bun run type-check — run in CI

Follow-up

• Wire the judge into scenarios where a rubric is more honest than a substring.
• Subagent/orchestration evals (child workflow with a mocked definition loader).
• Store per-commit reports and chart pass rate/cost over time.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 29, 2026 9:51am UTC

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adds evaluation suite for the agent tool-use loop.

The evaluation suite has no identified runtime blocker, but its import paths must satisfy the repository requirement before merging.

Findings

1.  Tool feedback goes unchecked ▶
2.  Tool arguments are not checked ▶
3.  Relative imports violate app requirement ▶

Summary

The PR adds eight deterministic scenarios that drive the production OpenAI-compatible streaming tool loop, score outcomes, and write JSON and Markdown reports.

• The suite exercises dispatch and error accounting without a provider key.
• Its scripted model does not inspect tool feedback, and scoring does not check dispatched arguments, limiting the regressions it can detect.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Scenario script] --> B[Scripted model turns]
  B --> C[Production streaming tool loop]
  C --> D[Stub tool results]
  D --> C
  C --> E[Scoring]
  E --> F[JSON and Markdown reports]

Loading

_{Reviews (1) · Last reviewed commit: "feat(evals): add agent tool-use evaluati..."}

[vercel]

@sudoKrishna is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

[sudoKrishna]

Addressed all three findings in f619479:

• the scripted model now reads and asserts tool feedback, so a dropped result fails the case;
• a new tool-arguments check compares each executed call's arguments to the expected ones;
• eval imports are absolute now.