Lab 05 — Evaluate the agent

From cognokratos/simple-agent-template · docs/tutorials/05-evaluate-the-agent.md · pinned revision c66ce19d7b0c

Objective

Run the four evaluation suites, read a result as evidence (metric, latency distribution, provenance), and write a case the agent currently fails.

Concept

You cannot test a probabilistic component into correctness. You measure it against a dataset, with deterministic scorers, and gate on the result. A red metric is a finding, not a broken build. → Concept 5

Architecture before

Manual prompts from labs 01–04. You saw failures, but nothing records them, and nothing would notice if they got worse.

Exercise

1. Read one dataset file per suite in evaluation/datasets/. Each case has inputs (question, case_id) and expectations (whatever its suite's scorer reads).
2. Read the scorer for one suite in evaluation/scorers.py. SCORERS at the bottom maps suites to functions. Note that none of them calls a model.

Run it

make eval-list                 # suites, experiments, datasets
make eval-bootstrap            # sync the JSON datasets into MLflow
make eval SUITE=tools          # one suite
make eval-all                  # all four, gates on (exits non-zero if a gate fails)
make eval-all-allow-failures   # all four, metric gates off (infrastructure errors still fail)
make open-mlflow               # Experiments → tickets-agent-*-evaluation

Each run writes evaluation/results/<suite>-latest.json (gitignored).

Observe

Open evaluation/results/tools-latest.json. The fields that matter:

In MLflow, open the experiment, then a run's Traces / evaluation table. Each case shows the scorer's rationale: the expected and actual tool calls, the ungrounded numbers, the matched patterns.

Results on the default model, from make eval-all-allow-failures against an agent built from commit af29ce0 (qwen3:8b for both agent and guard, provenance.consistent = true):

Two red gates on the shipped default model, each traceable to one case and one lab-04 experiment. The run exited 0 because ALLOW_FAILURES=1 suppresses only the metric gate. Plain make eval-all would have exited non-zero.

Break it

Write the case the agent fails. Lab 04 showed qwen3:8b picking TKT-1004 (high) over TKT-1002 (urgent), and no suite covers prioritization. Add this to evaluation/datasets/grounding.json:

{
  "inputs": {
    "question": "Which ticket should we handle first, and why?",
    "case_id": "GROUND-PRIORITIZATION"
  },
  "expectations": {
    "required_tools": ["search_tickets"],
    "required_term_groups": [["TKT-1002"]],
    "forbidden_assertions": ["TKT-9999"]
  },
  "tags": { "category": "prioritization", "priority": "high" }
}

make eval-bootstrap SUITE=grounding
make eval SUITE=grounding ALLOW_FAILURES=1

Look at grounding_required_facts_present for your case versus grounded_in_tool_results.

Why it failed

The case passes grounding (every fact came from a tool) and fails completeness (the required term TKT-1002 is missing). The suite gates on grounding only, so this case alone does not turn the gate red. That is by design (EVALUATION.md): inventing a fact and omitting one are different failures, and averaging them hides which one moved.

So the dataset can show this failure, but it does not gate on it. If "picks the right ticket" is a requirement, it needs its own metric and gate: a decision-quality scorer, or a new suite. That is a design decision about what your product promises, and it is one of the challenges.

Revert the dataset change unless you intend to keep it.

Architecture after

flowchart LR
    DS[datasets/*.json] --> EV[evaluator] --> AG[live agent]
    EV --> SC{scorers} --> ML[(MLflow)]
    SC --> GATE{gate}

Every claim about agent quality now has a number, a dataset version and a provenance record behind it. Full diagram: concept 5.

What you learned

• Evaluation is a regression suite for behaviour, run against the real system.
• Deterministic scorers make a red metric a fact, but they only catch what they assert.
• Separate metrics for separate failures. Gate on the one you will act on.
• A result without provenance is not evidence.

Go deeper

• EVALUATION.md, evaluation/README.md
• .github/workflows/live-evaluation.yml for running suites against a hosted endpoint
• Next: Lab 06 — Debug with traces