Having the agent running is just the first step.Author: elune
Article compiled, source: ME News
Having the agent running is just the first step.
The real challenge is determining whether it is stable, accurate, or has quietly degraded due to a single prompt or model update.
These 10 evaluation methods are worth knowing for every AI engineer.
1. Golden Set
Prepare a set of fixed and frozen test cases.
After each modification to the prompts, model, tools, or workflow, rerun this set of cases to determine whether the system has improved or quietly failed in certain scenarios.
It is the most fundamental baseline in the Agent evaluation system.
Recommended tool: OpenAI Evals
Can be used to build a reproducible benchmark suite and compare the performance of different models or system versions.
2. LLM Judge | LLM as Judge
Use another large language model to evaluate open-ended responses based on pre-defined scoring criteria.
This method is especially effective when tasks do not have a single correct answer and cannot be judged as right or wrong through string matching or fixed outputs.
For example, a judge model can evaluate whether the response is accurate, complete, relevant, and follows the user’s requirements.
Recommended tool: OpenEvals
Provide ready-to-use evaluators for LLM applications to quickly set up automated review workflows.
3. Multi-dimensional Scoring | Rubric Scoring
Don't just give the Agent a vague "quality score".
Should be evaluated separately:
- Accuracy
- Integrity
- Expression style
- Security
- Response speed
- Invocation cost
A composite score may mask underlying issues.
For example, a decrease in the overall score may not be due to incorrect answers, but rather a sudden increase in tool invocation costs; an increase in the overall score may also come at the cost of reduced security.
Recommended tool: DeepEval
Supports creating custom metrics and scoring different quality dimensions independently.
4. Trajectory Eval
Do not only evaluate the final answer provided by the Agent, but also assess its entire process in completing the task.
Including:
- Have you selected the right tool?
- Are the tools called in a logical order?
- Is the same invalid action being performed repeatedly?
- Are any necessary steps missing?
- Did you correctly adjust your decision based on the tool's results?
The agent may eventually arrive at the correct answer, but the intermediate steps may be inefficient, fragile, or even risky.
Recommended tool: AgentEvals
You can review the agent’s actions, decisions, and tool calls throughout the full execution trace.
5. Tool Unit Tests
Write separate tests for each tool used by the Agent.
Use fixed inputs to verify fixed outputs, without involving the model.
This allows you to break down the issue:
Is the issue with the agent’s reasoning, or with the underlying tools, interfaces, or MCP Server?
It only makes sense to evaluate whether the Agent correctly invoked the tool after ensuring the tool itself is reliable.
Recommended tool: MCP Inspector
Can be used to check and test the MCP Server, tool parameters, and return results.
6. Regression Suite
Save past real execution cases and re-run them after each update to prompts, models, or toolsets.
Then compare the results of the new and old versions to check:
- Did the originally correct task fail?
- Has the output format changed?
- Is there an increase in tool calls?
- Have latency and costs increased?
- Do certain edge cases degenerate?
A better average performance in the new version does not mean it hasn't broken existing capabilities.
Recommended tool: Promptfoo
Support running repeatable evaluation suites, catching regression issues, and integrating check processes into CI.
7. A/B Testing in Production
Randomly distribute real user traffic between two different versions to compare their performance in a live environment.
Can be tested:
- Two sets of prompts
- Two models
- Two Agent workflows
- Different tool combinations
- Different response strategies
A version with higher offline scores does not necessarily lead to higher user success rates.
What truly matters are the actual results, such as task completion rate, user adoption rate, conversion rate, human takeover rate, and issue resolution rate.
Recommended tool: GrowthBook
Provide feature toggles, controlled experiments, and product analytics capabilities.
8. Human Review
Periodic samples of live transaction records are reviewed and scored by human evaluators.
Manual review can identify issues missed by automated evaluations and can also be used to calibrate LLM judges.
Need to pay special attention to:
- Do model scores align with human judgments?
- Are the evaluation criteria clear enough?
- Does the judge model favor lengthy responses?
- Automatically assess whether any critical errors have been overlooked.
Automated evaluation cannot fully replace human judgment.
Recommended tool: Argilla
Help the team collect human feedback, review model outputs, and compile the results into a high-quality dataset.
9. Shadow Run
Run the candidate version on live traffic without displaying its output to users.
The production environment still uses the old version, while the new version runs only in the background for comparing their performance.
This method is suitable for high-risk updates, such as:
- Replace the core model
- Rewrite the system prompt.
- Integrate new external tools
- Modify Agent decision logic
- Expand tool permissions
Shadow running helps teams identify issues in real traffic before official release, while avoiding direct impact on users.
Recommended tool: Langfuse
Track production runs, compare candidate versions, and monitor evaluation results.
10. Red Teaming
Attack your own system before the attacker does.
The testing scope includes:
- Jailbreak attack
- Prompt injection
- Sensitive data leak
- Privilege escalation
- Tool abuse
- Malicious files or web content
- Unintended external action
Red team testing is especially important for Agents that can invoke databases, send emails, modify files, execute code, or access internal systems.
Recommended tool: Garak
Can scan for security vulnerabilities and unsafe behaviors in LLM systems.
Offline evaluation tells you that the system functions properly in a test environment.
The online evaluation tells you: the system will continue to function properly after going live.
You may not need to set up all 10 evaluation mechanisms at once.
A more practical approach is:
Review the most recent Agent failure, then prioritize deploying the two evaluation methods that could have identified the issue earlier.
Establishing a gold test set first, then supplementing with regression tests or manual sampling, often prevents a large number of basic incidents.
Worth saving.