Visual overview
S3 stores prompt/golden datasets and reports, Bedrock runs managed model evaluation, and CloudWatch connects candidate quality to production behavior.
Technical reference
Evaluation turns probabilistic behavior into comparable release evidence. Keep datasets, rubrics, model versions, and thresholds stable enough for a meaningful comparison.
Include normal tasks, hard cases, policy cases, and previous production failures with expected evidence/rubrics.
Report separate dimensions and category floors rather than relying only on one aggregate score.
Calibrate automated judgments against human-reviewed examples and inspect evaluator bias/disagreement.
Measure evidence recall/rank/relevance separately from whether the answer uses that evidence correctly.
A correct final sentence does not excuse unsafe tools, invalid arguments, excessive steps, or failed recovery.
Block changes that regress critical categories even when the overall average improves.
Service limits and capabilities can change. Values shown here reflect the current AWS documentation; use the linked official sources below as the source of truth.
Evaluation starts with a decision, not a metric list
Define what decision the evaluation should support: choose a foundation model, approve a prompt change, validate a new retrieval configuration, compare an agent toolset, or detect production regression. Build a dataset representing important tasks, difficult edge cases, safety cases, and known historical failures. Where an objectively correct answer exists, include a reference; where multiple good answers exist, define a rubric instead of forcing one exact string.
Separate quality dimensions. Relevance asks whether the answer addresses the request; factuality/correctness asks whether claims are accurate; consistency asks whether behavior remains stable across equivalent conditions; fluency measures presentation; safety/policy measures allowed behavior. A single aggregate score can hide a severe regression in a critical category, so report by category and define release-blocking floors.
LLM-as-a-judge scales semantic scoring but is still a model
Amazon Bedrock model evaluation can use a judge model to score a generator model's responses and produce explanations. AWS supports built-in and custom metrics for supported evaluation jobs, with prompt datasets stored in S3 and reports written to an output location. This makes large semantic comparison practical when exact-match metrics are insufficient.
A judge model can have bias, position effects, verbosity preference, model-family preference, or sensitivity to rubric wording. Calibrate judge results against human-reviewed examples, keep the rubric stable during comparison, randomize ordering when evaluating pairwise outputs where possible, and track disagreement. For consequential or ambiguous tasks, human evaluation remains an important source of evidence rather than a failure to automate.
Evaluate RAG and agents at the component that can fail
RAG evaluation should distinguish retrieval quality from generation quality. Measure whether expected evidence appears among retrieved candidates, its rank/relevance, latency, and whether the final response is grounded in that evidence. A perfect generator cannot compensate for absent evidence, so a failure labeled only 'wrong answer' loses the diagnostic value of the evaluation.
Agent evaluation adds task completion, correct tool selection, argument validity, number of steps, unsafe/unnecessary actions, tool-error recovery, and final result correctness. Trace-based evaluation is especially valuable because an agent can reach a plausible answer through a dangerous or inefficient path. Where current Bedrock agent-evaluation capabilities apply, use managed evaluation alongside application-specific traces and business outcomes.
Move from offline scores to controlled production evidence
Offline evaluation provides repeatability; production comparison exposes real traffic and integration effects. A/B tests compare alternatives across users or sessions, while canary deployments expose a smaller share of traffic to a candidate before wider rollout. Define assignment, sample size, safety stops, and success metrics before looking at results to avoid choosing a winner after the fact.
Store evaluation artifacts with the release they justified. Continuous evaluation should re-run golden and adversarial sets after model, prompt, retrieval, guardrail, or tool changes. Add newly discovered production failures to the regression suite so incidents improve future release gates.
Key takeaways
- 01
An evaluation suite should map to a release or selection decision and represent critical categories, edge cases, and historical failures.
- 02
LLM-as-a-judge scales semantic scoring but needs calibration against human-reviewed examples and a stable rubric.
- 03
Evaluate RAG retrieval separately from answer generation and evaluate agent trajectories separately from final prose.
- 04
Use per-category floors so a strong average cannot hide a critical safety or correctness regression.
- 05
Tie evaluation reports to release manifests and continuously grow the regression set from production failures.
Official AWS sources
Use these primary AWS resources for the source material behind this article and for deeper reference.