Visual overview
Bedrock provides model and RAG evaluation capabilities; S3 can hold evaluation datasets and outputs for reproducible comparison.
Technical reference
Metric formulas are stable concepts, but available Bedrock evaluators, model support, and service features can change. Always define the business meaning of an error first.
Of the items predicted positive, the fraction that were truly positive; useful when false positives are costly.
Of the truly positive items, the fraction detected; useful when missing positives is costly.
Balances the two when a single classification score is useful, but still hides class- and threshold-specific tradeoffs.
Useful for certain text tasks, but lexical overlap does not by itself prove factuality or usefulness.
Compares generated and reference text in an embedding-like representation rather than only exact n-gram overlap.
Scales qualitative scoring but must be calibrated because the judge model can be inconsistent or biased.
Service limits and capabilities can change. Values shown here reflect the current AWS documentation; use the linked official sources below as the source of truth.
Metrics must match the task and the cost of error
Evaluation starts with an explicit success definition and a representative held-out dataset. For classification, a confusion matrix separates true positives, false positives, true negatives, and false negatives. Precision asks how many predicted positives were actually positive; recall asks how many actual positives were found. F1 combines precision and recall. Accuracy can be useful for balanced problems but can hide failure on a rare, important class.
Regression uses numerical error measures such as MAE or RMSE, while ranking, forecasting, and clustering require different metrics. No metric should be chosen merely because it is familiar. A fraud model, medical triage model, recommender, and demand forecast have different failure costs. Thresholds also matter: the same probability model can produce different precision/recall tradeoffs depending on the decision threshold chosen by the application.
Generative output needs multidimensional evaluation
Foundation-model outputs can be valid in many forms, so exact-match accuracy is often insufficient. Reference-based text metrics include ROUGE, which emphasizes overlap useful in summarization contexts, BLEU for generated-versus-reference text comparisons such as translation, and BERTScore, which uses contextual representations to compare semantic similarity. Each is only a proxy and can disagree with human usefulness or factuality.
Human evaluation can assess correctness, helpfulness, tone, safety, and domain nuance that automated metrics miss. LLM-as-a-judge uses an evaluator model to score generated outputs against defined criteria and can scale evaluation, but the judge is itself a model with biases and failure modes. Amazon Bedrock model evaluation supports automated/model-based and human-oriented evaluation workflows depending on the selected approach and current feature support.
Evaluate the complete RAG or agent system, not only its model
A RAG application has at least two quality stages. Retrieval should surface relevant evidence; generation should correctly answer using that evidence. A model may appear to hallucinate because retrieval failed, while excellent retrieval can still be undermined by poor generation. Bedrock RAG evaluation supports computed metrics for retrieval and generation so teams can compare knowledge bases or external RAG sources using a defined prompt dataset.
Agents add further measures: task completion, correct tool selection, parameter accuracy, number of steps, latency, side-effect correctness, policy violations, and cost. Business metrics such as user satisfaction, escalation rate, productivity, conversion, or cost per successful interaction decide whether an AI system creates value. Offline benchmark improvement is useful evidence, but it is not a substitute for production outcomes and safety monitoring.
Key takeaways
- 01
Precision, recall, F1, and accuracy answer different classification questions; class imbalance and error cost determine which matters.
- 02
Generative metrics such as ROUGE, BLEU, and BERTScore are proxies and should be paired with task-specific evidence.
- 03
Human evaluation captures domain nuance; LLM-as-a-judge scales scoring but introduces evaluator-model limitations.
- 04
RAG evaluation should separate retrieval quality from generated-answer quality.
- 05
Agent and GenAI evaluation should include end-to-end task success, safety, latency, cost, and business outcomes.
Official AWS sources
Use these primary AWS resources for the source material behind this article and for deeper reference.