Motorway and AWS Pioneer Advanced Evaluation Pipeline for Production-Ready AI Agents

In a significant leap forward for artificial intelligence deployment in high-stakes commercial environments, Motorway, a leading UK-based online car marketplace, has collaborated with the AWS Prototyping and AI Customer Engineering (PACE) team to develop and implement a groundbreaking evaluation pipeline for its AI-powered dealer stock search agent. This innovative system dramatically transforms how car dealers locate vehicles, replacing arduous hours of manual filtering with intuitive natural language queries. More critically, it establishes a robust framework for ensuring the reliability and accuracy of AI agents operating with real financial implications, reducing incorrect results from 1 in 8 queries to an impressive 1 in 50, and cutting issue detection time from hours to mere minutes.

The Evolving Landscape of Automotive Retail and AI Integration

Motorway stands at the forefront of digital transformation within the automotive sector, orchestrating a dynamic daily auction that connects up to 8,000 dealers with a vast inventory of up to 2,500 vehicles. The sheer volume and velocity of these transactions underscore the need for highly efficient and accurate tools. Traditionally, dealers navigated complex vehicle listings through laborious manual processes, often involving extensive filtering of CSV files and rigid database queries. This time-consuming approach not only hampered productivity but also limited a dealer’s ability to quickly capitalize on market opportunities or respond to specific customer demands.

The promise of conversational AI offered a compelling alternative: a system that could understand and process natural language, allowing dealers to simply "talk" to an agent. Imagine a dealer asking, "Find me diesel SUVs under 25k near my dealership," or "something sporty and automatic for a family." This vision promised to unlock unprecedented efficiency and a vastly improved user experience. However, translating this promise into a reliable, production-ready solution presented a formidable challenge. The core dilemma lay in ensuring that an AI agent, while delivering confident-sounding responses, genuinely worked reliably when significant financial transactions were at stake. The automotive marketplace, estimated globally at trillions of dollars, demands precision, and any misstep by an AI could lead to financial losses, eroded trust, and operational bottlenecks.

The Unique Imperative for Agent Evaluation

The evaluation of Large Language Models (LLMs) typically focuses on the quality of text generation—coherence, factual accuracy, and relevance of responses. However, AI agent evaluation operates on a fundamentally different plane. While an LLM might be the "engine" of an AI agent, the agent itself is the "entire car," and its evaluation must assess how the whole system performs under diverse, real-world conditions, akin to how a car drives in traffic, rain, or with a full load of passengers.

Traditional LLM metrics fall short when assessing the complex, multi-step workflows of an AI agent. They cannot determine if the Motorway agent correctly identified the appropriate search tool for a query like "Grade 1 Suzuki models" or if it accurately passed filter parameters to a vector database like LanceDB. Crucially, they fail to ascertain whether a dealer refining previous search results receives a contextually accurate response. The stakes are particularly high for agents with around 1,500 concurrent users during peak hours; a single tool selection error or a semantic search misinterpretation directly impacts user trust and, by extension, Motorway’s business.

The critical dimensions for agent evaluation extend far beyond mere text quality:

• Task Completion: Agents execute multi-step workflows where partial completion can be common and problematic.
• Tool Use Correctness: Incorrect tool selection or faulty parameters can entirely derail complex workflows.
• Reasoning Coherence: Flawed underlying reasoning leads to unpredictable failures as operational conditions change.
• Reliability and Consistency: The inherent non-determinism of LLMs means identical inputs can yield different outputs, which is unacceptable for a reliable service.
• Safety and Compliance: Autonomous agents can initiate real-world actions, necessitating strict safety protocols and adherence to regulations.
• Cost and Efficiency: An agent requiring an excessive number of API calls per task may prove economically unviable.

A precise query such as "Volkswagen Golf 7-12 years old" might work flawlessly, but a colloquial variant like "I’m looking for an older VW" could fail without proper evaluation of the semantic search layer, highlighting the need for robust testing across varied linguistic inputs.

A Collaborative Solution: The End-to-End Evaluation Pipeline

Motorway and AWS PACE developed an end-to-end evaluation pipeline that integrates the Strands Agents SDK with Amazon Bedrock AgentCore, a fully managed service designed for deploying and operating AI agents at scale. This pipeline represents a comprehensive approach to operationalizing generative AI workloads, mirroring the GenAIOps lifecycle. It employs a two-phase evaluation strategy: rigorous build-time evaluation to preemptively catch issues before deployment, and continuous production monitoring to identify and address challenges missed by synthetic tests.

The core of this strategy is a three-layer evaluation framework that agents must successfully navigate before deployment:

1. Build-Time Evaluation: Catching Issues Before Deployment
During development and continuous integration/continuous deployment (CI/CD) cycles, the pipeline leverages the strands-agents-evals framework. This framework offers output validation, trajectory evaluation, multi-turn conversation simulation, and automated experiment generation, all natively integrated with agents built on the Strands Agents SDK. It provides three primitives: a Task (a single agent interaction), an Evaluator (which scores the agent’s output), and a Registry (which manages multiple tasks and evaluators). Tests are structured into distinct layers, employing different grader types.

• Three Types of Graders:

  • Code-based Deterministic Graders (Layer 1): These are fast, cost-effective, and reproducible. They measure tool selection, parameter passing, and trajectory ordering. For example, a ToolSelectionGrader verifies which tools were invoked, while a TrajectoryOrderGrader checks the sequence of calls.
  • LLM-as-Judge Graders (Layers 2-3): Utilizing models like Claude Sonnet 4.6, these offer flexibility in assessing reasoning quality, output helpfulness, and goal success. While non-deterministic, their variability is managed through the pass^k metric.
  • Human Review (Calibration): Used sparingly for edge cases and safety concerns, human review calibrates the prompts used for LLM-as-judge evaluators.

• The Three-Layer Assessment Framework:

  • Layer 1: Tool Usage (>95% Threshold): Focuses on whether the agent called the correct tools with the right parameters. This is measured deterministically using code-based graders. A ToolSelectionGrader ensures the right tools are chosen, and a TrajectoryOrderGrader validates the call sequence.
  • Layer 2: Reasoning (>85% Threshold): Assesses the logical coherence of the agent’s decision-making process. HelpfulnessEvaluator and TrajectoryEvaluator use LLM-as-judge scoring to ensure the agent’s reasoning is sound, preventing unpredictable failures even if the final output appears correct.
  • Layer 3: Output Quality (>90% Threshold): Evaluates the final user-facing response for helpfulness, accuracy, and actionability. OutputEvaluator and GoalSuccessRateEvaluator employ LLM-as-judge evaluation to confirm that users receive useful and well-formatted information.

Crucially, all three layers must pass their respective thresholds for an agent to be deployed. A failure at any layer acts as a blocking gate in the pipeline.

Handling Non-Determinism with pass^k:
Recognizing that LLM outputs can vary between runs, leading to misleading single-trial results, the run_all_layers() function incorporates a num_trials parameter. The pass^k metric, borrowed from code generation research, measures the probability that at least one of k independent trials successfully generates a correct output. For customer-facing agents, pass^k is paramount, as users expect consistent quality in every interaction. An agent with a 75% per-trial success rate only has a 42% chance of passing three consecutive trials (0.75^3), underscoring the need for multi-trial evaluation.

Test Case Management and Multi-Turn Conversations:
Test cases are meticulously organized by category: positive (what the agent should do), negative (what it should refuse), and multi-turn (conversational coherence). Motorway’s test suite expanded from 50 to 150 cases in three months, directly informed by real user behavior and production monitoring. The framework also includes ActorSimulator to generate realistic multi-turn interactions and InteractionsEvaluator to score context retention across turns, addressing critical issues like context drift and pronoun resolution that single-turn tests miss.

2. Production Monitoring: Continuous Oversight with AgentCore Evaluations
Once a Strands Agent is deployed to Amazon Bedrock AgentCore Runtime, AgentCore Evaluations provide continuous, real-time monitoring. This is achieved through OpenTelemetry instrumentation, an industry-standard observability framework, with observability traces sampled at 1-5% and metrics aggregated to Amazon CloudWatch for analysis and alerting.

• Two Monitoring Approaches:

  

  • On-demand evaluation: Allows for detailed analysis of specific agent interactions by selecting spans from CloudWatch logs, ideal for debugging and validating fixes.
  • Online evaluation: Automatically samples live traffic at a configurable rate (typically 1-5%) and applies evaluators in the background, providing continuous performance insights.

• Built-in and Custom Evaluators:

  • Built-in: AgentCore offers pre-configured evaluators for common scenarios, such as Builtin.Helpfulness, Builtin.GoalSuccessRate, Builtin.ToolSelection, and Builtin.Correctness.
  • Custom: For domain-specific requirements, users can create custom LLM-as-a-judge evaluators. Motorway’s implementation includes custom evaluators for DataFreshnessEvaluator (validating auction-cycle timestamps), SafetyGuardrailEvaluator (blocking automated bidding actions), and DealerDataScopingEvaluator (enforcing data isolation). Other examples include LatencyEvaluator and CostEvaluator.

• Key Metrics Tracked: CloudWatch dashboards and alarms provisioned by an AWS CDK stack track vital agent health metrics:

  • Task completion rate (>95% target, <80% alert)
  • Tool selection accuracy (>95% target, <90% alert)
  • Helpfulness score (>0.83 target, <0.58 alert)
  • Response latency P50/P99 (<2s/<10s target, >5s/>15s alert)
  • Hallucination rate (<2% target, >5% alert)
  • Cost per interaction (monitor trend, >2x baseline alert)

Operationalizing Quality: The Deployment Pipeline as a Gate

Evaluation is not an afterthought but a critical quality gate embedded within the deployment pipeline. This five-phase pipeline ensures that any failure blocks deployment and feeds new insights back into the system:

1. Build-Time Evaluation: Integrated into CI/CD, this phase runs initial strands-agents-evals tests.
2. Staging Validation: The candidate agent is deployed to a full staging environment for comprehensive testing against realistic scenarios.
3. Shadow Mode: A crucial intermediate step where the candidate agent processes a copy of real production traffic in parallel without affecting live users. Outcomes are compared, and a deviation threshold (e.g., 2%) automatically pauses deployment if exceeded. Shadow mode is vital for catching integration errors, data discrepancies, and unforeseen traffic patterns that synthetic tests might miss.
4. A/B Testing: For major releases, controlled rollout to small user groups allows for real-world performance comparison against the existing agent.
5. Production Rollout: Full deployment once all preceding gates are cleared.

This rigorous multi-stage gating ensures that only thoroughly validated agents reach production, minimizing risks and maximizing user satisfaction.

Tangible Results and Transformative Impact

The implementation of this comprehensive evaluation pipeline has yielded dramatic improvements for Motorway’s dealer stock search agent. Prior to its adoption, the agent exhibited an 87% tool selection accuracy, meaning 1 in 8 dealer queries returned incorrect results. The team faced an average of 12 production incidents monthly, with issue detection taking an average of four hours after impacting dealers.

Post-implementation, the results are transformative:

The business impact of these improvements is profound. Dealers can now complete complex vehicle searches in minutes rather than hours, with unwavering confidence in the accuracy and currency of the results. This enhanced efficiency directly translates to improved dealer productivity, faster inventory turnover, and a more competitive marketplace for Motorway. The reduction in production incidents and detection time also significantly lowers operational overhead for Motorway’s engineering and support teams, allowing them to focus on innovation rather than reactive problem-solving.

Broader Implications for the Future of AI Agents

Motorway’s success with AWS PACE provides a critical blueprint for any organization seeking to deploy production-ready AI agents across diverse industries. The core lesson remains: a fluent or confident-sounding AI response does not inherently signify that the agent has performed the correct action. Rigorous verification of tool selection, parameter correctness, reasoning coherence, and consistency across repeated runs is non-negotiable for building trustworthy AI systems.

These patterns are universally applicable to most multi-tool, customer-facing agents. Whether it’s a customer service agent querying knowledge bases and ticketing systems, a financial advisory agent pulling portfolio data and market feeds, or a healthcare triage agent accessing patient records and scheduling tools, the need for robust evaluation is constant. The development of this pipeline marks an evolutionary step in AI, shifting the focus from mere LLM performance to the comprehensive operational integrity of AI agents. It ensures that generative AI moves beyond experimental phases into reliable, high-impact applications that drive tangible business value.

Organisations looking to embark on a similar journey are advised to adopt a phased approach: build an initial test suite from real user queries (20-50 cases, including positive and negative scenarios), configure build-time evaluation with appropriate deterministic and LLM-as-judge graders, enable production monitoring with AgentCore Evaluations at a conservative sampling rate, and most importantly, establish a feedback loop that transforms production failures into new regression test cases. This iterative refinement process consistently improves evaluation quality and agent performance over time, fostering a culture of continuous improvement in AI operations.

The collaboration between Motorway and AWS PACE exemplifies how strategic partnerships and advanced engineering can overcome the inherent complexities of deploying AI at scale, delivering not just innovative technology but reliable, high-performing solutions that redefine industry standards. This sets a precedent for the secure, cost-effective, and impactful deployment of intelligent agents in the enterprise, ensuring that AI’s potential is realized with unwavering trust and operational excellence.