Artificial intelligence (AI) agents are intelligent systems that can reason and act on behalf of customers and employees. But to realize their full potential, we need to test and configure agents without disrupting live production environments. That’s where you as an admin come in.
Building an effective AI agent isn’t just about deploying advanced models—it’s about ensuring these agents work reliably and consistently in real-world scenarios. Your goal is to get fast user feedback and use those signals to improve the accuracy and performance of AI agents in a production environment.
Admins play a crucial role in a strong testing and evaluation (Evals) framework, and you can follow a step-by-step process to test Agentforce AI Agents using the Agentforce Testing Center. Let’s dive in!
Autonomous AI agents are powerful but often encounter common pitfalls that can limit their effectiveness.
By testing agents, you can avoid these pitfalls and anticipate all the ways customers and users might interact with an agent.
Admins building agents need to accurately test all the different ways a customer may pose a question or interact with an agent. In addition to Agent Builder, which features a Plan Tracer for investigating the reasoning process of an agent, the new Agentforce Testing Center coming in January 2025 allows teams to test topic and action selection at scale. Testing Center will be auto-enabled for all Agentforce customers to use in their production sandbox instance.
Using natural language instructions, Testing Center can auto-generate hundreds of synthetic interactions—such as requests a customer may make when engaging with an Agentforce Service Agent—then test them in parallel to see how frequently they result in the right outcome. Admins can then use the test data to refine instructions so the expected topic is more frequently selected, improving the end customer experience.
Agent Testing and Monitoring isn’t just a checkbox—it’s the foundation for admins to build reliable, scalable, and trustworthy AI agents. Testing Center provides an innovative way to use AI to test AI, from creating a test dataset to evaluating thousands of test cases in one job to gain the confidence that the agent you build is ready to be deployed in a production environment.
The AI Agent Testing Loop is a step-by-step process to improve AI agents. It starts with creating test scenarios, selecting evaluation metrics, and running automated tests. Admins then validate results, and the feedback is used to refine the agent’s instructions and performance, ensuring better results with each cycle.
Start by understanding your agent’s scope and capabilities. Each agent includes topics (domains of expertise) and actions (tasks the agent performs). For example:
In Agent Builder, you can customize an out-of-the-box agent to any industry and any use case. You can create a job to be done by your agent by defining topics, which include natural language instructions and additional guardrails: a library of actions the agent can take using tools you already have in Salesforce, such as flows, Apex, MuleSoft APIs, and prompt templates.
Navigate to Topic Detail from Agent Builder. Then, review the Description/Scope and other details that describe the capabilities of the topic, as well as when and under which circumstances it will be evaluated by the agent at run time.
AI-generated tests for Agentforce: Clicking on the Batch Test button in Agent Builder can auto-generate hundreds of synthetic interactions, such as requests a customer may make when engaging with Agentforce Service Agent.
Alternatively, you can also create test cases directly from Testing Center. It uses the same logic of using AI to generate more test scenarios based on the Agent Metadata and what Knowledge the Agent has access to through Einstein Data Library.
A good quality test case dataset has three quality attributes:
To ensure the data has coverage and diversity, review the generated test dataset and see if the expected topic column covers all the topics associated with the agent you want to test. Here’s how to do this: Select the agent, open in Agent Builder, then go to Topic Details.
You can view a complete set of topics/actions that were evaluated for requests to the agent by navigating to Agent Builder and viewing the responses to requests. For example, here’s what your Builder will include when “Query Acme” is input to the agent. Observe that the EmployeeCopilot__GeneralCRM topic and QueryRecords action were evaluated.
A test case validates how the agent processes input (utterances) and generates responses. Each test case includes:
Once the test data is ready:
Testing Center: AI-assisted evaluation
You can view the results by selecting the Test detail view.
While automated testing handles most scenarios, human validation ensures responses align with nuanced user expectations. This step catches subtle issues, like tone mismatches or context-specific inaccuracies.
Testing is an iterative process. Use test results to:
A continuous feedback loop ensures your agent evolves and stays relevant as user needs and business goals change.
Review the action’s scope in Agent Builder, then design scenarios based on expected user interactions.
Run tests whenever you update topics, actions, or prompts to validate changes and maintain quality. For example:
Yes, Testing Center results are stored and can be analyzed using tools like Tableau or Salesforce Data Cloud reports.
Yes, Agentforce Testing Center uses AI to evaluate the test cases so you don’t have to manually. It will decrease the number of Einstein Requests used based on the number of test cases you are running. You can see the details of Einstein Request Consumption in Digital Wallet.
Testing is the foundation of building AI agents that are reliable, efficient, and trusted. By following this guide, you’ll ensure your Agentforce Agents consistently deliver exceptional results while adapting to evolving needs.
Thank you to Principal Product Designer Jon Moore and Senior Director of Product Management Deepak Mukunthu for collaborating on this article.
Chicken, meet egg As admins, we’re often confronted with a classic conundrum: How do you convince an organization to adopt new technology without having the technology to demonstrate it in the first place? This is compounded when the technology you want to demonstrate requires other technologies you may not have access to show. For example, […]
Have you ever wished you could give your users exactly the right guidance at just the right moment? That’s the magic of Next Best Action! Whether it’s helping a sales rep spot the perfect opportunity for a follow-up or guiding a service agent with a recommended solution, Next Best Action allows you to create custom […]
There is a lot of excitement around artificial intelligence (AI) right now. New uses and new models are changing the way we think about how AI can support and improve the ways we work. At Dreamforce ’24 one of the big announcements was Agentforce, a powerful companion to help Salesforce users resolve cases, answer questions, […]
