Artificial intelligence (AI) agents are intelligent systems that can reason and act on behalf of customers and employees. But to realize their full potential, we need to test and configure agents without disrupting live production environments. That’s where you as an admin come in.
Building an effective AI agent isn’t just about deploying advanced models—it’s about ensuring these agents work reliably and consistently in real-world scenarios. Your goal is to get fast user feedback and use those signals to improve the accuracy and performance of AI agents in a production environment.
Admins play a crucial role in a strong testing and evaluation (Evals) framework, and you can follow a step-by-step process to test Agentforce AI Agents using the Agentforce Testing Center. Let’s dive in!
Autonomous AI agents are powerful but often encounter common pitfalls that can limit their effectiveness.
By testing agents, you can avoid these pitfalls and anticipate all the ways customers and users might interact with an agent.
Admins building agents need to accurately test all the different ways a customer may pose a question or interact with an agent. In addition to Agent Builder, which features a Plan Tracer for investigating the reasoning process of an agent, the new Agentforce Testing Center coming in January 2025 allows teams to test topic and action selection at scale. Testing Center will be auto-enabled for all Agentforce customers to use in their production sandbox instance.
Using natural language instructions, Testing Center can auto-generate hundreds of synthetic interactions—such as requests a customer may make when engaging with an Agentforce Service Agent—then test them in parallel to see how frequently they result in the right outcome. Admins can then use the test data to refine instructions so the expected topic is more frequently selected, improving the end customer experience.
Agent Testing and Monitoring isn’t just a checkbox—it’s the foundation for admins to build reliable, scalable, and trustworthy AI agents. Testing Center provides an innovative way to use AI to test AI, from creating a test dataset to evaluating thousands of test cases in one job to gain the confidence that the agent you build is ready to be deployed in a production environment.
The AI Agent Testing Loop is a step-by-step process to improve AI agents. It starts with creating test scenarios, selecting evaluation metrics, and running automated tests. Admins then validate results, and the feedback is used to refine the agent’s instructions and performance, ensuring better results with each cycle.
Start by understanding your agent’s scope and capabilities. Each agent includes topics (domains of expertise) and actions (tasks the agent performs). For example:
In Agent Builder, you can customize an out-of-the-box agent to any industry and any use case. You can create a job to be done by your agent by defining topics, which include natural language instructions and additional guardrails: a library of actions the agent can take using tools you already have in Salesforce, such as flows, Apex, MuleSoft APIs, and prompt templates.
Navigate to Topic Detail from Agent Builder. Then, review the Description/Scope and other details that describe the capabilities of the topic, as well as when and under which circumstances it will be evaluated by the agent at run time.
AI-generated tests for Agentforce: Clicking on the Batch Test button in Agent Builder can auto-generate hundreds of synthetic interactions, such as requests a customer may make when engaging with Agentforce Service Agent.
Alternatively, you can also create test cases directly from Testing Center. It uses the same logic of using AI to generate more test scenarios based on the Agent Metadata and what Knowledge the Agent has access to through Einstein Data Library.
A good quality test case dataset has three quality attributes:
To ensure the data has coverage and diversity, review the generated test dataset and see if the expected topic column covers all the topics associated with the agent you want to test. Here’s how to do this: Select the agent, open in Agent Builder, then go to Topic Details.
You can view a complete set of topics/actions that were evaluated for requests to the agent by navigating to Agent Builder and viewing the responses to requests. For example, here’s what your Builder will include when “Query Acme” is input to the agent. Observe that the EmployeeCopilot__GeneralCRM topic and QueryRecords action were evaluated.
A test case validates how the agent processes input (utterances) and generates responses. Each test case includes:
Once the test data is ready:
Testing Center: AI-assisted evaluation
You can view the results by selecting the Test detail view.
While automated testing handles most scenarios, human validation ensures responses align with nuanced user expectations. This step catches subtle issues, like tone mismatches or context-specific inaccuracies.
Testing is an iterative process. Use test results to:
A continuous feedback loop ensures your agent evolves and stays relevant as user needs and business goals change.
Review the action’s scope in Agent Builder, then design scenarios based on expected user interactions.
Run tests whenever you update topics, actions, or prompts to validate changes and maintain quality. For example:
Yes, Testing Center results are stored and can be analyzed using tools like Tableau or Salesforce Data Cloud reports.
Yes, Agentforce Testing Center uses AI to evaluate the test cases so you don’t have to manually. It will decrease the number of Einstein Requests used based on the number of test cases you are running. You can see the details of Einstein Request Consumption in Digital Wallet.
Testing is the foundation of building AI agents that are reliable, efficient, and trusted. By following this guide, you’ll ensure your Agentforce Agents consistently deliver exceptional results while adapting to evolving needs.
Thank you to Principal Product Designer Jon Moore and Senior Director of Product Management Deepak Mukunthu for collaborating on this article.
Welcome to “How I Solved It.” In this series, we do a deep dive into a specific business problem and share how one Awesome Admin chose to solve it. This episode has a twist—the admin is me! Key business problem What’s the one case that makes your support team’s blood run cold? It’s the ‘all-caps’ […]
Just as our world has become AI centric, it’s hard to see a working world in which you won’t hear about vibe coding. Vibe coding is an amazing shift in how applications are built, using natural language in a conversation to control how a solution is being built. Agentforce Vibes is Salesforce’s implementation of it, […]
Out of the box, Agentforce provides Salesforce Admins with a plethora of AI models to work with from leading vendors like Anthropic, OpenAI, and Google. When using tools like Prompt Builder, admins can select these models specifically to see if different models achieve different results. While they typically produce fairly similar results, newer or larger […]
