How to Create a Continuous Optimization Loop with Salesforce Einstein and Data Cloud

As a Salesforce Admin, you know how important it is to get timely feedback on solutions your users rely on. When it comes to solutions like Einstein Copilot—where the user interface (UI) is pervasive, operates in real time, and might give slightly different answers to different users—how can you stay ahead on the robustness of prompts and actions? If a user witnesses an answer with a hallucination, or provides data irrelevant to the question, how can they easily report this behavior?

You need to be able to see how trust, like data masking and toxicity detection, is working to ensure the safety and accuracy of generated responses. You also need the flexibility to easily add additional custom metrics to measure how artificial intelligence (AI) is working for you. Having these capabilities allows you to identify areas of improvement and optimization across your predictive and generative AI-powered apps.

Salesforce’s Einstein 1 Platform, a comprehensive suite of AI-powered tools, offers immense capabilities, and when combined with a continuous optimization loop, it becomes a game-changer. Utilizing the Einstein 1 Platform and Data Cloud, admins can allow users to easily give feedback, create reports to get a high-level view of the quality of responses, and set up flows to get alerted on feedback or audit data and take appropriate action.

In this post, you’ll learn how to set up a continuous loop of feedback to help improve your users’ AI experience on the Einstein 1 Platform. Let’s break down four easy steps you can take to create that continuous optimization loop all while working under the Einstein Trust Layer.

Step 1: Turn on Einstein generative AI and feedback data collection

Data Cloud is critical for this exercise because it can be used to maintain and update large quantities of data, fast. Data Cloud supports a large volume of diverse data collection as well as the ability to have your data visible in Salesforce and accessible through automation and flows.

As part of the Einstein setup process, customers have to opt in to store feedback data in their Data Cloud instance. This enables data-driven understanding via reporting, analytics, and monitoring/alerting. Once you’ve decided to opt in, various audit and feedback signals are ingested in your Data Cloud data lake objects (DLOs). These are automatically mapped to data model objects (DMOs).

To opt in to store feedback in your Data Cloud instance, start by turning on Einstein generative AI and feedback data collection and storage from Einstein Setup. You must first turn on Einstein and ensure Data Cloud is fully provisioned in your Salesforce org. By turning on Einstein generative AI and feedback data collection and storage, you consent to storing your data in Data Cloud.

• From Setup, enter “Einstein Setup” in the Quick Find box, and then select Einstein Setup.
• From the Einstein Setup page, turn on Einstein Generative AI Data Collection.

Einstein generative AI audit and feedback data appears in Data Cloud within 24 hours after you turn on collection and storage. You can turn off data collection and storage at any time using the toggle. When you turn off data collection and storage, the data stream collecting the data pauses. Any reports show a gap between the time you turn it off and turn it back on.

Step 2: Track Einstein Trust Layer functionality with Audit Trail

Once this is enabled, you can monitor the usage of generative AI in your Salesforce org. With Salesforce Audit Trail, you can track the various modifications that are done to your org (like updates to fields, picklist values, etc.). The audit trail, along with feedback data, is stored in Data Cloud. Use this data to see how the Einstein Trust Layer protects your company’s sensitive data from exposure to an external large language model (LLM). You can also use it to verify the safety of the generated response.

Step 3: Install the out-of-the-box reporting package and dashboard

The out-of-the-box reporting package and dashboard make it easy to access the Einstein generative AI and feedback data report package directly from Data Cloud.

1. Use this link to install the Einstein Generative AI Audit and Feedback Data reports package.
2. Then, in Data Cloud, click Dashboards. In the Search Recent Dashboards field, enter “Einstein Generative AI and Feedback Data Dashboard”.

This gives you a quick glimpse of the weekly number of users, number of generative AI requests, user feedback thumbs up, thumbs down, and tokens used. You can easily drill into any of these metrics to analyze deeper.

One of the key use cases of the Audit Trail is reviewing the effectiveness of personally identifiable information (PII) data masking in prompts.

You can create a simple report to do this.

1. Select report, choose Data Cloud, and use the Gen AI Gateway Request DMO.
2. Choose relevant fields including timestamp, model, prompt, and masked prompt.
3. You can find a prompt with private information like credit card number, phone number, or email and confirm that those fields are marked before sending the prompt to LLM, ensuring data privacy.

Step 4: Set up alerts and take proactive steps to optimize your AI performance

In addition to reporting and analytics, you can proactively monitor and get alerted on feedback or audit data and take appropriate action.

Use case: Let’s say you want to get proactively notified when more than 10 users provide negative feedback (thumbs down). You can create calculated insights in Data Cloud to aggregate negative user feedback and schedule it to be updated on a regular basis. Then, use Data Cloud-triggered flows to get notified if the count is more than 10. The notification could be an email, Slack message, or creating a case. Once you get notified, you can dig deeper into the data to understand potential reasons for negative feedback and take corrective actions.

Improve your users’ AI experience with the Einstein 1 Platform

When you enable generative AI features for your users, user feedback and audit logs are saved in Data Cloud. This makes it easy for you to gather insights from audit data and feedback and share them with your team, and continuously optimize the quality and efficacy of your users’ experience with the Einstein 1 Platform. Prebuilt Data Cloud reports and dashboards make it easier and faster for you to analyze the generative AI and share insights like user trends, acceptance rates, and response feedback.

Remember, optimizing the deployment of generative AI at your organization is not a one-time effort. It requires ongoing monitoring, analysis, and refinement to adapt to changing customer needs and market dynamics. By continuously leveraging trust audit logs and LLM feedback data, you can help your organization stay ahead of the curve and drive success for the business.

It’s time to unlock the full potential of the Einstein 1 Platform and achieve remarkable results with CRM solutions like Sales Cloud, Service Cloud, Marketing Cloud, and Commerce Cloud.

Resources

• Salesforce Help: Einstein Trust Layer
• Salesforce Help: Generative AI Audit and Feedback Data