As a Salesforce Admin, you know how important it is to get timely feedback on solutions your users rely on. When it comes to solutions like Einstein Copilot—where the user interface (UI) is pervasive, operates in real time, and might give slightly different answers to different users—how can you stay ahead on the robustness of prompts and actions? If a user witnesses an answer with a hallucination, or provides data irrelevant to the question, how can they easily report this behavior?
You need to be able to see how trust, like data masking and toxicity detection, is working to ensure the safety and accuracy of generated responses. You also need the flexibility to easily add additional custom metrics to measure how artificial intelligence (AI) is working for you. Having these capabilities allows you to identify areas of improvement and optimization across your predictive and generative AI-powered apps.
Salesforce’s Einstein 1 Platform, a comprehensive suite of AI-powered tools, offers immense capabilities, and when combined with a continuous optimization loop, it becomes a game-changer. Utilizing the Einstein 1 Platform and Data Cloud, admins can allow users to easily give feedback, create reports to get a high-level view of the quality of responses, and set up flows to get alerted on feedback or audit data and take appropriate action.
In this post, you’ll learn how to set up a continuous loop of feedback to help improve your users’ AI experience on the Einstein 1 Platform. Let’s break down four easy steps you can take to create that continuous optimization loop all while working under the Einstein Trust Layer.
Data Cloud is critical for this exercise because it can be used to maintain and update large quantities of data, fast. Data Cloud supports a large volume of diverse data collection as well as the ability to have your data visible in Salesforce and accessible through automation and flows.
As part of the Einstein setup process, customers have to opt in to store feedback data in their Data Cloud instance. This enables data-driven understanding via reporting, analytics, and monitoring/alerting. Once you’ve decided to opt in, various audit and feedback signals are ingested in your Data Cloud data lake objects (DLOs). These are automatically mapped to data model objects (DMOs).
To opt in to store feedback in your Data Cloud instance, start by turning on Einstein generative AI and feedback data collection and storage from Einstein Setup. You must first turn on Einstein and ensure Data Cloud is fully provisioned in your Salesforce org. By turning on Einstein generative AI and feedback data collection and storage, you consent to storing your data in Data Cloud.
Einstein generative AI audit and feedback data appears in Data Cloud within 24 hours after you turn on collection and storage. You can turn off data collection and storage at any time using the toggle. When you turn off data collection and storage, the data stream collecting the data pauses. Any reports show a gap between the time you turn it off and turn it back on.
Once this is enabled, you can monitor the usage of generative AI in your Salesforce org. With Salesforce Audit Trail, you can track the various modifications that are done to your org (like updates to fields, picklist values, etc.). The audit trail, along with feedback data, is stored in Data Cloud. Use this data to see how the Einstein Trust Layer protects your company’s sensitive data from exposure to an external large language model (LLM). You can also use it to verify the safety of the generated response.
The out-of-the-box reporting package and dashboard make it easy to access the Einstein generative AI and feedback data report package directly from Data Cloud.
This gives you a quick glimpse of the weekly number of users, number of generative AI requests, user feedback thumbs up, thumbs down, and tokens used. You can easily drill into any of these metrics to analyze deeper.
One of the key use cases of the Audit Trail is reviewing the effectiveness of personally identifiable information (PII) data masking in prompts.
You can create a simple report to do this.
In addition to reporting and analytics, you can proactively monitor and get alerted on feedback or audit data and take appropriate action.
Use case: Let’s say you want to get proactively notified when more than 10 users provide negative feedback (thumbs down). You can create calculated insights in Data Cloud to aggregate negative user feedback and schedule it to be updated on a regular basis. Then, use Data Cloud-triggered flows to get notified if the count is more than 10. The notification could be an email, Slack message, or creating a case. Once you get notified, you can dig deeper into the data to understand potential reasons for negative feedback and take corrective actions.
When you enable generative AI features for your users, user feedback and audit logs are saved in Data Cloud. This makes it easy for you to gather insights from audit data and feedback and share them with your team, and continuously optimize the quality and efficacy of your users’ experience with the Einstein 1 Platform. Prebuilt Data Cloud reports and dashboards make it easier and faster for you to analyze the generative AI and share insights like user trends, acceptance rates, and response feedback.
Remember, optimizing the deployment of generative AI at your organization is not a one-time effort. It requires ongoing monitoring, analysis, and refinement to adapt to changing customer needs and market dynamics. By continuously leveraging trust audit logs and LLM feedback data, you can help your organization stay ahead of the curve and drive success for the business.
It’s time to unlock the full potential of the Einstein 1 Platform and achieve remarkable results with CRM solutions like Sales Cloud, Service Cloud, Marketing Cloud, and Commerce Cloud.
Are you a Salesforce Admin managing multiple orgs? If so, Data Cloud One is the game-changer you’ve been waiting for. This new solution simplifies the complexities of using Data Cloud across multiple orgs, making it easy to access a unified Customer 360 profile and take action. Utilize powerful features like calculated insights, Data Cloud-triggered flows, […]
As Salesforce Admins, you’re well-versed in delivering Salesforce features to your users and stakeholders. Now, we’re making it even easier for you to get the most out of Salesforce by adding cross-department features built into your existing CRM. The exciting news from Dreamforce ’24 is Salesforce Foundations, which brings built-in, key features to Sales Cloud […]
Data Cloud provides the ability to integrate multiple sources of data and create your own data lake to use with AI. However, having access to a myriad of potential sources means we also have an array of ways to authenticate against them. Authentication patterns have flourished over the years of web development to help keep […]
