Einstein Prediction Builder: Thinking Through Predictions with Bias in Mind

Machine learning is an incredibly powerful tool that can be used for anything from suggesting an ice cream flavor to predicting customer churn. But, as much as we’d like it to be, it is not magic. Applying machine learning to a business problem involves finding patterns in your historical data and using those patterns to make predictions about new data. This means that if your data includes patterns that you don’t want to replicate going forward, or if the data you provide only represents a fraction of your use cases, your predictions will likely be inaccurate and unfair. As admins, we know how much easier and more efficient a user’s experience becomes when these patterns and predictions work correctly. So, let’s explore how these unwanted patterns can arise in your models and unintentionally lead to biased machine learning tools. We’ll also focus on how to think about your use cases and leverage Einstein Prediction Builder to build machine learning apps that you can trust are working in the way you expect them to.

To begin, let’s use a toy example. Say we train a model to detect if something is a square based on a dataset that looks like figure 1. Figure 1 includes various objects, all with different attributes like size, color, and shape. All of these objects are correctly labeled, but notice anything else? All of the squares are red, and all of the other shapes are blue. To a human that’s just something to note, but to an algorithm the color attribute has become a differentiator in what is labeled a ‘square’ or ‘not a square’. Because of this, a model trained on figure 1 might incorrectly identify the red cloud in figure 2 as a ‘square’.

The model above performs differently on shapes based on their color, but is that a problem? Determining if we should be concerned about a discrepancy in accuracy depends on how we intend to use this model. Let’s forget about color for a second and think about a model that performed differently on records based on gender. This should sound like a red flag to someone predicting something like ‘what starting salary should we offer’. However, in a different situation, a model that performs differently on various genders may actually be a good thing. For example, say we have a prediction to market a women’s daily vitamin; we may be comfortable knowing it is more accurate for women.

The difference between the two above examples is the impact. In the case of the vitamin prediction, ads better targeting the group of people the vitamin was designed for is great. Alternatively, for a model predicting salary, gender discrepancies result in an unfair disadvantage to one group. We would call this model biased.

Now, there are a lot of different definitions of bias, but for the purpose of this post we will use the term biased to mean “to wrongfully impose a relative disadvantage on persons based on their membership in some salient social group e.g., race or gender.”

Creating a Prediction Builder app with bias in mind

Now let’s walk through a more in-depth example and look at how we can leverage our new knowledge of biased models when creating a machine learning app.

Let’s say Claire and her friend Yoseph own a game store. Yoseph handles the vision for the store: who their customers are and what products they carry. Essentially, Yoseph is the business expert. Claire, on the other hand, reigns the technical side. She spends her time building out the new website and tackling the latest technological challenges. As the Salesforce Admin, Claire heard about a new tool, Einstein Prediction Builder, that lets her create some machine learning models using all the data she has stored in Salesforce — cool! Claire knows the store has a new board game in stock and wants to build a model to predict who is most likely to buy it. She has also been reading up about bias in AI and wants to be sure that the tools she builds are in line with her company’s values. As she creates her models, she goes through the following steps to ensure she understands what’s going into them.

Step 1: Meet with the business expert

First things first: To create a successful model, Claire will need to know more than just how Salesforce is configured. Since she built out the company’s Salesforce instance, she can tell you exactly what fields are on what objects. But that doesn’t mean she knows what’s going into those fields, or how they are being used day to day. Additionally, she will need to understand the details of the question they want answered by the model, while also considering how this model will be used. To figure all of this out, she sits down with her business partner, Yoseph, and goes through the following questions:

-What are we trying to predict? How will this prediction be used? Does it make sense to have a machine answer this question? (For example, as discussed above, predicting something like starting salary may not be a good choice.)
-Are there certain groups of our data we want to make sure our app doesn’t treat differently? Do we want this app to work on only a particular segment of our data, or for all of our data? What data will we want to use as examples?
-Given our example set, will any type of customer be left out? What fields should we include? Are there fields we are no longer using? Are there fields containing information we don’t want in the model? How will the results of the model be used?
-What are the business metrics we are tracking? How will we evaluate success?

Together, Claire and Yoseph come up with the following answers.
We are predicting if a given customer is interested in board games. This will be used when selecting who to market the new board game to in the next email blast.
Yes — we’re comfortable with a machine answering this question.
As part of our mission, we try to keep gender away from the game decision process. We want to keep this in mind when creating our machine learning app.
We only send ads to customers over 18, so our model should only be using those customers.
Examples would be customers who have input their preference of game. This should be pretty universal across all different kinds of customers. It should also be true for gender since that’s what we are most interested in.
There are some legacy fields we no longer use; those can be ignored. We should also ignore fields that represent gender.
The results will be used to help determine who will get an email that markets the new board game.
We want to track the % of ads in emails that are clicked; we want a higher % clickthrough rate (CTR). CTR is the proxy we use to determine if a given customer is actually interested in board games.

Step 2: Go through the flow with your answers

Claire leaves the meeting with a notepad full of takeaways from their discussion. They’ve answered the above questions, and she’s ready to plug that information into the Prediction Builder Setup!

Yoseph and Claire have agreed that they will build an app that predicts which customers may be interested in board games. They’ll use the results to send out targeted email blasts offering a discount for the new board game. As Claire sets up her prediction using Einstein Prediction Builder, she keeps her list close at hand.

Claire quickly gets to the page where she’ll determine what data the model will use. In their discussion, Yoseph brought up that although this board game appeals to both kids and adults, they have a store policy of not targeting kids with ads. Because of this, Claire makes sure this app is segmented so it only makes predictions for, and looks at examples of, people over 18. Since Claire knew how the model was going to be used, she could adjust the data used in the app to align with the company policy and values.

Claire knows they will be predicting the ‘interested in board games’ boolean field and now needs to select what examples the model will use. Claire has been reading up and knows that the example set, or training set, is an important decision. The model should use examples of customers who have input their preferences; by having examples of both customers who were interested in board games and customers who were not, the model is able to learn the differences between those types of customers.

Before continuing, Claire takes a second to think about the implications of this example set. She knows from their original conversation that she and Yoseph want to be intentional about making sure their model doesn’t perform differently for people of different genders. She gives Yoseph a quick phone call to discuss if he thought there would be any distinction between people who have and haven’t input their preferences, and he assures her that there shouldn’t be. Claire isn’t convinced, so she runs a quick report on customers who input their preferences and looks at the gender breakdown. Everything is in a range she was comfortable with, so Claire continues on but makes a note to run a similar report on the output of the model so she can verify this again at the end.

On the next page, Claire must select which fields to include in the model. Einstein has a robust way of figuring out which fields contain helpful information and ignoring the rest. Because of this, Claire can leave most fields in and only exclude those she knows contain information that should definitely not be in the model. There are certain fields Claire knows are no longer in use, so to clean up the data she removes the ‘fax number’ field. Claire also removes the fields ‘gender’ and ‘nameChanged’ since she doesn’t want any data on gender influencing the model. Yoseph had informed her that ‘nameChanged’ actually maps very closely to gender, since most customers who change their names have historically been women after getting married.

Step 3: Inspect

Voila! Einstein returned a model! Now Claire can look through the scorecard and double-check she’s comfortable with how Einstein will make its decisions. She learns some interesting things; fields she hadn’t expected to be relevant seemed to have a high impact on the model, such as ‘familyMemberId is null’. Claire realizes this must mean that being part of a family, in other words having a family member linked to the customer account, impacts whether or not people are interested in board games. That makes sense. Most of the other highly impactful predictors are ones she expected, like ‘attendedGameNight’. For more information on the scorecard, see this post.

There was one field that was being used in the model she wasn’t comfortable with: zip code. Since the model is being used for an email blast, location of the customers is unimportant. Maybe there was a link between people in different zip codes preferring board games, but Claire also knows that zip code has a close relationship with race. She consults with Yoseph and they make the business decision to not use zip code, so Claire quickly edits the model to exclude that field. In fact, by eliminating this possible zip code bias, Claire may have just expanded to new untapped board game markets!

Step 4: Monitor and iterate

Claire is happy with the scorecard of the new model so she enables the prediction. Remember that report on gender that Claire had created earlier? She can now create a similar report looking at the results of the model for the different groups. As the model continues to churn away in production, she can keep an eye on this report and similar ones. These reports allow her to monitor the model and iterate when needed. She can even share these with Yoseph so the whole business has visibility into the models and knows they can trust them. See this post to learn more about using reports.

Machine learning is an incredibly powerful tool for a Salesforce Admin. It’s important to remember that a successful machine learning app is one that both answers the desired question, and does so in a way that is aligned with your business policies and values. This starts with having the important conversations with your business partner, and continues through the creation of the model and into monitoring its use in production. Creating an effective model is an ongoing process. By inspecting, monitoring, and iterating with those original questions in mind, you can create not just a successful model but one that your company can understand and trust.

To continue learning more about AI and machine learning, be sure to join us for the first episode of the new Learn AI with Salesforce series on July 15, 2020 to skill up on key AI concepts, use cases, and hands-on demos from Salesforce Product Managers, data scientists, and special guests. The first episode of the series will cover how you can Meet the Digital Imperative with an AI-Powered Platform and be hosted by Marco Casalaina, SVP, Einstein Products, Salesforce and Sanjna Parulekar, Senior Manager, Product Marketing, Salesforce Einstein.

Here’s all the details you need!

? Wednesday, July 15

⏰ 9:00 a.m.—10:30 a.m. and 4:00 p.m.—5:30 p.m. PT

? Register here

Then, join us on Trailhead Live