Let AI do the work: Duplicate lead detection to keep your CRM data clean

Your processes are only as good as your data. If it’s incomplete, messy, or unorganized, data can lead you astray. Your business applications should include features to help facilitate a clean database so those apps yield the best analytics and actionable insights.

In a CRM—Customer Relationship Management—solution, your account and contact records are crucial. This app is, after all, the way you track your customers’ actions, your communications with them, their preferences, needs, and history with your company. These records need to be complete, and they need to be correct. There’s no negotiation there.

Your database can hold a lot of these records, which can be pulled in from a variety of avenues (manual entry, integrations with eCommerce, websites, marketing pages and forms, etc.). Without oversight, it’s easy for a database to quickly become inundated with the exact type of record that makes your data useless (think incomplete, duplicate, unorganized records). Plus, extra records fill up your data capacity faster and can impact the system’s performance.

But Dynamics 365 Sales offers duplicate detection for your records. Duplicate detection works by flagging fields with identical unique values (last names, email addresses, phone numbers, company names, etc.—info that is unlikely to be common) across several records. From there, you can decide how to handle these overlaps.

Now, CRM also includes a new duplicate lead detection feature that detects not only exact matching records but near matches as well. This is a great addition because, as all sales and marketing individuals know, the Leads entity is perhaps the messiest entity in a CRM due to their impermanent nature. After all, they are potential customers at the beginning of the sales funnel. Some of them work out, and lots don’t.

So, let’s discuss how we can use this new feature to detect duplicate leads when they enter our system!

Turn on the new duplicate detection feature on leads

Before anything happens, you have to enable the feature, and you’ll need the System Administrator role to do so.

Go to the Area picker in the bottom left menu of Dynamics 365 Sales. Pick the “App Settings” area. In the left navigation bar you’ll see the section “Data Improvement” under which is “Duplicate detection.” (This is specific to the new lead duplicate detection feature. To see current duplicate detection rules, you can head to “Advanced Settings” in the Power Platform Admin Center.)

Toggle the setting on! The setting may take time (hours or even a few days) to activate, and you also may need to close out and reopen your app before you can use the feature.

Using lead duplicate detection

When you enable duplicate detection for leads and you have a record selected, you’ll see a “View Duplicates” button in the top navigation ribbon of the Lead list page. This button will only appear if the selected record has duplicates.

Duplicates are identified using an AI-based fuzzy matching algorithm using these rules:

• exact matches of email addresses
• exact match of phone numbers
• similar name and/or company name, and
• similar names in the same email domain

This “fuzzy matching” varies from the old lead detection rules that checked for only exact matches, not close matches. Close, but not exact, matches are actually the space where a ton of errors reside. Let’s look at an example.

In the image below, these two leads are being flagged as possible duplicates because they have a similar spelling in their last names (Murphy vs Merphy) and because they have the same exact company name (Contoso). The old duplicate lead detection would not have flagged this because Murphy and Merphy are not exact matches. But with the fuzzy matching algorithm, this new feature does!

Another way this feature works is when you’re in a lead record, you may see a yellow bar across the top of the screen that says, “Possible duplicate records found.” Clicking the “View possible duplicates” button in this ribbon will open a window with the details of the current record and possible duplicate records.

In this example, we see that these three records have the same phone number and two of them share a company name.

You can select one of the records and click Delete or Detach. Detaching tells the system to keep all records and that they are not duplicates of each other. Delete will remove that selected record permanently.

Considerations

If you’re not seeing duplicate detection, there are a few things you can check to get it up and running:

1. Dataverse search must be enabled in your environment.
2. These fields need to be enabled in the “Quick Find All Leads” view in the environment you’re working within.

   • firstname

   • lastname
   • emailaddress1
   • telephone1
   • companyname

   To do this, you’ll need to go to make.powerapps.com and follow these instructions.

3. Users must have at least read permissions for Process under the Customization category to use the feature. To learn more about security in the Dynamics 365 CRM apps, we have this blog for you!
4. Make sure these processes are “Active” to make sure detection will work properly:

   • CheckforDuplicatesAction

   • DuplicateDetectionTriggerAction
   • GetDuplicatesAction
5. You can enable these on the Processes page under Settings > Advanced Settings. From that page, go to Settings > Process Center > Processes.
6. If you’re using a custom app instead of the out-of-the-box Sales Hub or Sales Professional app, then you may need your partner to help you get this feature set up properly.

Best Practices for clean data

When you keep your data clean, you can find what you’re looking for faster, your data analysis is more accurate, and—especially in the case of clean customer relationship management data—you can nurture your current and prospective clients better by having all their historical and contact data intact and in a single location.

Besides using duplicate detection in CRM, let’s touch on a few other best practices for keeping your data clean!

• When importing data, make sure that data is clean. You want to follow all the practices that you follow in your CRM with the data you’re about to put into it, or else you're creating more work for you and your team. When importing into Dynamics 365, you have the option to check for duplicates, which will use any published duplicate detection rules. This is a GREAT way to keep your data clean during import.
• Identify what data is important to your business and include that data as “required” fields in CRM.

  • Having the minimum required data for your leads – for instance – can help you with lead scoring and qualification, and it can help you better market or sell to the lead based on your specific business.

  • Maybe you offer great Birthday discounts – you’d absolutely want to collect that data to then market that discount at the appropriate time.
  • Maybe your salesperson really needs to know the job title of a contact so they can better approach them for B2B sales.
• Try to incorporate as many of your “required” fields as possible in your marketing forms and when manually gathering data for initial record setup as is appropriate. If you need a birthdate, include it. If you need a job title and a business name, include those. Get ahead of the game and just ask for those things right off the bat.
• You can also prioritize data by including those columns in your business process flows. Maybe you must capture X, Y, and Z info to qualify the lead, that convenient taskbar makes your important data easier to complete.
• Use reports! You can create an Advanced find query and export it to Excel, create a dashboard, or even get Power BI – a data visualization tool through Microsoft – involved to help you identify outdated or unqualified records that have missing data or are duplicates. Even creating a few custom views in your lists or in a CRM dashboard can be of great use, too.
• Use those reports during regularly scheduled and routine CRM maintenance. Decide how often you will run reports and clean up your system. Maybe monthly or quarterly will work for you. You can delegate this to your team, as well, and put someone in charge of spearheading CRM data audits.
• Use the “least privilege” security approach in your systems. This is a practice Microsoft recommends where a user is only given exactly the permissions or roles they need and nothing more.

  • Ideally, you’d avoid granting too much permission to an employee who doesn’t need it—they might make changes they shouldn’t and not even realize their error.

  • Remember that security roles can be edited temporarily or permanently on an as-needed basis. You can provide increased permissions when they are needed and remove them when they are no longer needed.
• Create and maintain standard data practices. What are required fields, how do you qualify or disqualify a lead, how do you merge records, is record deleting allowed, what job roles have what permissions for viewing and modifying records? Write these down so they are followed.
• Finally, focus on properly training your team. You want to invest time in teaching your team these standard data practices you’ve established, as well as making sure they are well-versed to use CRM competently.

You need accurate, complete data all in one place so your team can make the best decisions when interacting with each potential and current customer on an individual level. And, you need that accurate date to analyze your sales processes to make organization-wide changes to hone your sales approach, ultimately gaining more, happier customers. Duplicate detection and these best practices will hopefully get you closer to that goal of a clean database.