A/B Testing Basics

A/B testing allows you to create two versions of an email and measure how a small change affects the results of your email or campaign. For example, you might test:

• Two different subject lines
• Different button colors
• An image at the top versus the middle of an email
• Different call-to-action language
• An image versus a GIF
• Personalization within a subject line or greeting

Even seemingly small changes can influence how recipients interact with your email and may help improve your return on investment.

Test One Variation at a Time

The most important A/B testing best practice is to limit each test to one variation at a time. If you change a subject line, font size, image placement, and button color at the same time, you will not know which change caused a difference in engagement. Instead, treat your A/B test like a science experiment and test one variable at a time. This keeps your results more accurate and helps you understand which content and visual elements resonate most with your audience.

How A/B Testing Works

In Dynamics 365 Marketing, the system establishes a test group from your audience. Version A, the control, is sent to part of the test group. Version B, which contains the variation you want to test, is sent to another part. The system analyzes the results for a specified amount of time and determines a winning version based on the metric you selected. That winning version is then sent to the remaining audience.

Choose What to Test Based on Your Goal

Before creating an A/B test, think about the audience receiving the email and what you want the campaign to achieve. For a sale announcement, you might keep the text short, rely more heavily on visuals, and test a subject line or the color of a button that directs recipients to the sale. A newsletter might contain more text, sections, links, and buttons, giving you additional elements to test. You could also compare call-to-action language. For example:

• “Browse new styles”
• “Shop the sale”

Or you could compare subject lines such as:

• “Open for an exclusive discount”
• “15% off select styles”

A/B testing helps you determine which option performs better with your specific audience.

Create the A/B Test

Start in the email builder under Marketing Execution and create your marketing email. Once you know what you want to test:

1. Open the A/B testing option in the right-side action panel.
2. Select +Add test.
3. Choose the area you want to test: the header, email body, or full content.
4. Give your test a descriptive name.
5. Select Version A and configure your control version.
6. Switch to Version B and enter the variation you want to test.

For example, if you are testing subject lines, Version A contains your original subject line and Version B contains the alternative. Elements that are not part of your A/B test should remain the same between both versions. This allows you to isolate the effect of the element you are actually testing.

Configure the Test in Your Customer Journey

Once both versions are ready, add the email to a customer journey. Open the Properties for the email tile to configure the delivery portion of the A/B test. Here, you can define parameters such as:

• Test group distribution
• Test duration
• Winning metric
• Default version if the results are inconclusive

The metric you choose should directly relate to the element you changed. If you changed the subject line, for example, open rate is a logical success metric because recipients see the subject line before opening the email. If you changed a button, link, or call-to-action inside the email, click-through rate may be a better metric because it measures whether recipients interacted with that content.

Set the Distribution and Duration

One example configuration is:

• 10% of the audience receives Version A.
• 10% receives Version B.
• The test runs for six hours.
• The system compares the selected performance metric.
• The winning version is sent to the remaining 80% of the audience.

You can also select a default version to use if the test results are inconclusive. This lets you experiment with a relatively small portion of your audience before using the results to determine what the majority will receive.

Look Beyond the Winning Metric

The metric used to select the winning version is important, but it does not have to be the only measurement you review. When evaluating the overall results of an email campaign, you may also want to consider:

• Conversions
• Website traffic
• Sales
• Revenue
• Overall engagement

These measurements can provide additional context about whether an email ultimately supported your broader marketing goals.

Use A/B Testing to Keep Improving

A/B testing makes controlled experimentation possible without requiring you to create completely different campaigns. Small changes can provide useful information about your audience and help you make more informed choices in future emails. As you continue testing, you can use what you learn to refine your subject lines, calls to action, visuals, personalization, and other content. Experimentation always carries some risk, but A/B testing keeps that experimentation controlled and focused. Over time, the results can help you better understand your audience and continually improve your email marketing strategy.