The Ultimate Amazon Feedback Email Experiment: How A/B Email Testing Can Improve Amazon Product Reviews and Seller Feedback

Chapter 1: Using A/B Email Testing for Better Amazon Buyer-Seller Messaging

What Is A/B Email Testing?

A/B testing, also known as “split testing,” allows you to create and run two comparison messages and analyze email opens, click-throughs, and conversion rates to determine the better performer. Within the Seller Labs Pro Communication Center Powered by Feedback GeniusA/B testing works alongside message analytics to help you identify and optimize the message content and configuration that best helps you achieve your communication goals with regard to seller feedback, product reviews, and customer service.

This is a great feature if you’ve been using Seller Labs Pro or Feedback Genius for a while and you want to further optimize your account, or if you’re using Buyer-Seller Messaging within Amazon’s Seller Central and you want to create more-sophisticated, more-effective messages than what’s available there. With A/B email testing you can improve both your messages and your business by using data to boost conversion rates and better engage with buyers.

Update: September 2020: Amazon has discontinued the usage of emojis within Buyer-Seller Messaging.

Why Run an Amazon Feedback Email Test?

A/B testing tells you how well your messages are performing and it provides opportunities for improvement. 


Say that you have a message sequence that requests seller feedback or product reviews from your customers and your conversion rate for that sequence has dropped. Obviously you’d wonder why that’s happened and how you can boost conversions.

You could change the subject line or edit the message itself and change how you make the request, but in doing so, you’d lose your original message and be left with no benchmarks. 

Instead, keep the original message, duplicate it, and using that duplicate, create a slightly different counterpart message using A/B testing so that you can compare which of the two messages yields the highest conversion rates. With this A/B testing method, there’s no guessing which message is more effective. You have your answer because you’ve run the two messages head to head in your own Amazon feedback email test scenario and the results will provide the data that tells you which message works best.

A/B email testing for buyer-seller messaging is not available in Amazon Seller Central. To reap the rewards of split testing, you’ll need to use the Seller Labs Communication Center Powered by Feedback Genius, just one of the many tools within Seller Labs Pro (start your free 30-day trial of Seller Labs Pro today). A/B testing is just one of the many powerful Pro features that goes beyond the basics and limitations of Seller Central. 

Finding Winners: How to Create and Run an Amazon Feedback Email Test . . . and Leverage the Discoveries

Within the Communication Center, start by creating an A message and a B message, essentially two counterpart messages that you want to compare. These messages can have different message content, different subject lines, and they can even be sent at different times. You might try creating two copies of one message and simply changing one thing between them so as to isolate a single variable and see how the different messages perform, or you could run two entirely different messages and see if one approach really moves the needle more than the other. 

Next, divide your orders based on the numbers contained in your customers’ order IDs. For example, in message A, you would enter in 12345 and that message would only be sent to orders ending those digits. By the same token, in message B you would enter in the remaining numbers (67890 which would be the other orders). You can also divide up your orders by even or odd numbers. For example, message A would be sent to orders ending in even numbers 24680 and message B would be sent to orders ending in 13579. You do you, just be consistent and balanced in your splits.

Collect the Data and Then Let It Guide You

After splitting your messages by order number and doing a few weeks of sending, you can check your message Analytics to see which message yielded high conversion rates. You can check as frequently as every week, but we recommend letting the A/B Amazon feedback email test continue for at least a month to get a clear idea of your message’s performance.

A/B testing is a powerful tool that can help you optimize your Amazon seller account to yield higher conversion rates for product reviews, seller feedback, or even just to know that your messages are getting noticed by your customers. You can even become more proficient in your messaging tone and better establish your brand’s voice so you can speak to customers in a way that resonates with them and invites them to be brand advocates and repeat buyers.

Chapter 2: Using A/B Email Testing to Optimize Buyer-Seller Messaging Open Rates

If one thing about selling on Amazon is certain, it’s that nothing about selling on Amazon is static or fixed. The challenges posed by frequent changes to Amazon policies and within consumer behaviors make it nearly impossible for sellers to predict what’s next or how to prepare for it. But the same challenges also present great opportunities for sellers who are nimble and who understand how making data-driven choices can lead to steady growth and picking up what the competition fails to collect. This is the essence of real optimization. A great place to start is by achieving dramatic, positive changes by using A/B testing to optimize your Amazon buyer-seller messages open rates.

Optimization Always and in All Ways

Just as you should be looking to optimize your product listings using keyword discovery tools and to optimize your Amazon Sponsored Advertising using PPC tools with intelligent suggestions, so should you be looking to optimize your customer communications using automated buyer-seller messaging. And I know that many sellers are doing just that and to excellent effect, but are you optimizing how you optimize?

You’re probably familiar with Chad Rubin of Skubana. Chad’s a long-time Seller Labs partner and a Seller Labs Pro user who’s had great success getting reviews and feedback with the program. Our Communications Success Team reviewed Chad’s account to identify areas where he could elicit even better results. Because there are always tests to run and settings to tweak and data to gather, we indeed found some areas where we thought Chad might optimize. Here I’ll focus on A/B testing with regard to email open rates, just one of the areas where sellers, including power sellers like Chad, can get even better.

Case Study: Improving Buyer-Seller Messaging Open Rates

Scenario: We looked at open rates for two of Chad’s messages (“General Feedback” and “Warranty”).

Recommendation: In early November, as a result of recommendations by the Seller Labs Communications team, Chad rolled out A/B testing wherein he tried two variations of each message’s subject line in order to see if one performed better than the other in terms of opens (remember that you can also do A/B testing with entire messages, not just subject lines). 

Result: During this time, a couple of things happened:

  1. Chad’s overall open rates increased for both the General Feedback and the Warranty emails as both versions of both subject lines performed quite well. Well done, sir!
  2. Chad got data indicating which of his subject lines (original message A or new message B for each message) got the better open rates. In these cases, the numbers indicated that the original subject lines Chad had been using performed a bit better than the new ones he was testing. That’s a great find as it confirms that Chad’s been using subject lines that work. But are these the best subject lines that he could be using? Not necessarily, which is why our team recommended revisiting A/B testing again in the future, this time with still-different subject lines that Chad could run against his originals.

Running an Amazon feedback email test is easy, and fast, and powerful. It can yield amazing insights that enhance your customer communications and get you more and better reviews and feedback.

Chapter 3: Lessons, Key Takeaways, and Next Steps

  1. Revisiting, even revisiting what seems to be working, in pursuit of even-better results is the essence of optimization. Be it with your product listings or your Sponsored Products ads or your buyer-seller message subject lines or body content, you can always test and find ways to improve. Remember that what works best today is not what will work best tomorrow, so test again and in new ways always.
  2. Don’t be discouraged if your improvement is only by a small percentage. That’s still an improvement and there is gold in the margins. Over time, enough small improvements add up to a big difference.
  3. Never ever stop testing, trying, and pursuing even incremental improvements. There is always a better way to be found through testing and fortune favors those who aren’t afraid to listen to the data speak and to act upon it. Or, as our friend, seven-figure seller and Seller Labs Pro user Liran Hirschkorn, put it so well:

“I constantly want to be testing everything to see what could be working better. In internet marketing, this is called split or A/B testing. With the Seller Labs Pro Communication center Powered by Feedback Genius, I can choose to send half of my customers one email, and the other half a different email, and then test which one worked better. For example, if I want to increase my email open rates, I can start by leaving my entire email sequence the same and only change my subject lines. In one subject line, I can test ‘How do you like your garlic press?’ while simultaneously testing ‘Here’s a simple way to clean your new garlic press’ in another sequence.”

You Owe It to Yourself to Find Your Best Techniques

If you’re ready to see what A/B testing can do for your open rates and overall buyer-seller messaging, all you need is a Seller Labs Pro account. If you don’t have one yet, get a free 30-day trial and start improving all aspects of your Amazon game.

Or, if you’d prefer to leave A/B testing and your Amazon buyer-seller messaging optimization to the experts, check out Seller Labs Managed Services Amazon Buyer Seller Messaging Optimization.

Stay Up to Date on All Things Amazon and eCommerce

    A cup and a folder

    We're sharing all the secrets it takes to be a successful Amazon seller.

    This tell-all guide goes back in the vault soon, so act now!

    Unlock Access

    Let’s stay in touch!

    Add your email to stay up to date 
on the latest Amazon, eCommerce, and Seller Labs news!