A/B testing your content on social media

When publishing content online, attracting readers to your content is critical. The combination of headline and image is key, with social networks often using these in their "link previews" when your content is shared on their platforms. With such a large proportion of traffic coming from social media, a compelling first impression is critical. But if you have several different ideas for headline and image combinations, how do you determine which headline and image combination will generate the most engagement on social media platforms?

A/B testing is a very common way of testing out variations, but can be tricky due to the way Facebook, Twitter and LinkedIn handle their link previews internally. A/B testing requires the ability to quickly change, while social networks will attempt to cache the key information about an article for a long time. How can we get the benefits of A/B testing on social media while dealing with this constraint?

What is A/B testing?

A/B testing is:

a way to compare multiple versions of a single variable, for example by testing a subject's response to variant A against variant B, and determining which of the variants is more effective

A/B testing headline variations

A/B testing is very common on e-commerce sites, as well as with online publishers. If we have a button on a product page to buy an item, we may have a few different ideas for the possible button label ("Buy now!", "Checkout", "Get this item"), maybe a few different ideas on the colour (green, orange), and possibly some ideas on the position the button should be in. A/B testing allows us to generate some tests, and see which performs better. Taking our label text, we'd split our users into 3 groups at random. The "random selection" is important here, as in theory it should stop our results being influenced by the attributes of one particular group (age, income, location, etc).

1/3 would see "Buy now!" as the button text, 1/3 "Checkout", and the remaining third of users see "Get this item". We leave this test run for a while, and then check to see if any one of those groups had sales significantly higher than the others. For example, "Buy now!" ends up converting at a 2x higher rate than the other two options. We've a clear winner, so we can now end the test and show "Buy now!" to all users. We can repeat this test with button colours and positions, ultimately ending up with the most optimal version of our product page.

It's a similar story for online publishers - trying to find the right combination of headline and image is a big challenge. Large publishers like the New York Times often go through up to 8 variants of their headlines before settling on the best one:

The Times also makes a practice of running what are called A/B tests on the digital headlines that appear on its homepage: Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour. At the end of the test, The Times will use the headline that attracted more readers.

There's a great writeup here about how the NYT headlines evolve over time, before settling on a winner.

Testing multiple headline variants

How do social media companies complicate this?

When a link is shared on social media, Open Graph tags are used to generate link previews. These are tags which a site can provide to give a headline, image, and some other meta data about a site, in order to entice visitors from social media.

<meta property="og:title" content="Parisians overwhelmingly vote to ban for-hire e-scooters from their streets | BreakingNews.ie">
<meta property="og:description" content="Paris mayor Anne Hidalgo hailed the vote as a success.">
<meta property="og:type" content="article">
<meta property="og:url" content="https://www.breakingnews.ie/world/parisians-overwhelmingly-vote-to-ban-for-hire-e-scooters-from-their-streets-1457296.html">
<meta property="og:site_name" content="BreakingNews.ie">
<meta property="og:image" content="https://img.resized.co/breaking-news/ey[...]=/parisians-overwhelmingly-vote-to-ban-for-hire-e-scooters-from-their-streets.jpg">

Open Graph tags converted to link preview

The above code appears in the code of the article, and is read by networks like Twitter, Facebook, and LinkedIn the first time it is shared on their networks. That code is then used to generate the link preview in the tweet screenshotted above.

A challenge with A/B testing here is that social networks are huge, and try to optimise their resources. If 50 million people are going to share a link to an article on your site, Facebook, Twitter, and LinkedIn don't want to make 50 million hits to your site to generate a link preview - ideally they'll make a single request the first time the site is shared, then cache that in their systems for a long time. This works well for performance, but does mean that these networks don't have a great view of any changes that may happen in the original page after the first share. Ask anyone who works on a busy site about problems updating fast-moving news links, and you'll hear all kinds of horror stories about this! On Facebook, for example, this information will be cached for around 14 days, so any changes made to your site won't be seen in the meantime. There are ways to request that cache be purged, but it's a manual process, which doesn't fit neatly into the rapid changes that A/B testing requires.

Social share flow

When somebody shares a link on a social media network, the flow to generate the link preview is:

1. User adds a link to their post
2. Social media network checks their cache, to see if they have seen this link before
3. If not, a request is made from the social media network to the link being shared
4. Any open graph tags found on that page will be read, and saved to the network cache
5. The link preview information is returned
6. A link preview is shown to any user viewing the tweet, facebook post, etc

To keep response times low, the social media networks ideally want steps 3 and 4 to happen as little as possible - after the first share, the flow for subsequent users sharing the same content should go from step 2 straight to 5. This reliance on cache keeps things fast, but does make it very difficult for us to run A/B testing in a traditional manner. Changes we make to the site for different user groups won't be reflected in the content shared to social networks. As a large portion of our audience is likely seeing our content on social media, that's a huge part of our potential audience excluded from our optimisation tests. Even if we do run A/B tests on our own site, when users share the content, it will always generate the same link preview, as the link preview is tied to the site ultimately being shared.

The Solution: meta refresh!

The solution is deceptively simple - we can create a small HTML page for each variant we want to test, containing nothing other than the open graph tags, and a meta refresh to send the user to the publisher's article. When the social media bots see this link being shared, they will crawl the content, but not pay any attention to the meta refresh. The bots will follow redirections if they are sent via a 301, 302 or 307 response code, but if we send a HTTP 200 response code, and the bots will happily read the open graph tags in our header, and won't execute the redirect, so as far as they're concerned, this is a stand-alone page, not a thin redirect. However, when a user visits this page, their browser will respect the refresh, sending them through to the target page without any perceptible delay. This means that publishers can test a variety of headlines for engagement without worrying about social media networks caching the open graph tags for the main article.

Let's take an article at url example.com/article-123456. If we visit that URL, we see the full article. However, let's add a URL parameter to indicate we're running a test, and want to show the first test in our group, test1. This will look something like https://example.com/article-123456?ab=test1. When that parameter is detected, instead of generating the article page, we generate a short HTML page like the below:

<!DOCTYPE html>
<html>
  <head>
    <meta property="og:title" content="Variant A Headline" />
    <meta property="og:image" content="https://example.com/variantA.png" />
    <meta property="og:description" content="Variant A description" />
    <meta property="og:type" content="article">
    <meta property="og:url" content="https://example.com/article-123456?ab=test1">
    <!-- The critical bit! This forces the site to appear to instantly reload the original article -->
    <meta http-equiv="refresh" content="0;url=https://example.com/article-123456?fromab=test1" />
  </head>
  <body>
  </body>
</html>

The meta http-equiv="refresh" is what tells the user's browser to reload, after 0 seconds, and go to https://example.com/article-123456?fromab=test1. This will render our original article, with the parameter fromab=test1 allowing us to ensure that the headline and image we generate on the article page match what the user has seen in the social media preview for their test group.

// Article page
// Our A/B variants
$testHeadlines = [
    'test1' => 'A very interesting headline',
    'test2' => 'A headline which is very interesting',
];
// Default headline
$headline = 'An article headline';
// Check if we're coming from an A/B test referral, and update
// the headline accordingly
if (request()->has('fromab')
    && isset($testHeadlines[request()->get('fromab'))) {
    $headline = $testHeadlines[request()->get('fromab')];
}
...
<title>{{ $headline }}</title>
...
<h1>{{ $headline }}</h1>

We can also modify our social share buttons on the page to ensure that anyone selecting "Share to facebook" gets the share dialog pre-populated with our A/B test mini HTML page.

// Generate facebook share link
echo 'https://www.facebook.com/sharer/sharer.php?t='
    .urlencode('Variant A Headline')
    .'&u='.urlencode('https://example.com/article-123456?ab=test1');
// https://www.facebook.com/sharer/sharer.php?t=Variant+%0AA+Headline&u=https%3A%2F%2Fexample.com%2Farticle-123456%3Fab%3Dtest1

A key thing to note in the code above is the og:url value - it is pointing at the mini HTML page we've built, not the original article. Social media networks will often use this value to de-duplicate sharing of articles where different urls will be used, often due to link trackers which don't change the content. For example, the urls example.com/article-123?utm_tag=x, example.com/article-123?fbcli=2334, and example.com/article-123?more_tracking_tags=1 all generate the same content, with the tracking tags having no impact on the body of the page. Using example.com/article-123 as the og:url in this case allows the social media networks to generate link previews from their cache of example.com/article-123. This keeps the number of requests to example.com lower, and also means more consolidated statistics. In our case, we want to have a headline and image tied to this particular test, which is why it's key that the og:url points to the mini HTML page for this test, not the original article.

Are there other uses for this technique beside A/B testing?

This technique can be useful to help drive engagement for things like polls, surveys, predictors and the like. Take the example of a football tournament predictor - these are often great for engagement, in the lead up to a major tournament like the World Cup, asking readers to predict the outcome. Usually in the prediction flow, when you get to the end and see your result, you're encouraged to share the predictor tool on social media, but the shared link ends up as a generic "Try our predictor"-type prompt. Using the mini HTML strategy above, we could create mini pages which contain the predictions a particular user has made, and invite some friendly competition within their social network from fans of rival teams.

Personalised poll results

The same approach could be taken for polls, quizzes, and any type of page which looks for this sort of reader engagement. Personalising the shared result can lead to a significant uptick in overall engagement.

What are the potential drawbacks to this approach?

Some social networks will take the number of times a link has been shared as a sign of popularity, and may apply a visibility boost to it. If users are sharing different links which appear to go to the same place, the social network may not give "credit" to the original link for all of those posts. In a short-lived A/B test, the impact of this should be minimal, but is worth considering if following the "share my choices"-type promotion from the football example above. In that case, the extra engagement from a more compelling share image may more than offset reduced organic visibility of the main link being shared, but it's certainly worth considering the trade-offs!

In summary, A/B testing is a valuable tool for publishers looking to optimize their headlines for engagement on social media platforms. However, the caching of open graph tags can make it challenging to test multiple headlines. By creating small HTML pages for each title variant, publishers can test a variety of headlines for engagement without worrying about social media networks caching the open graph tags for the main article. Give it a try and see how it can improve your engagement rates!