How to build a SEO experiment framework (full guide with examples)

About the author

Jeff Chang (@JeffChang30) is a growth technical leader at Pinterest and angel investor.

Introduction

SEO as a growth channel is a huge channel for many companies due to the volume of traffic Google provides. Pinterest and other sites receive millions of visits a day just from Google traffic alone. This post will go over the basics on how to go scale up your SEO traffic efforts from scratch, while later posts will go into more advanced tactics.

Building a simple experiment framework

The first thing you have to do is build a basic SEO experiment framework. This may be sound intimidating at first, I want to convince you that you can build a basic one easily. To be able to run SEO experiments, all you need is a utility function that buckets pages into the right groups and logs traffic. Then, it’s just a matter of running a query and doing calculations.

Here’s a sample bucketing function that is called when a user visits a page from Google. It returns the group the page is in, and you use the group to determine which variant of the page to show.

Screenshot 2018-04-15 15.24.43.png

After implementing this function and setting up an experimental treatment, you will end up with a logs that look like this:

To aggregate the data, all you have to do is run a simple query:

SELECT dt, experiment_group, count(*) from seo_experiment_table

GROUP BY experiment_group, dt

And then you’ll end up with some results like this:

Screenshot 2018-04-15 15.27.25.png

From there, you can build a graph that looks like this:

If the SEO experiment is successful, then there will be a sustained increased level of traffic in the enabled group. The formula we use to calculate the additional gain in traffic from shipping this experiment is:

Traffic gain = ((avg enabled after - avg enabled before) - (avg control after - avg control before)) / Experiment group size

For this experiment:

((68.7K - 45.8K) - (50.7K - 49.4K)) / .1 = 215K additional traffic per day when fully shipped

From this basic SEO experiment framework, which is essentially just a bucketing and logging function, you can upgrade it now or later by automating the data aggregation and graph displays. Another extension is that since you have a mapping of experiment group to user ids, you can also compare downstream events such as signups or logins. Finally, another important thing to note is that we require that all experiments that change the layout or content of unauth pages to be run in conjunction with a SEO experiment, in order to avoid inadvertently dropping traffic. This is key to preserve your hard earned traffic. In order to apply this at scale, you should build automated SEO experiments into your unauth experiment framework.

High leverage SEO success factors

There are many success factors that affect Google search rankings, and it changes all the time as Google’s algorithms change. However, some of these factors are hard to directly control. Ranked roughly by ROI (low time cost, high impact), they are titles/descriptions, additional relevant content, internationalization, mobile (AMP), and performance. I’ll talk a bit more about titles and descriptions and content since they are the easiest to optimize and you should try them first. Since each factor probably has enough content for its own blog post, I’ll leave the others for a later time.

Titles and descriptions

Titles and descriptions are the easiest way to increase your SEO traffic because as long as you have an SEO experiment framework and you can run a lot of experiments easily, each variant just requires a new copy. The impact of these experiments can be quite large too, as I’ve seen small copy tweaks lead to over 10% increases in traffic. You should never assume that your current title is the best possible title because you’ll usually be surprised if you run a few more variants. Also as you run these experiments, you learn more about the type of titles that work well for your product.

When coming up with new titles to test, you have to consider two main audiences: users and googlebot. If your title entices users and increases click through rate, Google in general will increase your traffic. If your describes the content on the page well, googlebot knows better what to rank you for and in general will increase your traffic.

Let’s take a look at a sample search title and description. Every single part of this text was carefully tested against multiple variants. From tests, we know:

  • What numbers perform well

  • ‘Images’ vs ‘pins’ vs other variants

  • ‘on Pinterest’ vs not

  • Descriptions at the end of the title that are cut off to users but give googlebot better signal

  • ‘Explore’ vs other synonyms

  • What board metadata to include

In fact, titles and descriptions are so important that once you start scaling out your SEO team, you can consider building a tool to run these experiments from UI so somebody could just produce tens of variants and have them automatically tested.

Conclusion

SEO is not easy to learn and work on. There are many complex factors to optimize, and search algorithms are constantly changing. However, it is greatly rewarding and is the “rocket fuel” of many web based products. I hope that from this post, you get a good idea of how to scale up SEO efforts from scratch. After that point, it’s all about optimizing more search factors and upgrading your data tools. Each factor is complex and could probably have an entire book written about them, so it’s a continuous experimentation and learning process. Have fun!

Takeaways

  • Build a SEO framework

  • Experiment with high leverage SEO success factors such as titles and content first

Related:

9 best initiatives for growing million visitor SEO landing pages | The definitive guide to SEO at scale

Scaling new growth opportunities series

Part 1: Intro to growth engineering

Part 2: SEO - Get started 10x'ing your traffic

Part 3: Signup and login basics

Part 4: Growth monitoring done right

Part 5: Start your journey to the top of the app store - mobile acquisition basics