Do blogs with audio support SEO? Maybe. Our Amazon Polly experiment and results…

When Eric Siu announced on the Marketing School podcast that he increased his dwell time by 19% with a text-to-voice plugin called Polly, I was intrigued. Increased dwell time generally gets you more organic traffic because it signals greater user engagement, which increases the chances that Google will rank you well.

Moreover, we had already installed Polly on our blog for improved user experience, and we did so long before Eric shared his thoughts. But we hadn’t measured the results yet. I was also skeptical about whether the claim was legit, especially since dwell time isn’t a metric you can see in Google Analytics (although I assume he means Average Time on Page).

Since we were one of the first to offer blogs with audio, I knew there are few examples of Amazon Polly in action, let alone data on Polly’s impact on user engagement. So, I looked at our data. I used three methods to measure Amazon Polly’s impact to get as accurate and objective an assessment as possible. But first, let’s explain what Polly is in more detail. But first…

What is Amazon Polly?

Amazon Polly is a WordPress plugin that offers readers the option to convert text to voice on your website. This text-to-voice plugin can be installed directly on your WordPress site. You can see it in action by clicking the play button on the audio player above this article. The audio is a bit robotic, but voice technology is getting better every year and will get more natural over time.

The price is extremely cheap because the converter uses Amazon Web Services, a giant cloud computing platform. You may only pay a few cents per a 1,000-word article.

Now, let’s get into the different methods I used to measure results.

Method 1: A three-month overall year-over-year analysis

In my first approach, I analyzed the average session duration of individual landing pages that have existed for at least a year. That way,  we can see the before-and-after impact of equipping our blog with audio. Since we didn’t turn on Polly for every blog post, the process to identify each eligible post was manual. We have plenty of new content with Polly enabled, but not as much old content.

I looked for any noticeable changes in average session duration from April 1st to June 30th, 2017 versus April 1st to June 30th, 2018. I chose these three months because they were the most recent at the time I collected the data. Here are the results:

• For the B2B Facebook marketing strategy blog post, duration increased from 24 seconds to one minute and 12 seconds. This data consists of 317 sessions in 2018 and 123 sessions in 2017.

• For the [Infographic] How to Test Web Page Performance For Free blog post, duration increased from 5 to 9 seconds. This data consists of 218 sessions in 2018 and 155 sessions in 2017.

• For the 6 Student Recruitment Strategies for Higher Education blog post, duration increased from 14 seconds to 39 seconds. This data consists of 57 sessions in 2018 and 69 sessions in 2017.

• For the How to: Use Analytics to Map URLs with Page Titles blog post, duration increased from 4 to 40 seconds. This data consists of 58 sessions in 2018 and 32 sessions in 2017.

• For the How We Extended Slack to Boost Productivity and Return Results blog post, duration decreased from 20 to 6 seconds. This data consists of 40 sessions in 2018 and 26 sessions in 2017.

• For the Digital marketing team structure blog post, duration increased from 43 to 38 seconds. This data consists of 38 sessions in 2018 and 25 sessions in 2017.

As you can see, most of the posts experienced a noticeable increase in session duration. However, these results are not conclusive because of the following factors that I couldn’t control for:

• The traffic each post received was small, except for the digital team structure one. Outliers in such a small sample size can skew the accuracy of the data.
• The content in the blog post may have become slightly outdated. Although all these blog topics are evergreen, there are parts to them that may have become obsolete or irrelevant over time. The publish dates on the blog posts, some dating back to 2014, don’t help convince people that these posts are up to date. Old publish dates may influence readers to bounce off a post earlier than they would’ve otherwise.

Despite these influencing factors, the increase in session duration has been mostly positive.

Method 2: Event tracking

The problem with the first method is that it considers all sessions for each blog post. But it probably shouldn’t count the users who did not notice or use Polly. If we wanted to look at only the people who used Polly to see how much longer they stayed on the site, we would need a second method.

As mentioned, there’s a Polly player at the top of our articles that you can click. Using event tracking, we can measure the activity of only users who have clicked the play button.

Recently, we redesigned the player to look prettier, but it used to be just a button. Before the redesign, there was also a Polly web player at the bottom of each article, too, but we couldn’t track the one at the bottom because it wasn’t compatible with event tracking.

While we may be missing out on tracking a few clicks from the player at the bottom, this method adds increased accuracy by excluding users who visited our blog with audio but did not use the Amazon Polly player.

This added layer of precision is well worth the small cost in accuracy from being unable to track the bottom player. Few people are going to see or interact with the player below the article. Heatmap data shows that only around 18% of readers get to the bottom of our posts.

Amazon Polly is only turned on for blog posts, so the data should also be clean of Polly events from other web pages, like our About page. So, what did the data show?

Since inception, Google Analytics says there have been 18 sessions where a user clicked the Polly button. The first valid event occurred on March 1, 2018. The Average Time on Page for these sessions was 3 minutes and 36 seconds, which is high.

The average time on page of events ranges from 5 seconds to 22 minutes. The image below shows you some of the variance of average time on page by blog post.

Similarly, I found that session duration ranged from 20 seconds to 62 minutes. Through common sense, we can assume users didn’t spend anywhere close to an hour on a blog post. Instead, it’s more likely that a user browsed the Internet on another tab while leaving our blog post open on the original tab.

Given the small sample size, we can’t make any conclusions about Polly improving time on page, but we can say the average time on page is formidable, which is another sign of Polly’s positive impact.

And it’s fascinating what a small percent of people who viewed a page used Polly (We had around 8,000 visits to our site a month), which could imply it’s not as useful to users as we thought.

Method 3: Organic overall site assessment

In this method, I assessed the site as a whole for average session duration with a segment applied to filter out any traffic that wasn’t organic. Since we cared about whether Polly improves SEO, we wanted to focus on traffic from search engines. Rather than examining specific blog posts, I compared the entire site’s average sessions duration year over year during the period in which we believe Polly was enabled: February 1st to July 31st.

I found that the average session duration climbed slightly by 5.49% from one minute and seven seconds to one minute and ten seconds. This jump is small, but at least it wasn’t a decline.

You’re probably wondering if this method is accurate since it takes a broad assessment of the site as a whole, which factors in duration from various pages without Polly. And you’d be correct. But the bulk of the traffic on our site comes from blog posts, so we wouldn’t be as worried about service pages or About pages affecting the cleanliness of the data. However, there are blog posts without Polly turned on that do skew the data a bit.

So, why did I use this third method? I wanted yet another measure of Polly’s impact and use on our site. While some uncontrollable factors influence the data with this method, it still gives a ballpark measure of how influential Amazon Polly is in the organic performance of the site as a whole, including the parts that don’t involve Polly. I can still get a general sense of how a tool is working if different metrics are all pointing in a positive direction.

As you can see, I discovered with this method that Polly has a tiny impact on the site as a whole.

Takeaways

When it comes to dwell time, average time on page, or average session duration, it’s important to clarify that longer is not always better for SEO. For product and service pages or blog posts that answer quick questions (e.g., “How tall is The Rock?”), shorter duration is better because the length indicates that users are finding the solution to their problem quickly and experiencing less user frustration. Google is sophisticated and smart enough to recognize these nuances and grade each webpage on a case-by-case basis.

WebMechanix’s blog topics require at least a couple minutes of reading to get most of the value out of them. Therefore, our posts fall more comfortably under the “longer is better” measurement. So, did Polly increase how long users stayed on our pages?

Based on the data from all three methods used, it seems there are mostly positive signs that Polly has encouraged users to stay longer on our site. That said, I cannot conclusively credit Amazon Polly as the cause of the duration changes because of small sample sizes, other factors we can’t control for, and limitations on how precisely we can track users. In addition, the results vary since we saw decreases or only slightly positive increases for some blog posts.

Because of some varying results, I can say that our web player hasn’t provided consistent, clear, dramatic improvements to our user experience across the board. In marketing, we must use the Pareto principle to get the most impact out of our limited resources and time. Based on this point and the fact that Polly’s impact on the site as a whole has been minimal, we don’t think installing Amazon Polly on your blog will have anything close to monumental results. If you do install it, you will probably see some slightly positive improvements. And make sure the player is out of users’ way, so it’s more helpful than intrusive.

Notably, there is little evidence that Polly is harming user experience, so until we see data that proves it is, there is no reason to take Polly off the blog.

I like text-to-speech readers, and I use them frequently to listen to articles because it’s easier than reading them. Based on my gut, Polly has benefitted the blog, because audio offers another popular way to consume content that most blogs don’t have. If you have the budget, hire someone to record the audio so that the blog audio is more natural or read it yourself, which Mark Manson does on his blog.

I’m excited that our company is willing to quickly try new tech to improve user experience—it’s a clear reflection of our innovative nature. If I get enough requests, I can release a follow-up article that dives into the before-and-after once we’ve increased our sample size and analyzed the data through more precise methods, such as measuring by the exact date Polly was turned on for each post based on revision history.

Now, I’ve got some questions for you. Have you considered using Polly on your blog? Have you seen it on other blogs? What do you think of its effectiveness? Let me know in the comments below.

Grab a free case study to discover how simple changes increased conversion rate by 53%.