Is Your Measurement Methodology Monkey-Proof?

Beware sudden changes to algorithms or source material that can throw a monkey wrench in your analysis or give you a false sense of achievement.

Jesper AndersenLast Friday, Facebook announced that starting this week, the company will only count viewable impressions for pages’ organic reach. The move will make organic reach more easily comparable to Facebook’s ad reach, which has always been calculated using viewable-only impressions.Up until now, organic reach has been calculated using the number of times a piece of content was loaded into a user’s news feed. But the content did not have to appear on an actual screen to be counted as an impression.The consequences of Facebook’s changes, according to social media professionals I have spoken with, could be a drop in the reported organic reach of Facebook pages by as much as 20 percent, but will likely wary greatly from page to page.Following Facebook’s latest changes to their algorithm, which deprioritised branded content in favour of interactions with family and friends on the platform, this latest news could be seen as adding insult to injury. However, it is important to remember that Facebook’s new calculation of organic reach is actually more accurate and that the old inflated reach numbers were never ‘true’.

Adapting your measurement methodology to the ‘X factor’

The recent changes to Facebook’s algorithm and the way it calculates organic reach is a perfect example of the dilemma that constantly faces communication professionals, when we do measurement and analysis.If we want to accurately benchmark our work and its results over time, we need to take into account not just our own efforts but also the surrounding circumstances. These are elements, which may be beyond our control but whose impact we nonetheless must include in our analysis to get an accurate result.Some typical examples include:

• The volume of your media coverage is increasing nicely despite no extra effort on your part. It turns out to be the result of a gradual increase in the number of media outlets being monitored by your service provider.
• The number of readers you reach with your print media coverage is dropping, despite the same number of articles as the year before. The explanation could be an actual drop in print readership or print media outlets going out of business.
• Changes in your social media reach or engagement can be the result of algorithms changing.
• A drop in your share-of-voice can be caused by your competitor getting embroiled in a crisis or shitstorm, generating a lot of negative coverage, and so is not necessarily a bad thing for you.

4 key questions to ask yourself

When analysing your data, before you draw any conclusions, always ask yourself these questions:

1. In what ways have the fundamentals changed compared with the last period we are benchmarking against? Are we using data based on the same source material (e.g. list of media outlets)? Have we experienced a noticeable increase or decrease in social media following?
2. Are we aware of any significant changes to the media eco-system that may help explain what we are seeing in our analysis? For instance, the introduction of an online pay-wall causing readers to drift to other (free) news outlets? Or changes to social media algorithms that affect the calculation of user statistics?
3. Have any competitors entered or left our market and upset the previous status quo? In what way?
4. What are the major public agendas currently (think #MeToo, Fake News, climate change, Brexit etc.) and could they be having an impact on our results, good or bad? How do we allow for these issues when benchmarking one period against another?

Once you take these questions into account, you will have a much better understanding of how different factors may or may not have skewed your analysis, leaving you better able to determine the true outcome and impact of your communication.  [author]About the Author: Jesper Andersen is a strategy advisor and international keynote speaker specialised in communications measurement. He runs his own consultancy, Quantum PR Measurement. [/author]