Tag Archives: social media metrics

Measurement 2020 and Other Fantasies

23 Sep

At the 3rd European Summit on Measurement held in Lisbon in June 2011, standardization, education, ROI and measurement ubiquity emerged as the key themes in response to a call to set the Measurement Agenda 2020.  Delegates to the conference voted on 12 priorities they thought were most important to focus on in the period leading up to 2020.  The top four vote-getters became the Measurement Agenda 2020:

  1. How to measure the return on investment of public relations (89%)
  2. Create and adopt global standards for social media measurement (83%)
  3. Measurement of PR campaigns and programs needs to become an intrinsic part of the PR toolkit (73%)
  4. Institute a client education program such that clients insist on measurement of outputs, outcomes and business results from PR programs (61%)

For a very nice overview of the Lisbon session and the Barcelona Principles that came before, read this post from Dr. David Rockland of Ketchum who chaired the Barcelona and Lisbon sessions.  David pretty much said it all on these sessions, so I’ll just add a couple of comments and share a few thoughts on what I believe the future of measurement 2020 could be.

The rallying cry coming out of Barcelona has been focused and loud – death to AVEs!  Will there be a similar thematic coming out of Lisbon and what might it be?  My money is on standardization, borne out of cross-industry cooperation.  As David points out in his post, and in the words of AMEC Chairman Mike Daniels, “The Summit identified some significant challenges for the PR profession to address by 2020.  However, what we also accomplished in Lisbon beyond setting the priorities was to harness the commitment and energy of the industry to agree what we need to do together.”  The current cooperation and collaboration between industry groups – AMEC, Institute for Public Relations, PRSA and the Council of PR Firms is unprecedented in my time in this industry and is focused on tangible outcomes.  Cross-organization committees are already at work developing standard metrics for social media measurement for example.  The spirit of cooperation is uplifting.  While the outward thematic appears to be standardization, cooperation is the enabling force.  

I was also struck by the symmetry of the call to end AVEs in Barcelona and the call to codify ways to measure ROI in Lisbon.  One follows the other.  In my opinion the primary reason AVEs exist is because PR practitioners feel pressure to prove the value of what they do, and quite often they are asked to describe the impact in financial terms.  AVEs are perceived as a path of least resistance way to express financial value.  Except, as we all know, AVEs don’t really have anything to do with the impact public relations creates.  They are a misguided proxy for financial value.  Hence the need for research-based methods to determine true return on investment.

All of the priorities coming out of Lisbon are excellent goals for the industry.  And like David Rockland, I believe they will be achieved, and be achieved before 2020.  Here are three other items on my wish list for Measurement 2020:

Word of Mouth/Word of Mouse Integration: For those of us focused in social media and other digital technologies, we can’t allow our digital lens to color what is fundamentally an analog world.  Research studies suggest the majority of word of mouth happens in real life.  From an influence perspective, I don’t think too many would argue that word of mouth from a trusted friend or family member is more powerful than word of mouse from someone you follow on Twitter.  Digital cross-platform research is difficult enough, but when one huge platform is ‘real life’, we have significant challenges in measurement.  WOMMA and others have made early attempts to define measurement approaches for offline WOM, but much work remains.  We need ways to assess its impact and then we need to think about ways to attribute value to that impact.  Mobile is a wild card here as it becomes the preferred platform for online activity.  The need to triangulate online, mobile and ‘real life’ measurement presents significant challenges today, and may still by 2020.

Cookie Wars: We all know the measurement versus privacy showdown is coming, right?  The first shots have already been fired.  The collection of source-level personal data, enabled by cookies, is crucial to measurement and insights but has the potential for misuse or unintended disclosure.  Some sophisticated consumers have had their fill of cookies.  Although the broader issue might be framed as social sharing versus privacy control, how it plays out will have a direct impact on digital analytics and measurement.

Integrated Measurement across the Paid Earned Shared Owned (PESO) Spectrum: Measurement has increasingly become integrated.  It began with integrated traditional (Earned) and social media (Shared) measurement and then progressed rapidly to Earned, Owned and Shared, which is where most integrated measurement programs are today.  Many leading-edge integrated programs today also include advertising or Paid media.  By 2020, integrated measurement across the PESO spectrum will most likely be the norm and not the exception.  A key enabling element here in my view is some base level of agreement on how each area should be measured and standard metrics for each.  It will take significant cooperation between industry groups, vendors, agencies and major customers/clients for cross-discipline standardization to move forward effectively.  We are at the beginning of this movement in 2011.  By 2020, it will be fascinating to look back and see how all this plays out.

When looking ahead to 2020, I am reminded of a measurement discussion pulled together by PRWeek a couple of years ago.  Many of the Measurati attended.  In response to a question of where measurement will be in five years, David Rockland replied (paraphrasing here), ‘Who knows?  Five years ago who would have guessed we would all be focused on how to measure social media?’  So, there is a certain fantasy element to discussing 2020 challenges in measurement.  What are your measurement fantasies?

Don’t Let the Tool Tail Wag the Measurement Dog

19 Jul

Social media listening and measurement tools are sexy.  Well, at least to those of us in research and measurement – it’s all relative right?  In the last three years or so there has been an explosion of social media tool vendors and platform choices.  Tools are sexy and important, but in the grand scheme of things are being overemphasized to some degree.  We are letting tools decide what we can measure without giving sufficient thought to what we should measure.  We are letting the tool tail wag the measurement dog.

There are several steps and decisions that should be addressed prior to selecting a tool or suite of tools.  Consider this diagram as a starting point to help you think through these interim considerations and decisions:

OBJECTIVES

Proper social media objectives should be measurable (indicate change in metric of interest and timeframe) and aligned with desired organizational outcomes.  Understanding the social media objectives will suggest broad parameters the measurement program, and ultimately the tool decision, must operate within.  For example, geographic coverage requirements, type of content to be considered and on-platform engagement capability may all be strongly suggested based on a review of social media objectives.

PROCESS

In addition to comprehending organizational or business outcomes, it is essential to understand the business process the social media program will address or drive.  If the program is marketing oriented, the sales funnel process (Awareness/Consideration/Preference/Sales/Loyalty) may be most appropriate.  For a brand-building campaign, the brand pyramid (Presence/Relevance/Performance/Advantage/Bonding) is what you want to measure your program impact against.  Other business processes that are commonly addressed by social media programs include customer service and support, CRM, corporate reputation and lead generation.

METRICS

Understanding the requisite business process the social media program is driving is crucial because each business process drives specific metrics.  For example, the sales funnel drives a specific metrics set:  percentage of unaided or aided awareness; percentage of the target audience who would consider the product/company; percentage who prefer the product/company; incremental sales revenues; percentage who would purchase the product again number or the number/amount of repeat purchases.  For B2B companies, the lead generation process would drive a different set of metrics: number of incoming leads; percentage/number of qualified leads; lead conversion rate; sales revenues generated.  In addition to the business process metric sets, there are other metrics areas like Exposure and Engagement we will want to address.  Reach/opportunities to see, share of positive discussion, comments/post ratio, number of @ mentions and RTs per 1000 followers are examples of ‘standard’ metrics that might be applicable for many social media programs.

Understanding how the social media program drives a specific business process is also important to our ability to describe the impact or, in some cases, return on investment the program has created.

DATA SETS

Each metric has data requirements, usually two pieces of data per metric – a numerator and a denominator.  Examine the set of metrics you have defined for your social media program.  Catalog all the specific pieces of data you need to compute the various metrics.  For example, the data needed to compute the basic sales funnel metrics and some ‘standard’ metrics might include:

  • Number of individuals in the target audience
  • Number of survey respondents
  • Number of respondents ‘aware’ of the product/company
  • Number of respondents who would consider/seriously consider purchasing the product/doing business with the company
  • Number of respondents purchasing the product
  • Amount of sales revenue directly attributable to the program
  • Number of purchasers who purchased again
  • Total branded mentions
  • Volume of positive and negative mentions
  • Number of posts
  • Number of comments
  • Number of RTs and @ mentions
  • Number of followers

TOOLS

Armed with an understanding of all the data needed to calculate the metrics required to measure the social media program, you will be able to assess which tools or classes of tools best deliver the data you need.  Pick the best three to five tools for further evaluation.  You most likely will find no one tool can deliver the complete data set you need.  It is common to need two or more tools, e.g. web analytics package and social content analysis platform, in order to fully meet data requirements.  Budgetary constraints may also limit your ability to capture the entire data set required.

By addressing the interim steps leading up to tool selection, you will be able to make a more informed tool decision.  You also will have a much better chance of measuring what you should measure rather than settling for what you can measure.  No tool before its time.  Let the big dogs run.

Inflationary Twitter Audience Numbers Hurt Social Media Credibility

6 Jul

In yesterday’s New York Times, you may have read the article, Spinning the Web: P.R. in Silicon Valley, an interesting although not overly insightful piece.  From a social media measurement perspective, two items caught my eye.  The first, referring to Brian Solis, Principal of FutureWorks, about how he calculates social media audience figures:

“Instead of calculating the impressions an article gets by estimating a publication’s circulation and pass-along rate, Mr. Solis counts the number of people who tweeted about a company and their combined following, the number of retweets or clicks on links, as well as traffic from Facebook and other social networks.”

Toward the end of the article, we learn:

“By 6:30 p.m. on the day Wordnik went live, Brew’s staff calculated that 1.43 million people had seen tweets about it.”

Setting aside for a moment that the article and these sorts of audience metrics take a broadcast-oriented view of Twitter (Mr. Solis discusses the shortcomings of the NYT viewpoint here), the emerging view of audience measures for Twitter is to calculate the Followers of each person tweeting about the subject of interest, and then adding Follower numbers for each person retweeting the subject and so on.  The issue here, much as it is in traditional public relations, is that the audience figure that results from these sorts of calculations grossly overstates, by one or two orders of magnitude or more, the actual “audience” for these tweets.  It is a hypothetical number that assumes everyone that possibly could see a tweet has in fact seen it, and everyone who sees it is relevant to you/your brand.  This is fantasy of course.421922_p~3d-Cinema-Audience-Posters-763348

On the issue of relevant audience, here is a quick example.  At the time I pulled these figures, the audited circulation of the New York Times was 4,974,000.  Most PR practitioners getting a ‘hit’ in the NYT would claim this as their audience.  However:

  • If you were only trying to reach a C-Suite audience with your message, the actual audience reached would be 598,000 or 12% of the total circulation
  • If you were trying to reach Women, your audience would be 1,937,000 or 39% of the total
  • If you were trying to reach 25 – 54 year old Men, your potential audience would have been 2,930,000, or 59% of the total number.

There is a large difference between how many people theoretically can see a tweet, versus how many actually saw it/read it, versus how many of those seeing the tweet find it relevant to them, versus how many engaged with it by hitting a link or retweeting.  Part of my issue with this is the language we use to report the figures.  For the Brew staff to use these numbers to estimate 1.43 million people “had seen tweets about it” is wrong.  If they had said 1.43 million people had an opportunity to see the tweet, it would have been more realistic, although still greatly overstating actual relevant audience.

This problem of audience inflation has already been institutionalized in public relations.  The use of Impressions as an output metric does not mean a true impression in the branding sense, but rather an opportunity to see the content.  To make matters worse, many PR practitioners believe Impressions should be factored by either dubious pass-along readership figures and/or use of a multiplier to account for the mythical credibility advantage PR enjoys over impressions generated from advertising.  The simple fact is there is no research-supported, fact-based argument for using any adder or multiplier in public relations when calculating potential audience (here’s an IPR white paper on this subject I co-authored).

For Twitter and other social networks we lack demographics and data about tweet readership averages (i.e. what is the probability that any one tweet is actually read) that would allow for more precise audience estimates.  In the absence of data, believable assumptions should be used:

  • Out of all the opportunities to see, how many actually read the tweet?  10%?
  • Of those reading the tweet, how many find it relevant to them (or from the other perspective, how many of the readers are in your intended target audience)?  Maybe 10% again?

You can see how our audience estimate has already been reduced by a factor of 100.  This may well still be overstating the actual, relevant audience.  The issue here is that unrealistic and overstated audience figures have the potential to hurt credibility and call into question other data and metrics that may be more grounded in fact.  Actually the more meaningful metrics pertain to engagement or outcomes rather than exposure/outputs.  It is more meaningful that 40,000 visited the Wordnik website as a result of the campaign discussed in the NYT article than the overstated 1.43 million who were estimated to have seen the tweets.  40,000 is real.  1.43 million is fantasy.

A New Model for Social (and traditional) Media Measurement

29 Aug

In November 2007 I suggested the current Outputs, Outtakes, Outcomes model and taxonomy for public relations measurement was confusing and therefore often misunderstood and misapplied (Let’s put Outputs, Outtakes and Outcomes in the Outhouse).  At that time I suggested a simpler, more descriptive approach was in order and offered the following:

“What we need is a metrics taxonomy that is easier to understand and explain.  Perhaps simple and descriptive enough that we could skip the need for explanation altogether.   I propose the following three terms:
* Exposure – to what degree have we created exposure to materials and message?
* Influence – the degree to which exposure has influenced perceptions and attitudes
* Action – as a result of the public relations effort, what actions if any has the target taken?”

Since November, I have given a lot of thought to the E-I-A construct and how to improve upon it.   Some of the feedback to the model was the gap between Exposure and Influence was too great, and perhaps there should be an interim step called Understanding or Relevance.  There is also the social media dynamic to consider since the measurement model should be flexible enough to work for both traditional and social media.

What seems to fit best between Exposure and Influence, and adds richness to social media measurement, is the concept of Engagement.  Not only is it one of the hotter topics in social media, it is consistent with the desire to have more descriptive and easily understood metrics.   With Engagement we now have an category that nicely contains such emerging key metrics as view-thrus, duration spent with content, repeat commenters and comments/posts ratio.  It also works well for old school metrics like recall and retention.  Engagement is what helps set the stage for Influence to occur.  Engagement is necessary for communities to form.  Engagement is fundamental to brand.

Here’s a graphic that shows the new model and sample metrics that might be used at each stage:

Would love your feedback on this new Exposure — Engagement — Influence — Action model.

There are still a few challenges in adoption of the model, not the least of which is that there is no consistent definition of Engagement.  Current definitions range from the simple comments to post ratio used by BusinessWeek in their Reader Engagement Index, to the 8-term formula for Engagement offered by Eric T Peterson.  The next year should bring more clarity and consistency to our understanding and use of Engagement.  At least there is modest agreement on the specific metrics contained within the category of Engagement.

Thanks for reading, Don B