New Framework for Social Media Measurement – Update & Debate

The complete presentation, including speaker notes, I gave at the AMEC Summit in Madrid on June 7 introducing the AMEC Social Media Valid Framework is now available for download here. Please visit and download the slides if you are interested.

Whenever you throw a new idea or concept out there you really hope people see it. And you really, really hope they care enough about it to comment. Or better still, to challenge it and debate its merits. Simply put, critical thinking makes the concept better. Enter stage left, our provocateur, Philip Sheldrake, author of The Business of Influence. Philip has also been an active thought leader in the whole push toward social media standards through AMEC committees and the work of The Conclave. I respect his opinion very much. Philip wrote a rather long essay on his blog in response to my original post on the new Valid Framework for Social Media. Richard Bagnall, Chairman of the Social Media Measurement committee of AMEC posted a response to some of Philip’s concerns. I would like to address a few others now.  

First, one area I am not going to address in this post is whether or not any framework for social media measurement should be driven by Business Performance Management (BPM) principles and approaches. That subject is being debated very well in the comments under Philip’s original post – pretty interesting thread.

Stakeholders v. Customers v. Audiences

Philip questions if the framework, based on the words chosen to describe it, is oriented more toward customers (i.e. social media marketing) and not the broader concept of stakeholders. The intent was and is the make the framework broad enough to comprehend the majority of use cases for social media. Marketing is just one of the use cases. To be fair, the use of ‘audience’ in the description of Impact is a direct lift from the proposed standards document. We were trying to not reinvent the wheel. That said, I agree with you that saying stakeholders or publics is a more accurate description than ‘audience’ in many cases.

Influence

Hard to argue about influence with the guy who wrote a book on it, so I won’t. Philip makes the point that the model offers a definition of influence (Ability to cause or contribute to a change in opinion or behavior) but does not include the qualifying statements or concepts from the WOMMA Guidebook – the potential to influence (before) and the actual, observed influence (during/after). He goes on to say he prefers emphasis on the second. Fully agree. The short definition was offered for clarity and brevity. We will plan to reword this slightly to broaden. You’ll note the illustrative metrics shown in the framework with metrics are consistent with the during/after (e.g. change in purchase consideration) rather than before orientation. If you think about it, the before piece is really about targeting (who are the influencers we should engage?) rather than measurement (did we change opinion, attitudes or behavior of our target stakeholders?).

Value

Philip makes the point that in the model verbiage, the only reference to Value was financial impact whilst the standards definition reads, “the importance, worth or usefulness of something”. Philip also acknowledges this definition was a “last minute tweak”. It was actually being debated as the slides were being developed. We’ll broaden the definition to include tangible and intangible value to be in lock-step with the standards document.

Advocacy

Philip poses two great questions regarding this phase. His first point is a good model that includes advocacy should also be able to comprehend opposition and advocacy for competing agendas. Fully agree these are important to consider and the model certainly does not preclude that. Comprehending these factors would be addressed by one or more metrics within the framework (e.g. Net Positive Advocacy – positive minus negative advocacy) or simply as a point of analysis rather than measurement per se.

Philip’s second question is simply why Advocacy is shown after Impact. It is shown in this order to recognize the fact that most advocacy actually occurs post impact or conversion event. For example, in a sales context, advocacy generally occurs after someone has bought and had great experiences with the product or service. To be more accurate, we might also show Advocacy after Influence and before Impact. Certainly issue advocacy often happens as a direct outcome of a change in opinion or attitude. Rather than show it in two places, to keep it simple, we choose to show it in the sequence where it is most relevant. In practice one could have both pre and post-impact advocacy metrics in your measurement plan.  

Paid Owned Earned  

Philip simply questions the strategic value of this taxonomy and says it props-up organizational silos and reinforces misconceptions such as PR equals (only) Earned Media. I think the questions are valid ones. But as a practical matter many companies and organizations do use the taxonomy. That is a key reason why it is one of the framework alternatives. And while it can reinforce silos and misperceptions, I have actually seen it have the opposite impact – it recognizes integrated programs require integrated measurement and bring three key elements (different departments and sometimes different agencies) together under a common framework. Interestingly in PR, we now routinely develop programs that include earned, paid and owned elements. This reinforces the positive perception public relations today is not simply equal to earned media.

Impact

Finally, Philip questions whether any framework can consider programmatic-level or channel-specific metrics in terms of Impact. I think it can and does. Impact refers to outcomes. We can have macro and micro conversions. And we can have outcomes that are specific to programs (e.g. event attendance, voter registrations, subscriptions to a content series) or channels (download a whitepaper from the website); although I will agree there will be few channel-specific outcomes.

Keep in mind the metrics shown in the frameworks are illustrative, as Richard pointed out in his remarks, they are not meant to be exhaustive, definitive or recommended. They are illustrative of the emerging standard metrics for social media.

 

Thanks for caring.  

A New Framework for Social Media Metrics and Measurement

Last week in Madrid, AMEC (International Association for the Measurement and Evaluation of Communications) held their 5^th European Summit on Measurement. This one was entitled, Unlocking Business Performance – Communications research and analytics in action. One underlying premise of the program this year was that the time for talk is behind us and the time for action and putting into place the principles and practices of sound measurement is upon us. The later part of the program featured an update from Salience Insight Commercial Director Mike Daniels on social media standards including the recently published standardization effort from the cross-industry group called The Conclave which may be found here.

Once Mike discussed where we are with respect to standardization, Richard Bagnall (@richardbagnall), Chairman of the AMEC Social Media Measurement Group, and I as his vice-chair, presented a session on creating a new recommended framework for social media metrics and measurement. Essentially we tried to answer the question: how do we take the standards work coming from The Conclave and operationalize it to create proper social media measurement? Here is an overview of what we presented and what we are encouraging everyone to adopt and use. The framework templates, usage guide and a short video synopsis will be available for download from the AMEC website, Social Media Measurement section, in the next week or so.

Valid Metrics Framework and Social Media

Our journey begins with the Valid Metrics Framework, a measurement planning framework and template developed under the auspices of AMEC. The framework was designed to be flexible enough to address multiple aspects of public relations within a consistent measurement framework and approach.

Some of the most positive aspects of the Valid Metrics Framework are that it:

• Provides a mechanism to link activities to outputs to outcomes
• Tracks through the familiar sales funnel
• Helps create a focus on outcomes.

One of the applications of the Valid Metrics Framework was for use with social media programs. Two potential issues were surfaced by early adopters of the framework in social programs. The intermediary effect, which in traditional public relations is the impact on the media, seemed at odds with the social world of direct interaction between consumers and brands, and consumers with each other. And use of the marketing sales funnel, while familiar, was only relevant in a percentage of social media use cases and perhaps not the best way to model common uses of social media like customer relations and building relationships with stakeholder groups. Also, thought leaders like Forrester Research and McKinsey & Company had noted the traditional communications funnel was not necessarily funnel-shaped in social media. They described the discovery process that occurs when investigating companies and brands that often cause the consideration set to expand rather than be reduced, and the fact that a lot of engagement around brands happens post-conversion event. For all of these reasons our task was develop and recommend a better framework and approach.

Models and Frameworks

When we use the word model, we are referring to a representation of a system, in this case social media. In the original Valid Metrics Framework, the model used was the traditional sales funnel. A framework adds additional dimensions to the model and is operationalized with metrics. In the Valid Metrics Framework the additional dimensions are the phases – activities, intermediary effects and target audience effects. We looked at both of these aspects individually and collectively when considering alternative approaches.

We studied and evaluated about fifteen different social media and communications models. A couple of common patterns emerged. Several of the models, including Forrester’s Customer Lifecycle and McKinsey’s Customer Journey showed a post-purchase engagement/experience step. We judged this important to include in our recommended approach. And we considered the importance of Engagement and Influence, as two key concepts in social media marketing and measurement, and decided to try to make these two elements explicit in our model as well.

Suggested Social Media Metrics Model

The model we developed is derivative of the categories chosen by The Conclave (Note: Richard Bagnall and I also participate in The Conclave) to organize social media metrics and definitions. We took a slightly different perspective on the front end of the model and reordered the back-end to create this model for our new framework. The descriptions of the stages use the definitions from the smmstandrards.org work wherever possible.

You will note Engagement includes both interactions with owned social channels as well as earned social conversation of people ‘talking about you’ in social channels. The definition of Influence is clear and concise, the result of a lot of discussion and prevailing clear thinking. The concept of Impact includes the outcomes of social initiatives as well as the Value those initiatives created. (I usually advise to always attempt to measure impact, and attribute value when it is feasible and makes sense.) Advocacy includes a very helpful definition and conditions that exist with advocacy.

Creating the Framework

To create the framework, we explored various ways to address the ‘phases’ of the Valid Metrics Framework. Two ideas stood out:

• Use a simple structure that captures social media metrics from three key perspectives – programmatic-level, channel-specific and business. Programmatic metrics are those directly tied to social media objectives. Channel-specific metrics are just that, the metrics that are unique to specific social channels – tweets, RTs, Followers, Likes, Talking About This, Pins, Re-Pins. Business metrics are used to show the business impact of the campaign or initiative.

• Use Paid, Owned, and Earned media metrics for integrated programs containing these elements. Borrowing the definitions from Forrester, Paid are social channels you pay to leverage (e.g. promoted tweets, display ads), Owned are channels you own and control (e.g. website, Facebook page) and Earned is where customers become the channel (e.g. WOM, viral)

There are certainly other ways to think about this (e.g. Business Performance Management) and we intend to possibly add others based on industry feedback and suggestions.

The AMEC Social Media Valid Framework

Currently we have developed both versions with sample metrics (taken from the smmstandards.org work where applicable). We are calling them The AMEC Social Media Valid Framework. Here is the version with Program, Channel and Business Metrics shown.

Where Do We Go From Here?

Look for the completed frameworks on the AMEC website shortly. We encourage you to adopt the frameworks for use by your company or clients. If you like them and find them useful, please help promote them widely. And please provide your feedback on the proposed framework on this blog, or through the social channel of your choice. We’re listening and looking forward to the dialogue.

Time to Get Real About Social Media Audience Reporting

Though almost everyone would agree that social media is about engagement and not eyeballs, too much of digital and social media measurement is focused on audience size. How many Followers do we have? How can we get a million Likes? How many unique visitors did we have to our site this month? And unfortunately, audience size estimates in social media grossly overstate the actual relevant audience. We seem fixated and oriented toward ‘how many’, while our focus should be on ‘who’ and specifically, ‘who within our target audience’. Generally speaking, the advertising industry has led the way with audience measures and is ahead of where the public relations and social media camps are with respect to level of sophistication.

In television advertising, the concept of Target Rating Points is a refinement of Gross Rating Points where you only measure and get ‘credit’ for the percentage of the gross audience that meets your target audience criteria. In an effort to keep refining the audience data available, Nielsen has evolved from diary-based data to electronic data to software at the set-top box level that allows operators to monitor channels choices and changes. In audio-based media, Arbitron’s Portable People Meter recognizes today’s mobile world and begins to address cross-platform measurement. It is also interesting to reflect on the U.S. Congressional involvement in television audience ratings accuracy (or lack thereof as it were) that led to the formation of what is now known as the Media Rating Council in the early 1960’s. The time has come for social media audience research to greatly increase in sophistication, accuracy and relevance.

When we think about social media audience size measures today, the emphasis is on Opportunities To See (OTS), although almost never by this name. We might call them Impressions or Reach, but what we really mean is how many people had the potential to see this content item. There are two overarching issues here:

• Opportunities to see are not the same as actually seeing
• The metrics count all possible members of the audience, regardless of whether or not they are part of the targeted audience or can even buy the product or service.

OTS is also a prevalent metric in the public relations industry which has always focused on stating the highest possible audience measures. In traditional media we know the probability of any one person in the audience actually seeing the article in question is a fraction of the total audience – a reasonable estimate is 10% or less. So OTS greatly overstates the actually number of people who saw a given article. To compound the audience overstatement, we have the practice of using audience multipliers to ‘credit’ earned media for either a perceived credibility advantage over advertising or to account for pass-along circulation (see this IPR white paper for more on multipliers). Thankfully the practice of applying multipliers (and its evil cousin AVEs) is out of favor and rapidly on a path toward extinction.

In social media one can make the case the audience metrics situation is actually exacerbated in that the probability of any one follower seeing any one tweet, for example, is most likely an order of magnitude less than in earned media – my guesstimate is 1% or less. Before you call BS on this guesstimate, play around with a few Twitter factoids – the recent Pew Research study suggesting only 8% of Twitter users use it daily, the perishable nature of most individual’s twitter streams, and the fact that a reasonably high percentage of Followers of a brand are bots, and the reality is that only a small fraction of twitter followers actually see tweets, let alone find it interesting enough to share or comment on. And, of course, not all Facebook Likes see every post you make either. Riffing on the old, ‘if a tree falls in the forest…’, if you tweet into the twitterverse and no one sees it does it make an impact?

Evolving from ‘opportunities to see’ to ‘relevant audience’ measures.

Most social media campaigns have a specific target audience in mind, often described with demographics (Female, age 18 – 34), psychographics (who worry about feeding their family healthy food on a budget) and behavioral (access deal and coupon sites regularly) dimensions. Yet when it comes to reporting and measurement we take credit for the entire audience (total OTS) rather than the percentage of the audience that meets our targeting criteria. Trying to promote lingerie to 22 – 29 year old ladies? No worries, count all your Twitter Followers and all the visitors to your website – the men, the young and the old – everybody counts. Trying to sell camo clothing to male hunters? No worries, everybody counts – male, female, hunters, non-hunters and PETA members, too. Of course this all seems a little silly and strange and I suppose it would be if it wasn’t the way most social audience reporting is done today. It is unusual to see someone in social media, or PR for that matter, report only the relevant audience opportunities to see. Why is this? I believe there are three primary reasons:

1. Legacy – the PR industry has historically reported gross potential audience size rather than the relevant audience size. When social media came around, this same orientation toward gross audience measurement was used.
2. Data – there is a lack of consistent social media demographic and psychographic audience data available and it often resides in channel silos rather than cross-platforms. And often the audience data from one platform (e.g. ComScore) does not match the data available from another platform (e.g. Compete).
3. Standards – there are no standards for social media audience metrics and no codified best practices for audience measurement.

Where do we go from here?

First, we need a change in mindset of how we think about audiences. From ‘how many people theoretically had the potential to see our content’ to ‘how many of the people we were targeting actually saw our content’. Big audience numbers are irrelevant. Relevant audience numbers are big.

Next, as the demand for audience data that contains demographic, psychographic and behavioral data grows, it is reasonable to assume one or more of the large media data companies might start to aggregate and make the data available. Privacy concerns, cookies and other issues are also in play here.

And last but not least, industry standards for social media audience and engagement metrics and definitions are necessary for transparency and replicability that will increase credibility of social media measurement and reporting. 2012 will go down as the year that serious cross-industry progress on social media metrics standards began and gained momentum. There has already been a lot of progress (See this post from Katie Paine), and this week in Dublin at the 4^th AMEC European Summit on Measurement the theme is around attempting to define standards for social media metrics and measurement. To tune into the debate as it occurs in Dublin, monitor #SMMStandards and #AMEC2012.

What  are your thoughts on the need for social media metrics standards and the use of target rather than gross audience size estimates?