I was honored this year to be asked by Paul Holmes to serve as a judge for the Sabre Awards.  It was my first judging experience and I found it very rewarding and learned a lot.  Across the five categories I judged there was a lot of great work by obviously talented professionals.  With so much great work, it often comes down to which entry does the best job of demonstrating, through an effective measurement effort,  stated objectives were met or exceeded.  While there were good examples of this practice, there were many more submissions that simply failed to demonstrate the true success of their impressive campaigns. 

Here are the three most common oversights of the non-winning entries I reviewed: 

1. Objectives Not Measurable - The majority of the stated objectives in the entries, as written, were not measurable.  One cannot measure, ‘Increase Awareness’ or ‘Generate Coverage’.  One could measure, ‘Increase Awareness From 10% to 25% in the Next 12 Months’ or ‘Generate 1,000,000 Impressions in the First 6 Months of the Campaign’.  (See my previous diatribes on measurable objectives here) 
2. Strategies Masquerading as Objectives - If objectives are ‘what’ we want to accomplish, then strategies are ‘how’.  Sentences beginning with action words like, ‘leverage’, ‘educate’, ‘promote’ or ‘communicate’ are almost always a strategy and not an objective.  Also, media coverage is almost always a strategy and not the objective.  The vast majority of award entries had one or more strategies posing as objectives.
3. Measurement Misaligned With Objectives- By misaligned I am referring to an objective that is an outcome (or influence) supported by measurement of only outputs (or exposure).  If we are trying to create awareness or change an opinion, we can not demonstrate success by only reporting on the number of impressions generated.  Great programs articulate the desired business outcomes, write PR objectives aligned with these outcomes, and then report on the metrics directly tied to the PR objectives.   

In summary, many of the entries made the hard stuff (great creativity and execution) look easy, and the easy stuff (writing proper objectives, measuring the correct metrics) look hard.  Better than the other way around I guess.  And lots of room for improvement.

Question:  You are about to go into a meeting with the C-suite members of your company to report on the success of your PR campaign.  Would you rather be able to say:

A.)  The campaign is working well, we have generated over 12 million impressions    

or

B.)  The campaign is working well, we have reached 91% of our target audience an average of 4.2 times.  

OK, so it would have been best to be able to report a direct causal link between PR activities and sales with an estimated ROI of 250%, but, between the two examples above, most of us would rather have reach and frequency data than just impressions.  Reach is typically expressed as a percentage of the audience and therefore requires the practitioner to know the total audience size and how many of the audience were ‘reached’ with a given combination of articles.  You also need to understand media consumption dynamics.  In the advertising world this sort of analysis is common and expected.

The power of reach is that it provides a much more realistic estimate of possible impact than impressions.  For example, if you are targeting women and get a hit in the New York Times, most PR pros today would lay claim to nearly 5 million impressions.  Some, citing pass-along readership and/or a mythical PR credibility advantage, might even inflate this number by a factor of three and claim almost 15 million impressions (For additional discussion on the use of multipliers, please see this IPR White Paper) .  In fact, you have reached a little less than 2 million women with the New York Times hit.  Reach paints a more realistic picture.

In the UK, Metrica purports to have reach and frequency data gleaned from a primary study of 12,000 consumers.  They use it as a standard metric in their bespoke media content analysis offering.  So why don’t any of the U.S.-based media content analysis firms offer reach and frequency metrics?  The database development would be expensive but uptake most likely would be strong. 

While both impressions and reach only represent opportunities to see (OTC), at least the reach approach requires the potential reader to be someone who could actually buy your product or service.  Baby steps of progress.  Next, perhaps we’ll start to measure people who actually saw an article and took the time to read it.  Maybe we’ll use the word du jour and call these ‘authentic impressions’.  

As always, thanks for reading.  -Don B  

Look for 2008 to be a year of increasing accountability for the public relations profession.   Add in the very real possibility of an economic recession and, more than ever, the pressure will be on to prove the value of public relations. 

1.  Focus on efforts to measure the influence of blogs, social media and other online properties

There is a high level of agreement that blogs, discussion groups, social media and other online properties are shaping the way consumers learn and think about products and brands.   First generation attempts to measure their influence on consumers have been done using predominately traditional media metrics like Impressions.  In 2008 expect to see a stronger push toward specialized metrics to measure online engagement/dialogue rather than just exposure.  

2.  Ongoing shift  of PR measurement budgets towards Outcomes rather than Outputs

Historically the majority of public relations dollars have been spent measuring Outputs (Exposure) rather than Outcomes (Influence and Action).  Studies suggest the split has been perhaps 60%/40%.  Expect 2008 to be the first year where the split is closer to 50%/50%.

3.  Renewed efforts by content providers to reconcile copyright issues

Many of you may have recently received a letter from Burrelles/Luce notifying you as of April 2008 they will be adding copyright royalty fees automatically on behalf of content owners based on the average numbers of copies of print or electronic clips distributed.  Expect other content aggregators to follow suit in 2008.  

4.  Increased use of Marketing Mix Modeling for ROI determination

Procter and Gamble’s proactive outreach about their Marketing Mix Modeling (MMM) project (read more here) sparked a lot of industry debate in 2007 (and we suspect projects for modeling vendors).  The new Da Vinci agency being formed for Dell has already stated their ambition to combine art and science to tie marketing activities directly to business outcomes like sales and market share.  Look for MMM to be part of the science used by Da Vinci and for it to become mainstreamed in 2008.

5.  Additional efforts to find a ‘single metric’ measurement solution

For many in public relations measurement the Holy Grail is a single, powerful metric of success.  Microsoft took this approach with their ambitious measurement system development project last year as did the Canadian Public Relations Society (read more here)before them. Perhaps Da Vinci will also be seduced by the allure of the single metric.

Generally speaking, advertising delivers a lower ROI than public relations.  So why do we want to compare our results to those of an ad?  Because it is a path of least resistance to calculating ROI, flawed as it is.  Many people obviously believe a poor metric for ROI is better than none at all.

So, in the spirit of the 12 Days of Christmas, here are the 12 reasons why AVEs are a poor metric for public relations.  And one rationale for using them that makes at least a little sense.      

1. Advertisements and editorial articles are perceived differently by receivers/readers.  Editorial material benefits from the credibility of a third-party (the publication) by earning, not paying, its way into the magazine, newspaper or broadcast

2. AVEs equate an article with the appearance cost of an advertisement.  It does not speak at all to the results or impact that the article may have on a reader.  Advertisers do not judge the success of advertising on how much the insertions cost.  The true value of an ad or article is in what it does – the outcome or impact, not the cost of appearance.

3. AVEs do not address the value of several important aspects of public relations including strategic counsel, crisis communications, grassroots, viral campaigns or public affairs.  In other words, AVEs reduce PR to just the media dimension by only assigning a value in this area.  If only AVEs are used to assess PR value, the results will be much understated when considering the totality of value delivered by PR.

4. AVEs cannot measure the value of keeping a client with potentially negative news (e.g. layoff, scandal) out of the media, yet that may be the primary objective of the PR practitioner.  How much is it worth for a troubled company to notappear in the Wall Street Journal?  AVEs cannot address this.

5. Impression information for public relations is somewhat inconsistent.  Online impressions figures are not as reliable as print or broadcast, and are generally believed to be overstated  The fact that they are inflated skews AVE calculations to (pick one: somewhat, very, grossly) overstate the value of online media, often assigning unbelievable values to online articles compared to their print counterparts.  This hurts credibility and believability.

6. AVEs do not properly distinguish between hits/articles that appear in ‘high value’ columns or publications and articles in more general or generic publications.  The calculation is based on ad cost only.  The value of appearing in a Walt Mossberg column in the WSJ or on Oprah with your new book far exceeds the cost of an advertisement in the WSJ or on Oprah due to the implied or explicit endorsement with earned media.  Just look at what Oprah’s endorsement has done for Obama recently.

7. Advertising and PR actually work together synergistically, yet AVEs treat them essentially as equals or alternatives.  Ads that run in a climate of positive publicity actually receive lift from the PR.  Conversely, ads run in an environment of negative publicity will likely not be successful and/or may be perceived negatively by consumers/customers.

8. AVEs are generally calculated by mainly, or only, taking into account the physical size of the article, and then equating that to the cost/value of an advertisement of the same size.  Often, article valence is not even considered, so a predominantly negative article would add positively to the overall AVE calculation.  Others count the size of the entire article, even if only one paragraph directly addresses the company in question.

9. Some groups have devised their own ways to calculate AVEs.  PR articles are generally rated or scored as part of an algorithm used to calculate AVEs.  Factors considered might include brand prominence within the article, competitive mentions, overall article tonality and finally size/length of the article.  The problem here is there is no standard way to ‘score’ PR articles to implement an AVE system.

10. If you get a hit on the front page of a newspaper or a cover story in a magazine, there is no way to calculate an advertising equivalency since advertising space is never sold in these locations 

11. AVE results can be misleading.  AVEs may be trending up while important metrics like message communication, share of favorable positioning and share of positive press are falling.  Objects may appear larger than they really are.

12.  AVEs only apply to traditional media.  What is the AVE of a positive conversation about your company on a leading blog? 

If you are still searching for a rationale for using AVEs, there is one that has some merit IMHO.  That is the economic argument that advertising rates are established in a free market system.  Publishers can only charge what advertisers are willing to pay for a page in their publication.  Essentially the value of that page is established by this free market system.  Of course what appears on this page is not considered in the value determination.  I would suggest one refer to the value of a page determined in this way as the ‘Media Value’ rather than the Advertising Value.  By using the term Media Value, one eliminates the uncomfortable and unjustified comparison of an article generated by PR with an advertisement.  You are merely suggesting the page has a value and you use that value to determine how much the public relations content is therefore ‘worth’.  Media Value is still not a great way to assess the value generated by public relations for many of the reasons stated previously, but it seems to me to be less objectionable than AVEs.  

(Note: Much of this commentary about AVEs appeared in earlier posts and comments in this blog.  You can see them here)

As always, thanks for reading.  May the New Year bring you and yours happiness, health and prosperity.  -Don B

Did you ever have an idea and think that perhaps it is a ‘big idea’?  You decide to sleep on it to see how it sounds in the cold light of morning.  You wake up and realize, well this isn’t really a big idea.  In fact, maybe it’s just a nice idea.  Heaven forbid, some may even think it is a dumb idea!  Your excitement of the prior evening has been replaced by a flood of doubt.  That was my experience in thinking about a concept I’ll call Public Relations Residual Value (PRrv).

The concept is simple enough.  Public relations materials - releases, B-roll, product reviews, articles, blog postings, etc. - have a value beyond the transactional value that occurs at the time of their appearance.  By being archived on websites that are searchable, these materials have a shelf life or longer-term value that transcends the shorter-term transactional value.  The value is related to the degree to which the collection of PR materials form a database that may be accessed by search engines like Google, and the contribution the materials make toward search positioning on the specific search sites.

Various studies have shown the majority of traditional journalists begin their story with a Google search.  We also have ample anecdotal evidence that customers and consumers use search extensively to gather information and do comparative analysis of products and services.  Clearly, having positive PR materials among the top results for a given search has value, although it may be problematic to assign a specific value to this.

Public Relations Residual Value (PRrv) is certainly not a primary component of the value generated by public relations.  But, if one is attempting to examine the total value generated by PR - tangible and intangible, short-term and longer-term - then it may be worth considering.

Big, nice or dumb idea is for you to decide.  As always, your comments are very welcome and appreciated. 

Thanks for reading, Don B 

For additional viewpoints on the Output/Outtakes/Outcomes debate, see:

Jim Nail’s post here and

Dr. Tom Watson’s post here

In October I attended the 5th. annual Summit on Measurement sponsored by the IPR.  One of the speakers showed a chart listing public relations Outputs and Outcomes.  He listed press releases as an Output rather than Impressions, Number of Hits, Message Pick-up or any of the other metrics correctly referred to as Outputs (See the Dictionary of Public Relations Measurement and Research here).  Why bring this minor transgression up?  Because this is not an isolated occurrence.  Many public relations practitioners, even senior people, regularly have to pause for a moment to make sure they don’t get their Outputs confused with their Outtakes or Outcomes.  Outtakes is not often used in the U.S., it seems much more prevalent in Europe.  The terminology is confusing and is defined in different ways by different practitioners.  Further compounding the confusion is the fact the audiences we present our results or requests to rarely understand the terms and have trouble relating to them.  In short, the terms are too much ‘inside baseball’.   

What we need is a metrics taxonomy that is easier to understand and explain.  Perhaps simple and descriptive enough that we could skip the need for explanation altogether.   I propose the following three terms:

• Exposure - to what degree have we created exposure to materials and message?
• Influence - the degree to which exposure has influenced perceptions and attitudes
• Action - as a result of the public relations effort, what actions if any has the target taken?

The E-I-A construct is easy to understand and does a reasonable job of describing what we are trying to accomplish in public relations.  Here is a graphic that brings it to life a bit.

EIA graphic

There are lot of possible answers to this problem.  EIA may be one of them.  It would be great to hear whether or not you share the view that Outputs/Outtakes/Outcomes is problematic and what solutions you might offer in response.

As always, thanks for reading.  -Don B  

The recent debate on relationship measurement in this blog (original post and comments here) was thought provoking to me and I hope to you as well.  I believe there are at least three major areas of contention expressed or implied in the commentary:

Theory versus Practice

One of the differences in opinion I believe may be attributed to the philosophic view that, “Public relations is the organizational function responsible for relationships” according to Dr. Grunig, and the practical reality of how public relations is currently viewed by practitioners and the companies and agencies for whom they work.  The practical reality is the public relations function today is rarely viewed as responsible for relationships in a broad organizational sense.  The concept to me seems simultaneously too big and too limiting.  

As a practical matter, marketing or brand managers or customer service/care executives are more often held responsible for customer relationships, not the PR team.  The responsibility for employee relationships more often lies in the human resources department than it does public relations.  I could go on but you get the point.  There is a large disconnect between the idealized organizational relationship model and the reality of the corporate world today.  

Semantics

Many organizations today already perform various forms of research and measurement that might be categorized as relationship measurement.  Three quick examples:

• Voice of the Customer (VOC) and variants are research techniques used to involve customers and prospects into the product or process design.  A mechanism to have their voices heard before rather than after the fact
• Brand Studies are regularly used to evaluate and measure the strength of the relationship between consumers and a product or company.  Trust and emotional attachment are key concepts here.
• Net Promoter Index and other techniques used to assess the strength of relationships and satisfaction levels between customers and an organization

Level of Abstraction

Building on the first topic, I would pose the question; do we want to stake the future of the PR function on relationships?  Is that our highest and best use within organizations?   Clearly between media fragmentation and social media proliferation, the current focus of many practitioners on media relations is a path akin to that formerly taken by the dinosaur.  I believe the answer depends on the level of abstraction you want to apply.  Three different levels come to mind:

•  Relationships

• Influence

• Advocacy

A few thought leaders have organized their thinking around the concept of advocacy.  Pragmatists might argue the role of the profession is really about creating and facilitating influence - we create exposure and ultimately attempt to influence our constituents.  Both of these concepts to my way of thinking are a higher level of abstraction than relationships.  One might try to build relationships to increase influence.  Or one might use relationship building tactics to help create advocates for the brand or company.

The correct answer to this industry issue is clearly above my pay grade, but the debate is fascinating and important.  Thanks for participating. 

 –Don B     

There appears to be a groundswell of support among the measurement cognoscente (a.k.a. metricsexuals*) for the concept of Relationship Measurement.  Much of the early work in Relationship Measurement was done by Dr. James Grunig and Dr. Linda Hon, as published in their seminal 1999 paper; “Guidelines for Measuring Relationships in Public Relations” (get a copy here).  The basic premise of Relationship Measurement is that public relations is fundamentally about managing the relationship between an organization or company and all its stakeholders – employees, communities, customers, partners, etc.  Dr. Grunig and Dr. Hon identified six factors – four characteristics of relationships and two types of relationships - that collectively are highly correlated to relationship strength: 

1. Control Mutuality
2. Trust
3. Satisfaction
4. Commitment
5. Exchange Relationship
6. Communal Relationship

 In my opinion, Relationship Measurement is an interesting concept that has a place in the public relations measurement mix, primarily as a diagnostic tool.  It is not, however, a panacea nor should it be considered the ‘next big thing’ in measurement.  Here’s why: 

• With Relationship Measurement you are measuring the strategy (how) and not the objective (what).  A company wants better relationships with employees because that probably means their employee retention rates are higher, or their talent acquisition costs are lower, or perhaps because satisfied employees provide better customer service.  They want better relationships with customers because they may buy more products or services and are more loyal.  Relationships are the means to the end perhaps, but not the desired outcome. 

• There is no proof (yet) that better relationships lead directly to better business outcomes.  While it may be intuitively obvious to many of you that having better relationships is good for the business, the research necessary to demonstrate this has not be done in any broad-scale way of which I am aware.      

• It is difficult to demonstrate the value to the organization of public relations if we are focused on measuring relationships.  If you go into a CEO’s office and he/she asks how public relations is doing this year and you respond, “Great, relationship strength is up an average of 12% across the board”, you will get a very blank look in response.  Like it or not, public relations must demonstrate how it is helping to drive desired business outcomes if it is to be considered a core strategic function within the organization.   

In the coming months I’m quite sure you will hear more and more about Relationship Measurement.  Read the articles and the books.  Experiment with it and see if it meets your needs.  But please, can we keep a little perspective about what it is and what it is not? Thanks for reading.  I welcome your contrary viewpoints.  –Don B  

* props to Katie Delahaye Paine for this term (see Katie’s blog here)  

I’m sure many of you have followed the spate of consolidation that is sweeping our industry.  Echo Research (research firm) has acquired KWHR Network (measurement firm), which had a global reseller agreement with PRNewsWire (wire/distribution service).  Bacon’s (content aggregator) bought Delhaye (measurement firm) and rebranded as Cision (silly name from expensive branding process/firm).  TNS Media Intelligence (advertising services) bought Cymfony (measurement firm) and cleverly renamed it TNS Media Intelligence/Cymfony (doubt this gem came from expensive branding firm).

While one would hope these transactions have unlocked hidden value for their respective shareholders, it is not clear (yet) how the new organizations will better serve their customers.  Here is one personal example of how consolidation is acting against customer needs.  At MWW Group we have a proprietary measurement platform called Upshot.  For one client we set up a measurement system on Upshot with an XML feed from Bacon’s.  After a period of time we elected to migrate the system to Cymfony.  For consistency we decided to set up the system with a primary feed from Bacon’s.  We were informed by Cymfony that we could not use a Bacon’s feed because Bacon’s, er Cision,  refused to work with Cymfony because it competes with their measurement play, Delahaye.  So, Bacon’s lost a customer in this case.  It seemed misguided and petty to me at the time.  It still does.

Had a different experience?  Is industry consolidation working for or against you?