It’s been an exhausting week here in New Zealand. Most of the population has been staggering round bleary-eyed but elated having sat by TV screens into the small wee hours to watch our football team triumph in the FIFA World Cup. Oh, sorry – didn’t you know we won? Ryan and the boys might be on their way home, but the certainty here is that they were undefeated in the tournament and finished ahead of World Champions Italy.
Of course, the tournament is still in full swing and whoever lifts the trophy yet to be decided, but New Zealand’s outcome – for a country with just 25 professional footballers and ranked 78th – is an emphatic and glorious win, with a proud nation hoping someone will organise a welcome home victory parade. All of which only goes to show why it is essential to understand the difference between measuring outputs and outcomes.
This month saw another cross-border event, with the agreement and publication of the “Barcelona Declaration of Research Principles” at the 2nd European Summit on Measurement. Five global bodies and 200 delegates from 33 countries all voted overwhelmingly to adopt the following principles:
- Goal setting and measurement are fundamental aspects of any PR programmes.
- Media measurement requires quantity and quality – cuttings in themselves are not enough.
- Advertising Value Equivalents (AVEs) do not measure the value of PR and do not inform future activity.
- Social media can and should be measured
- Measuring outcomes is preferred to measuring media results
- Business results can and should be measured
- Transparency and Replicability are paramount to sound measurement
While I was delighted to see this agreement finally putting the lid on the unethical tosh and nonsense that is AVE, I was still frustrated to see this list so closely tied to media, when public relations is concerned with so much more. I was also a little concerned to see the list tucked under the heading of ‘research principles’, when research and research methods don’t get a mention. But hey, like NZ qualifying for the World Cup in the first place, it is a beginning.
The steps and stages of public relations and communications evaluation have been around for decades and any practitioner worth their salt has a robust process in place that informs and guides strategy, engagement and relationship building as well as assessing effectiveness against organisational outcomes. Generating coverage centric numerics and retaining a focus on media measurement without moving it into an evaluation process generally leaves people with a list of nice looking but meaningless numbers. After all, a measurement gives you a fixed unit – e.g. how long is a piece of string – but an evaluation provides you with the significance of the outcome – e.g. how well did the string tie my shoes, was it the right length for the job, would something else have been better?
If I look at our football team’s numbers – 1-1, 1-1, 0-0 – it tells me they didn’t win a match. Not a good look in a world-class soccer tournament. But if I evaluate against the desired nationally recognised outcome set before they left, which was to take part in the World Cup Finals and, if possible, get NZ’s first ever point, the numbers reveal the outcome was not only reached, but exceeded. The evaluation – but not the numbers – would also reveal that the team won the collective heart of a previously rugby-obsessed nation, (changed dynamic), increased active participation in soccer to the extent you can’t walk down the street without seeing someone kicking a soccer ball (changed behaviour), and there is no doubt at all here now that New Zealanders can, indeed, play soccer (changed perception).
Pre- and post- research would have provided me with the specific qualitative data I needed to demonstrate the difference made by our participation in the World Cup to the country’s relationship with soccer (social impact). Other relationships will have altered too – education and soccer, sports equipment suppliers and soccer, mainstream media and soccer – all important and measurable relationship strands within the overarching relationship outcome.
Once the boys are home, ongoing research will inform where the next ‘agreed outcome’ should be set. Will we still be happy with ‘win, lose or draw – you make us proud’ or will expectations be much higher based on the changed relationship? And exactly how will that relationship be sustained in the face of increased expectation and demand? Well that’s were we get to work. Such sustenance is an integral part of our role as public relations professionals, but without research holding hands with evaluation and measurement joining the dots between the two, we cannot do it effectively.
The Barcelona Principles are long overdue and are to be welcomed, but personally, I am looking forward to the next declaration in the hope it will really take this essential public relations discipline into the main stadium for the win.
[Note: Access the Barcelona Summit site for details of the Barcelona Declaration of Research Principles.
Great discussion happening here. For the record…
The case for AVEs:
http://ontherecordpodcast.com/pr/otro/public-relations-measurement-today-and-tomorrow-with-prime-research-ceo-mark-weiner.aspx
And the case against them:
http://ontherecordpodcast.com/pr/otro/pr-measurement-standards-declared-by-global-pr-industry-delegation.aspx
There is a chill attached to this debate. The significance of, in particular, print has its limitations. I did a little bit of work on the changing significance of different media last week (http://bit.ly/dkIqvw). It showed that the press is in even steeper decline as a medium.
It also exposed whatever advertising values was in a new light, namely it has less and less significance to the reader and the advertiser.
Would we even want to be associated with such a dodo?
In the meantime the stalwart face of the organisation, the web site, is also in decline (I first mentioned it in September 2006). Are websites no more than graffiti strewn signposts?
This is as pathetic as Mr Micawber contemplating a life of penury.
But more significant is the rise in the value of values.
Here is a real measure. One we can grasp. A measure about commitment, features, benefits, outcomes and trade-able pay and currency.
Here are the measures we can adopt, love and rely on. Far better than platitudinous weather forecasts for the beach at Barcelona, we see in everything we do, our constituents’ offering their view of our client’s value.
Any hard data on the decline of websites you can point to?
Ed. Note: “are” should be “is” in the last sentence. Hard to compose in these little boxes.
As right-minded as the Barcelona document appears, it still has not replaced the bastard term “impressions,” which is borrowed from advertising terminology and therefore closely linked to AVE. A new term is needed, hopefully one that describes what happens as a result of message delivery, not just “opportunity to see,” which are included in the term impressions.
Quite agree Bill – the other one that gets me foaming at the mouth is ‘PR Value’ – more made up numbers around clippings that take AVE and multiply them by yet another made-up amount (sometimes a multiple of three, sometimes five) on the basis that people will have ‘read’ whatever has appeared as opposed to ‘not reading’ an advert. Held up as a ‘measure’ such nonsense undermines true value and demonstrates very little understanding of public relations.
Judy – I appreciate that in-house there are many interpersonal and other political factors that can affect even the willingness of bosses and marketing colleagues to listen to arguments against AVE. However, being in-house can be an advantage, especially if there is a research function in the organisation. In my past experience, I have discovered these functions are invaluable in terms of having baseline information on attitudes or other key measures that PR can influence, in being willing to put PR questions into studies and in providing ongoing data that shows where PR has an impact. It gave us data that showed when car purchasers for example, recalled articles they had read as influencing their decision – and often of higher value to them than adverts they had seen.
Another thought out of the Motor Industry Public Affairs Association PR Masterclass yesterday when evaluation was discussed was a practitioner who stated that his bosses no longer asked for detailed figures or even specific articles, etc. The reason was that they trusted the PR person’s abilities and focused more on what they wanted that person to achieve.
This echoes my own early experience in house in a job where initially every cutting was clipped, put into a computer system (AVE calculated) and stuck in books displayed at board meetings. After a while, my marketing bosses observed that I was generating so much coverage that they couldn’t process it – they believed business benefits were being achieved from my work so preferred me to focus effort and budget on doing rather than counting.
That’s not to argue that evaluation isn’t important – but there are times with restricted time and money when it is best focused on achieving, especially when you are trusted as an expert in this area.
The personal example I was referring to was in-house. Which brings me to the point I’d like to state to consultants that it’s easy to take the high road when your relationship with the parties is on a part-time basis, rather than complicated by other factors.
Of course I welcome suggestions, online or off, on how to win this battle down the road.
And public thanks to Toni Muzi Falconi for sending me a PDF file of the study by John Swerling and company! (Sponsored by the Strategic Public Relations and Communication Centre.)
to the people who argue that they can’t say know because “clients ask for it.” If your clients asked for heroin or a nuclear weapon, would you provide it? Lets be real. Clients are sentient humans that have brains and education and are looking to us for advice. If we go along with stupid requests, sooner or later someone smarter will come along and replace us.
Sean: I can quantify PR’s contribution thru correlations to web trafffic, revenue, contributions, membership.. Isn’t it so much better to demonstrate busienss value and impact than some silly number based on total fiction?
Definitely interesting thoughts here and I totally agree with Heather that the solution is in our own hands.
I too have heard practitioners cry ‘But that’s what clients ask for’ more time than I can count and although each time I hear it, I provide them with a counter argument, still it persists. The fact that providing AVEs is a bit like keeping your eye and concentration on the stereo light on the car rather than checking the real dashboard for the operational statistics you need to stay on the road, like fuel, oil, speed and direction apparently passes people by.
Toni – I agree entirely with your thought that there was a time-zone blip that caused almost a generational lag in thinking, but perhaps the Barcelona Principles might just help those who are still being asked for AVEs to resist the temptation to provide them.
Heather – the ‘bastard child’ of AVE was born of initial illicit liaisons between advertising agencies and public relations practitioners in the late 1970s/early 80s when advertising was a dominant medium. Sadly it wasn’t confined to history along with bow ties, large cigars and long expense account lunches and lives on to haunt us all.
Oh – and I’m really sorry about Italy and England. Hope the hounding is not too great!
AVE is a bane of our practice, but only A bane. Indeed, many practitioners use it because, 1) it’s a dollar figure; and 2) the numbers guys like having a number. I had coffee with a practitioner today who uses AVE – and the published rate card AVE, too! I offered a couple of suggestions for more meaningful, accurate measures, which he made glad noises over, but… he uses AVE as a baseline — we had $X in AVE last quarter versus $Y AVE in the same quarter a year ago. This could be similar to the weighted media cost figure that Jeffrey/Rawlins/Jeffries-Fox wrote about recently, a proxy for reach, impressions, etc., that improves correlations (according to their paper).
As for the principles from Barcelona — as a profession, we still are struggling to quantify our business value. We haven’t been able to agree on ending an obviously inaccurate and flawed measure — how could we move mindsets from quantitative to qualitative?
Our marketing cousins whip us in measurement because their outputs are more closely tied to the buyer’s consideration process — effective ads are by their nature memorable, whereas keeping your organization’s name OUT of a negative story, or equipping employees to be more effective at work are less forward and easy to isolate in the communication mix.
Add to the issue: Marketing is taught in business schools and PR/Comms is not. The C-Suite frame is for marketing — and PR often is a difficult fit.
Sean
@Commammo
First of all, consider the fact that a year ago 80% of attendees at the Measurement Summit in Berlin raised their hands when asked “Who here uses AVEs” — so the fact that the same group a year later rejected the concept is the equivalent of the US making it to the World Cup at all! Secondly, there was lots of talk about next steps and enforcement and the first place I’d go would be to outright reject any and all award applications that include AVE at least by the organizations that agreed to the principles. The fact that they are still included and praised makes me want to puke. Shame on those folks
An 80% swing in just a year is definitely a win. I hope that the next steps are discussed and debated as widely as possible and would agree that AVE has no place in award applications. About five years or so ago I did some research into how measurement and evaluation were being presented in award winning campaigns around the world, looking at the published winners from around the globe and, at that stage, there was very little mention – or even a passing nod – to measurement and evaluation. Last year’s crop of globally available winners paid more attention to the topic, although, sadly, there was still an emphasis on media clips, mainstream media coverage and AVE. Like the change you observed at the Measurement Summit, progress has been made, but there is still a long way to go. Perhaps in the light of the Barcelona Principles those running award programmes will draw a nice big red line through any mention of AVE in future entries.
Interesting observations Judy and Toni. However the reality is that marketing people are generally more open to evaluation so why do PR people give them AVEs instead of discussing qualitative research which marketers use to assess recall of adverts etc? I feel the problem and solution is in our own hands here.
Judy,
you might wish to look up the last two or three editions of Annenberg’s GAP (general accepted practices) study coordinated by Jerry Swerling.
As much as I can remember there was a definite correlation somewhere in these studies between the reporting to marketing and the level of investmente in quantitative rather than qualitative evaluation and measurement.
Do we know whether any studies have been undertaken (by the AMEC or IPR) as to whether there is a correlation between a company believing that AVEs are a valid measurement and having its public relations department/staff report to a CMO?
As someone who has bumped up against this mindset in a professional capacity, I’d wager (a gentleperson’s bet) that research would demonstrate this is usually the case. After all, in many ways advertising is a closer kin to marketing than it is to public relations.
Sorry – that should be a 4-2 result today, if World Class football also rid itself of out of date methods of evaluating outcomes!
Cathy – as Toni, I like the link you’ve made here to the World Cup. Being English, your analogy means that my home nation is highly unlikely to ever achieve our desired outcome, which of course, is nothing less than to lift the shiny gold trophy (again!). Any other numbers most certainly won’t do. No need to comment on the 4-1 result today as the evaluation of the English team will be undertaken using a largely qualitative, reflective methodology over the next days, weeks, months, and years to come. Never will so many words be written in assessment of such a terrible set of inputs.
Onto the evaluation matter – I am not convinced that the Barcelona research principles are a step in any positive direction. As you note, the basics they contain have been well known in the “professional” and academic PR worlds for many years – and indeed, left behind by more recent work.
I totally agree with you that what is too often lacking in such approaches is the wider perspective of PR, the necessity for research methods and methodologies – and for me, most importantly, an understanding of setting relevant objectives. Evaluation of what has been done without having established what needs to be done strikes me as almost futile. There is that saying that if you don’t know where you’re going, any road will get you there.
My real concern however about such principles is that they will have little, if any, effect on the majority of PR practitioners. AVE is alive and thriving in my experience. I have just been a judge for a category in the CIPR Excellence awards (http://www.cipr.co.uk/content/2010-Excellence-Awards-results) – and understand that AVE raised its ugly head in the majority of all the entries in all the categories. Indeed, such ROI (as stated) was even praised in some of the judges’ comments.
In asking the entries about the use of AVE, the answer was invariably that although recognising it was flawed, it was asked for by management/clients/marketing colleagues, etc. Or, that it is a monetary value which is important to offer.
And that’s the good news – it doesn’t take much searching online or in conversation with practitioners to hear justification of the use of AVE – despite the equally prevalent information on its uselessness.
AVE is a bastard child with no recognised lineage (that I’ve been able to discover). It’s birth is undocumented but it continues to live on, regardless of all the stakes in the heart it receives from the academics and “professionals”. Much like England’s perennial dreams of 1966 and another World Cup victory – it seems that AVE is a measure that knows no end.
As the Italians say: MAGISTRALE !
(no reference to our hilarious and grotesque catastrophic adventure at the world cup… the situation is grave, but not serious…).
This, Cathy, is truly a brilliant post indicating that:
a) yes, the barcellona principles are long overdue and are welcome;
b) the barcelona principles however also appear far remote from our quickly develping global body of knowledge. So they are indeed useful, but mostly as a defensive tool.
Coming the day after the Stockholm Accords and with some of the leaders being the same (Global Alliance, IPR), one could wonder if the three hour flight from Sweden to Spain produced a time zone diference of maybe a decade.
As my Italian colleague Geronimo Emili voiced on the Ferpi website a few days ago commenting Barcelona: but…are we really still at this point in the international professional community?