Data is ‘seductive’: but how to ensure meaningful measurement?

“Instead of wishing people would trust you, instead it would be better to be considered trustworthy.”

Tim Harford, BBC economics journalist and host of radio show and podcast More or Less, reminded the IPA’s Signals in the Noise event of this insight amid discussions about on measurement, or more specifically what is meaningful measurement in advertising and media.

Delegates heard that there is a danger of data overload and overcomplication, the importance of transparent, trustworthy and meaningful measurement and ongoing collaboration between media owners, advertisers, regulators and agencies.

Data can be seductive but inequal

Richard Kirk, incoming chief strategy officer at EssenceMediacom, revealed five truths to audience planning measurement that he described as “self-evident”.

These included how “Audience data is fundamental to comms planning” and “Media planners still use the maxim of trying to reach the right person at the right time.”

Data, meanwhile is “seductive… especially granular data”, Kirk went on, after having warned that data is “not created or curated equally” and that “data sources are numerous and varied”.

Kirk highlighted how “more and more walls and obfuscations have been put around data” in media and audience planning, and, as a result, he thought it was “very very difficult” to find out how data is created and curated, making agencies’ jobs harder.

“We have lost sight of the value in many instances of the data and, because of that, we have started to assume all data is equal,” Kirk said. “We rely on data to sell in plans and we have been seduced by the more granular datasets available to us.”

Kirk explained that the industry had gone from quantifying incrementality to claiming credit, and from a system where ads were “a weak force felt widely” to a system where ads are highly powerful and make individuals act.

He explained: “Instead of moving towards data that can add value, unfortunately there’s a big push in the industry to move forward into even more obscure things.”

When discussing the value of joint-industry currencies (JICs) to agencies, Kirk “apologised” as he said JIC data was “not fashionable” in agencies but that their data is trustworthy and could help in agency work in part because it is not focussed on granularity, but was “sweeping and huge”.

In line with this, Kirk concluded: “I want to change the goal of audience planning to reach a big audience in a big medium with a big idea to make a big difference.”

Media is ‘officially’ too complicated

Rhian Feather, head of media planning, OMD, warned that not only has media now “officially” become too complicated with an ever-fragmenting landscape offering advertisers and media agencies more places to put ads and test formats, but that it is not going to stop getting more complicated in future.

This is partially because of growth in online adspend, she said, before remarking on a forecast which showed 90% of all spend by 2025 will be online.

Feather explained: “As soon as it’s online there are many, many more metrics that we can look at, and media plans that once delivered just at impression level now look at rows and rows and columns and columns of data that we absolutely can report on. First-party data is becoming more and more important and the ability to use first party data regardless of the category that we’re in is just more and more available and there’s more abilities to use tech partners.”

Using a football analogy, Feather compared media channel performance with the reams and reams of available football stats, like how fast and far players run, or how many touches of the ball they have, can stop you looking at the overall success and contribution of players.

“Sometimes it’s not about using every piece of data, it’s actually about using the key attributes to make sure they are leading you to the correct conclusions,” she noted, explaining that the real skill is in understanding what has happened and for what reasons.

Feather went on to describe four things that would have “the biggest impact on measurement and accountability” as: transparency, standardisation, cross-media measurement and consistency.

Even if we are “still a bit lost” on cross-media measurement, she said, JICs would be able to tackle each one as they are owned by the industry which then allows them to be trusted and a source of truth, they can do the job of 10 media owner reports.

CFOs agree to independent audits, and CMOs should do the same

In his presentation on “Why JICs matter to advertisers”, Chris Ladd, outgoing head of media at Nationwide, remarked that advertisers can get quite frustrated if there is no transparency, and there is a lack of transparency around existing processes.

“It is surely right we ask any media proposition that they can demonstrate its effectiveness with audiences,” Ladd said.

He urged for more collaboration and transparency, too, adding: “We need to work together to crack the challenges in marketing and advertising. Without independent insight and verification those difficult conversations with CFOs about marketing investment only get harder. Remember every CFO has accepted independent audit in their financial accounts so to gain the respect we must do the same.”

Beware of vanity metrics

Eleni Marouli, head of market developments at Ofcom, explained how the regulator used multiple sources of data for their work and warned against using data that may on the surface look appealing but is not a meaningful metric, echoing Kirk and Feather’s earlier talks.

She explained: “There’s so much data is actually what I call ‘vanity metrics’. You might have metrics that sound balanced, they might sound impressive, but they don’t actually tell you much.

“I often see in the trade press, for instance, information about total hours and total minutes streamed by service — it is usually a big number, it usually would have grown quite a lot year-on-year or month-on-month, but what is it actually telling us? In context it could actually be quite small and does not really make sense.”

Marouli said this presented an opportunity to try new methods and think about how to analyse data more effectively.

She added: “As a risk-averse regulator, as we should be, this needs to be carefully evaluated by thinking about how exactly we are being measured in a meaningful way, so we’re trying to move away a little bit from the total measurement piece to a meaningful metric.”

To find ‘signals in the noise’, she recommended being clear about what we are measuring and why, and more importantly, what are we not measuring and how to overcome those limitations.