Job vacancy: official adtech stats debunker

Some companies are conditioned to exaggerate, while others place blind faith in their own numbers – either way, someone will call them out, one bollocks stat at a time. By Dominic Mills.

Let’s imagine for a moment that I am a wealthy man and I wish to leave adland with a legacy – something useful that also (naturally) sees my name live on.

What would my endowment look like? The Mills Chair in Offensive Column Writing? The funding of an annual weekly retreat in, say, a Trappist monastery in the third week of June? There, Cannes refuseniks could cut themselves off from all communication.

Not quite, but the latter idea has a certain appeal.

No. My endowment would be called The Office of Independent Investigation of Irresponsible Number Wanging (OIIINW – not very catchy, is it? Suggestions for a better acronym gratefully received.) and it would fund a load of grumpy old gits to attend as many ad industry conferences as possible.

Their job would be to sit through presentations, mostly case studies led by adtech companies, and call out the bollocks use of statistics.

When I say ‘grumpy old gits’, I mean it in a nice way. The essential qualifications are experience; integrity; the ability to spot a statistical lie from the back of an auditorium; and the willingness to say, in effect, ‘the king has got no clothes’.

Take last week. A case study, involving an adtech-led activation on behalf of a well-known company, contained the claim that display ad ROI was boosted by 35%.

An early attempt by a questioner to determine what constituted the ROI came to naught. It puzzled me too: what is digital display ROI and how do you measure it? And to what extent could it have been influenced, as in this case, by a simultaneous TV campaign?

That was bollocks #1.

Bollocks #2 came, so to speak in the small print, where it transpired that some serious number wanging had been going on.

This revealed that the 35% ROI figure – let’s forget for the moment what it is based on – was based on a 9% increase amongst 26% of the audience.

Magically, this was extrapolated by a factor of four – thus 35% for 100% of the audience.

Even I, with the numeracy of gnat, know this this cannot be. This is a sub-GCSE maths failure.

Assuming the same ROI was proven among the other 74%, then the highest ROI it could be would still be 9%.

Any decline in ROI amongst the 74%, and the 9% figure would be less. Yes, it could have gone the other way too, but I suspect the offenders chose the best quartile on which to base their numbers.

Fortunately, there was a prototypical recruit for the OIIINW in the audience who spotted the lie.

What fascinates me here – and I have seen this sort of thing before – is the way adtech companies often make such outlandish claims. It’s counter-intuitive: you’d think that people who build algorithms for fun would be, in a thoroughly rigorous manner, all over the numbers.

But we have seen this time and time again with, for example, Facebook.

Me, I think there’s a combination of cultural forces at work here.

One, especially for those adtech companies chasing VC cash or a sale, they are conditioned to exaggerate.

Two, they have blind faith in their own numbers – hence, for example, Facebook’s failure to recognise that its claim to reach more 16-24s in the UK than there actually are were rubbish.

And three, they just don’t understand advertising and have little reverence for the norms and expectations of the sector.

This, I think, is because they are a) highly opportunistic in their approach – they could be in any sector, but this month it’s advertising – and b) hubristic.

But were, say, a junior planner caught playing fast and loose with stats in the same way, they’d be out the door pretty fast.

Were we to frame this as a binary choice for CMOs when it came to evaluation it would be like this: would you rather work with an agency whose DNA is built on an intelligent understanding of media metrics, or an outfit whose methodology is transparent but spectacularly wrong?

And here’s the motto the OIIINW could adopt: Careless Use of Stats Costs Credibility.

JIC revival, part II

As luck would have it, following hard on the joint IPA/ISBA JICs initiative the week before last, the latest Barb newsletter drops into my inbox.

One particular passage in a piece by CEO Justin Sampson caught my eye.

He wrote: “First we need to be clear about our scope; defining television isn’t as easy as it was, and that’s just in the Sampson household. It’s straightforward if we take the side of my eldest son: if he’s not watching on a TV set, it’s not television. But my younger son thinks of Netflix as television, regardless of which device he’s using.

“His definition is closer to the official one provided by the European Union, as part of its Audiovisual Media Services Directive. It’s not a quick read, but a couple of points stand out.

“First, the concept of a programme should take into account on-demand services that are, quote, television-like; in broad terms, this means they’re competing for the same audience as television broadcasts. And second, the concept of editorial responsibility is deemed essential for defining the role of the media service provider and, therefore, the definition of audiovisual media services.”

As I understand it, the dance between Google/YouTube and Barb continues, despite last month’s stumble.

But two initial hurdles have to be crossed first. One, all parties – and that includes advertisers and agencies – have to agree what it is they want to measure. But as YouTube wants budgets that currently go to broadcasters, that seems pretty unequivocal to me: it must be numbers that allow agencies to make comparisons between YouTube content and that put out by broadcasters.

And two, that Google’s red line – so-called viewer privacy considerations – must be eased. I’m told that, in the US, Google and Nielsen have made some progress on that area. Whether that works this side of the Atlantic remains to be seen.

Meanwhile, a larger issue – for Google and Facebook at least – looms. This, as referred to by Sampson, is the EU’s Audiovisual Services Directorate (AVMS), which the EU is currently seeking to refresh so that it includes social media video and which would therefore pull the online twins into the same regulatory regime as other broadcasters.

The AVMS underpins Ofcom – and is unlikely to change with Brexit – so any ‘refreshment’ would apply here in the UK.

Naturally, as this FT story details, the twins are resisting it like mad because, from their point of view, it opens a can of nasty worms.

Just to twist the plot further, last week Vodafone tightened its YouTube controls, and both major UK political parties were furious to find their ads appearing on hate sites.

None of this automatically means the twins will play ball with the JICs, but there’s a clear sense that the pressure to fall in with the media mainstream – with all that that entails – is building slowly but remorselessly.