Data lessons from a mathematical genius

Abraham Wald (1902 – 1950)

Richard Shotton journeys back in time to the Second World War, where he says brands could learn a thing or two about data…

Which figure, from another field, do you think advertisers could learn most from? One that might not spring to mind is Abraham Wald.

Wald, who Matthew Syed wrote about eloquently in Black Box Thinking, was a Jewish mathematical prodigy, forced to flee Vienna when the Nazis seized power.

After settling in the USA he worked for the Applied Mathematics Panel, applying his talents to the war effort. In particular, how to reduce the death rate among European bomber crews. Theirs was such a dangerous job that the historian Kevin Wilson referred to them as “ghosts already”.

That wasn’t an exaggeration – the RAF’s Bomber Command alone lost more than 55,000 men, half its entire force, during the war.

Wald’s team needed to determine which parts of the planes should be armour plated. Too much and the bombers would be slow and cumbersome, easy targets for German anti-aircraft defences. Too little and the crew were exposed – a handful of hits might bring them down.

The researchers set to work collecting data. As planes returned from a sortie his team recorded which parts had been punctured by bullets. When hundreds of bombers had been logged, a pattern became apparent. As the image below clearly shows, the areas most regularly hit were the wings, fuselage and tail.

The top brass were ecstatic, Wald’s methodology had convincingly identified the critical parts to armour. Case closed.

The danger of survivorship bias

But was it? Wald’s interpretation was radically different. His bosses had fallen victim to “survivorship bias”: assuming returning planes were representative of all aircraft. That was a mistake. The data collected omitted the crucial planes: downed ones. They held the clue to solving the problem, not the returning planes.

The parts peppered with bullet holes could be ignored on returning planes, after all they had managed to limp home. It was the unscathed spots that needed armouring. Any plane hit in these areas was at the bottom of the channel.

Wald’s counter-intuitive thinking was quickly tested and shown to improve survivorship rates. His thinking became the standard US policy until the Vietnam War.

This is more than a historical anecdote. It’s a guide to how best to use data, whether in the military or marketing.

Four key principles

First, don’t rely on intuition. Wald was a genius (a genuine one, not in the way we now bandy about the term) yet even he didn’t rely on introspection alone. He collected data, and only then, developed his hypothesis. Far too many plans and approaches are based on gut feeling alone.

Second, collect data in the simplest manner that answers the question. Wald’s methodology required pen, paper and a numerate assistant. Basic even for the 1940s.

This is important for marketing as too often we obsess over the complexity of our methodologies: econometrics, machine learning, artificial intelligence and so on. It’s as if we believe we can replace the hard work of thinking with the high costs of measurement.

Third, make sure the data is representative. Too obvious? Well, most digital tracking isn’t. The vast majority of online measurement collects short term effects only: sales, views or visits.

This means plans are optimised to the short-term, despite the fact that the majority of advertising’s effect occurs in the long-term. Any short-term metric needs to be complemented with long-term brand metrics through simple exposed and control tracking.

Finally, data alone is worthless, analysis is key. Agencies chasing advantage from data alone are deluded. Assuming that data tells its own story can lead to the wrong solution.

In Wald’s case it would have led to armouring inconsequential areas. Instead, we need a combination of best in class data collection and interpretative skills.

The best source of those skills is the science of decision making: namely behavioural economics and social psychology. It’s the agencies who best harness these approaches that will win the data war.

There’s a good case for Wald being the least known figure that advertisers could learn from. But that’s just my view. Let me know your suggestions.

Richard Shotton is head of insight at Zenith – follow him on Twitter: @rshotton