For the air force, the data conclusive and the way forward was clear. They wanted to add armour where the planes were being shot, the areas where all the red dots are. It’s very logical to think the same thing looking at the diagram. If you’re going to put more armour on the plane, you’d put it on the areas where the data says they’re being shot.
Fortunately, the air force had statistician Abraham Wald working on the project. He spotted that their analysis was missing a valuable part of the picture, the planes that were hit but that hadn’t made it back.
All those planes that returned to base and had their damage logged were the survivors. The bullet holes they were looking at actually indicated the areas a plane could be hit and keep flying – exactly the areas that didn’t need reinforcing.
Wald suggested that the air force should do the opposite of putting armour where the data said planes were being shot, and put it in the places they weren’t (all those white sections without dots). The result was that more planes came back safely.
Effective decision making requires a detailed look at data, but it’s very easy not to see the whole picture. In classical music we’re obsessed with audience data, but in many circumstances we’re not asking the right people.
The big crisis classical music is now facing is why audiences are continuing to fall in the post pandemic world when other industries have bounced back.
The question we are now asking is “how can we attract to more people to concerts?”. But if we think about survivorship bias, why are we asking our existing audience and making decisions based on their answers?
In my recent blog “Classical music audiences are vanishing… why aren’t we doing anything about it?”, I argue that the core reasons behind the drop in audiences is that for decades we haven’t adapted our product or how it’s marketed with the times.
Our existing audience are the survivors. They are the survivors of our bad product, they are the survivors of our bad marketing, and to bring this blog full circle, in some cases they are the survivors of our elitist and outdated concert dress.
If we want to improve things, we can’t make decisions based on the answers of the survivors. We might even have to make decisions that go against them.
One of my guilty pleasures is Kitchen Nightmares with Gordon Ramsey. If you’ve not seen it, Gordon goes into a terrible restaurant and helps the owners turns it around… usually with lots of shouting as the owners rarely want to change and shouting is kind of Gordon’s thing. Every restaurant is so bad that all the solutions are painfully obvious, so you as the viewer get frustrated at how ridiculously bad they are but also get to feel smart that you come to the same solution as a world famous chef.
Anyway, the reason I bring this up is one exchange between Gordon and owner of one of these failing restaurants. The owner started arguing with Gordon as he was worried that “the existing customers are going hate the changes”, to which Gordon replied “and how many people is that?”.
In being aware of the survivorship bias in our audience research, we may have to make decisions that are not only different to what our existing audience want, but may be in fact the opposite. The audience for the orchestra at the beginning of the blog may not only prefer the orchestra preferring tails, but they may actually hate any change in concert dress. But with audiences diminishing, Gordon would say “and how many people is that?
If we are going to bring new audiences to concerts and survive, we’re going to have to start doing some things that feel counter intuitive. We have to look for answers in the right places.
