Aug 09

Tales from the dark side: testing

That guy on the left? He thinks your ad could use a little pep at frame 22.

Uh oh, that guy on the left has an issue with Frame 16

With all that gushy love stuff in my last post, I feel a need to dredge up some painful memories to make up for it. This is what I call “fair and balanced.” So today’s topic will be that longtime nemesis of creativity: testing. We should start by grudgingly accepting that testing is just a fact of life with certain clients. However, one thing we should never accept — grudgingly or otherwise — is the misuse of testing. This not only kills great work and depresses people, it’s a horrific waste of money.

To make your blood run cold, I have a favorite example. (Kindly relive your own nightmares by adding your comments for the amusement of all.) Not sure if it’s still the case, but for a long time Intel had a large in-house group that specialized in this dark business. Their mission was to unrelentingly test around the world to make doubly-triply sure that Intel’s advertising convinced everyone and offended no one. Creative ideas were tested no less than three times: at the concept stage (to determine which ads to run), after production (so we could tweak before going public) and after the ads had already run (so we could tweak again and/or gain “learnings”). If you’re a fan of the Rack or other instruments of torture, you will especially enjoy the way they went at it with a 30-second TV spot. After viewing, individuals would be shown a series of 30 frames — one for each second of the commercial. They were asked if they could recall each frame, and in this way all 30 frames were rated on their effectiveness. Never mind that some of these frames were transitional and not exactly high points of the story. The research group would present a voluminous report, complete with suggested “improvements.” After all, we couldn’t have a spot in which the test audience was under-thrilled by the 4th, 12th and 21st frames. It was at this time that the creative team would jump to its feet to explain what most children pick up after their first viewing of Dumbo: a good movie has peaks and valleys, and the peaks don’t feel very peaky without the valleys. Intel seemed to be of the mind that if they were spending a million bucks, they should really get their money’s worth — out of every frame. Granted, a commercial does need to get noticed, but if I were an advertiser I’d be a little more concerned about what happens after the last frame. Like what kind of impression did the viewer walk away with?

The way different companies use or don’t use testing is a rich, rich topic that will likely come up often here. I’ll bet there are some really uplifting stories out there as well as excruciating ones, because (thankfully) smartness has a way of winning in the end. I’d love to hear which companies are doing it right and wrong.

Tags: , ,

  • Liebman

    Aha! So the underlying thesis of Ken Segall’s Observatory has now been revealed: smartness eventually wins. (Perhaps you should have add a spoiler alert on that final paragraph.)

    Disregarding your overly-optimistic stance, I have to say I thoroughly enjoyed the post. As a veteran of too many focus groups myself, I find it amazing that although everyone on the client side the one-way mirror knows that the process is seriously flawed, the group marches on like they know what they’re doing. And the consumers on the other side of the mirror aren’t much smarter (although their hourly rate is often slightly higher than the suits). It’s all enough to make any self-respecting creative cringe. But then all made up for when that binder full of dinner menus is brought into the room. Kobe beef burger anyone?

  • Focus groups… they can almost make you lose you love for M&M’s.
    All testing is frustrating, however, I almost prefer the quantitative format, rather than the qualitative. Whenever there’s room for interpretation, that’s when creatives tend to lose most of their arguments.

  • Hey Ken,

    If testing is a topic of interest to you, why don’t you show some alternative ways to test/develop ideas for brands, like what we did for Fatheads, Boniva, ERA, Uncle Bens …..or if you really want to go crazy with new techniques check out Video Chat Network….there is hope out there in research land.

  • Bryan Birch

    My favorite on research (testing) is the lampost analogy:

    A lampost can be used two ways. For illumination, or the way a drunk uses it—for support. Intel, in my experience, kept testing until they found agreement.

    My favorite focus group, (Taco Bell) had respondents talking about how they were eating more healthy. Nearly everyone, in every group. The client was thrilled. We pointed out that all the “fatty” snacks on the table were gone, and the healthy snacks were left untouched. But they had the support they needed to launch the “healthy menu” for Taco Bell. The biggest failure in the history of the chain.

  • ken segall

    Hey Bryan — I love that Taco Bell story. Classic. I have a sneaking suspicion we all have more than our share of good focus group stories and I hope others share theirs. We run into so many of these things in our business, where common sense tells you all you need to know, yet things go in the wrong direction.

  • ken segall

    Hi Rachel,

    I’m not sure I’d ever reveal any of that stuff :) But I do get your point. Testing has to change along with consumers, and what you’re doing adds a new dimension to the idea of testing. It’s immediate, just like the Internet. (So why is this reply a few days late?) You do give me hope!

  • Martin Olivero

    You asked which companies are doing testing right or wrong. I’ve certainly been a party to both – but I came across a presentation by Millward Brown just yesterday on that very topic that provides a much broader view and analysis – I know what youre thinking — its a research company what else are they going to say?! But trust me – if you have the time and the interest, its pretty insightful – check it out



    PS looooved the Johnny Walker piece