Jul 14

Analyzing Apple ads to death

Who’s the villain? Apple, its agency … or the company that’s scoring them?


Oh look. The latest ads from Apple — the ones created by Apple itself — haven’t scored as well as those created previously by its longtime agency.

I know it’s true, because I read it on the Internet.

These news stories are all based on a single report from Ace Metrix, which bills itself as “the new standard in television and video analytics.”

Please, say it ain’t so.

First, be aware that what appears to be a news story is actually a marketing pitch. It’s the same technique used by assorted computer security companies to drum up new business, which we’ve seen many times before.

They release a report with a sensational headline (New Virus Blows Up Mac Security Myth!), watch the news organizations eat it up, then happily field inquiries from clients impressed with their skills.

Ace Metrix sells marketing analytics software and competitive comparisons. Their findings generate stories, which at the same time generate PR for Ace. An excellent way to build “the new standard” in analytics.

But what exactly is the “Ace Score” of which they speak? If you have the stomach, read on.

Exposing each ad to 500 people, Ace calibrates creative effectiveness by two key measures — Persuasion and Watchability. In their own words:

“The Persuasion rating is based on the interactivity of six data elements – Desire, Relevance, Likeability, Attention, Information and Change – automatically captured and analyzed for each ad. Watchability measures the engagement that a person has with the ad. Watchability and Persuasion interact to create the Ace Score.”

This is the kind of language that gives talented people nightmares, because it often gives ammunition to people who aren’t particularly good at judging creative work.

I get that a lot of companies feel compelled to subject their ads to deep analysis. But — would you like to know how Steve Jobs analyzed an ad? He looked at it and said “I like it” or “I don’t like it.” After it ran, he gauged the reactions to it.

Ace’s type of analysis is the reason why so many companies, usually the bigger ones, begin to churn out drivel. They get more concerned with ratcheting up their “six data elements” than creating great ads.

Steve didn’t tolerate that kind of thinking. Apple’s history of great advertising is the validation of his approach.

If Ace is truly the “new standard” for measuring creative effectiveness, one would assume they’d have a pretty good track record, right?

Well, careful with that assumption thing.

In October 2012, using the same data points, Ace declared that the thoroughly embarrassing Microsoft Surface “Dancing” ad was the highest rated tablet ad ever, ahead of those from Apple and Samsung.

Not content to sit that far out on a limb, the Ace CEO climbed out even farther, adding “The new Surface ads have flipped the ‘Mac vs. PC’ campaign on its head introducing intense aspiration among consumers, as evidenced by very high ‘change,’ ‘desire,’ and ‘attention’ Ace scores.”

I don’t know about you, but I’m ready to fork over a few hundred grand to these guys for their analytic wisdom.

And there’s more.

Remember Apple’s emotional holiday ad, “Misunderstood”? It scored well in likability, attention and relevance — but it was found to be lacking in information, dooming it to a mediocre Ace rating.

This type of analysis betrays a total lack of understanding about the power of advertising, and the proper use thereof. It’s as if every ad has to press every button. If that’s true, you’ll have to get used to the idea that most of your all-time favorite ads were actually mediocre.

To stand out from the holiday clutter as it did, “Misunderstood” was quite extraordinary. Connecting with customers emotionally, it had relevance beyond the short time it ran. It contributed to the Apple brand image.

Apple shareholders should rejoice that Ace wasn’t consulted about squeezing in some of those missing data points.

Now that I’ve expressed my disdain for Ace-style research, I have a confession to make. I actually agree with Ace’s conclusion that recent Apple ad efforts — particularly the “What Will Your Verse Be?” series — have been less effective than earlier ones.

But trust me, that’s just coincidence.

Logic like Ace’s is the reason why great creative leaders look at research with a wary eye — if they look at it at all.

The gut instinct of a few talented, experienced and brave individuals is worth a great deal more than 500 people ticking marks on a survey.


  • Andrew F

    If I were to conceive of a satiric name for some scummy snake-oil analytics company I’d have called it Ace Metrix… But then later dismiss it as too obvious.

    To this piece all I can say is: True that.

  • Ronin_Jim

    When in doubt: Not everything that can be measured, matters, and not everything that matters can be measured.

  • filecat13

    Thanks, Ken.

    My high school English teacher was a constant repeater of “The masses are asses,” which I think he relished because he considered himself superior to everyday idiots. I didn’t completely agree with him, but I did extract from that refrain that people are very, very gullible.

    Companies like Ace Metrix rely on that gullibility and on the insecurity of Internet analysts and pundits, who feel if they can’t sensationalize a topic each day (or half day or hour) that their careers are in jeopardy. It’s clicks, not quality, that counts in the ethically- and morally-challenged Internet journalistic profession, and it’s a sad state of affairs.

    I see your article as “quality” not “clicks” and thank you for it. In this time of everyone claiming to be an online expert without any credentials or proof of analytical or journalistic skills, I look forward to your experienced insight.

    Congrats on your best selling Insanely Simple.

  • Ken, you say, “After it ran, he gauged the reactions to it.” Could you share how he did that? I’ve always been curious.

  • ksegall

    Steve had a really good sense of the mood out there. He got a feeling from all the reactions he’d get from people inside Apple, as well as all those emails he received from the outside world. A great ad would result in a flood of positive emails and calls. If there was a more muted reaction to an ad, he knew it wasn’t working as well as previous ones.

    Not sure why I didn’t think of this before, but we should have opened a thousand dummy email addresses so we could bombard him with compliments for each new ad that broke!

  • LOL And if he had found out you’d done that, could you have survived the shitstorm? :)

  • I’ve often wondered if Jobs paid any attention to the Mac web/news/rumor sites. I can’t imagine he did – he’d be as frustrated/aggravated on a daily basis as I am!

  • ksegall

    I don’t think he paid any attention to those sites. He trusted Katie Cotton to handle the media and was more concerned about Apple’s high-level press.

  • Peter E

    Whilst I like the various ‘verse’ ads because they (to me) show more style than what I tend to see from the other camps, I do feel that they lack some of the ‘punch’ that I’ve come to expect from an Apple ad. I’m accustomed to the feeling of just being happy, and in some off way, proud, after an Apple ad, and I’m not getting that these days.

    I previously wrote here about my own kids and their experience using iPad’s almost exclusively for their schoolwork, and for the younger of the two, novel writing. The device continues to amaze me in the way it has encouraged the kids to be creative in ways that would have had hurdles before.

    Whilst I’ve said that the ‘verse’ ads aren’t working as well for me, I do really like the premise of the ads. Writing your ‘verse’ is a great way to describe how something has affected your life, but the ads aren’t quite working for me. Certainly, the last couple, like ‘chicken fat’ come across more as a pitch of gadgetry more than an effect on people’s life.

    These metrics as you say, miss the point I think.

  • ksegall

    Every so often, I feel compelled to explain my feelings about the “Verse” spots, and others of that genre. Thanks for giving me the opening :)

    Basically, these ads say that people all around the world are doing amazing things with Apple devices, making their personal contributions, etc. — and that you can do it too. It’s a beautiful, lofty thought. But beautiful, lofty thoughts don’t get people buzzing. They tend to make current Apple customers feel great, but don’t seem to change any minds on the other side.

    I always use the Mac vs. PC campaign as my gold standard. With every new spot, people talked about that campaign. At work, at parties, in the bars. They were “loved” as much as advertising can be loved.

    Apple just as easily could have created a campaign about all the things people are doing with Macs around the world. In schools, in movie studios, in recording studios, in labs, on spacecraft, and so on. They would have made Mac owners very proud, but I don’t imagine they would have reeled in many new customers — because PCs make up 90% of the world’s computers, and there are great things being done on PCs too.

    You could say that’s why the Mac vs. PC spots worked so well. Apple was the underdog with a much better product to offer. There were seemingly infinite things Apple could talk about to demonstrate superiority, and do it in a highly entertaining way.

    It’s different in the world of phones and tablets. Apple isn’t the underdog, so it’s probably not a good idea to be going head-to-head with the competition. But that’s not to say that iPhone and iPad commercials can’t have the same high level of smarts and fun that used to run throughout Apple’s ads.

    I came to love Apple because it had a spirit that stood apart from all the other technology companies. Its ads were never “safe.” That’s my problem with the current crop of commercials. They’re ambitious, shot around the world, beautifully crafted … but they don’t make me smile. They don’t make me want to send notes to my friends and say “You have to see this one.”

    Videos like “Verses” work wonderfully on the Apple website, where people go to explore the products and learn how others are using them. I love that on apple.com, you can dive deeper into the stories and explore all the featured apps within. That’s really smart.

    For the mass market though, who are bombarded with 30- or 60-second spots on TV, I’d love to see Apple be smart, entertaining and controversial — and get people talking again.

    End rant mode.

  • Todd Kaprielian

    Good critical eye, Ken.

    Speaking of “what will your verse be,” your post reminds me of another scene from that film: A student reads out loud from the introduction to the poetry text book. In it, the editor states one should use a Cartesian grid to analyze the quality of a poem and its merit.

    Having no use for such a mechanical approach to poetry, Keating (Robin Williams’ character) proclaims: “Rubbish! Tear out the page. Tear it out!”

    A delightful moment.

  • Peter E

    Yes! That’s it! They do make me smile, but I’ve got to say that the last ad I went to the trouble of showing to my wife was ‘Misunderstood’. the rest we see on TV and they’re nice to watch but they don’t sell anything as such; they just reaffirm to the converted that they’ve made the right move.

    ‘Misunderstood’ for me showed an awesome new device doing something we couldn’t have done as easily before with one device.

    There is one moment though in ‘Gigantic’ that has me smile everytime I see it. when that kid drops his head on hid dad’s arm I return to when my kids were that size. A good thing that.

  • Cold

    Put the 1984 Apple ad through Ace’s mangle and see what score it gets… nuff’ said.
    This is what happens when the bean counters muscle in on the marketing department with their ‘analysis’.

  • Pingback: a Critique of Advertising Analytics | zmetro.com()

  • I have never gone looking for an ad for my favorite soap or car brand on the internet. I can’t remember seeing any big interest generated about Microsoft or Blackberry having new ads posted on the internet. But there seems to be great interest in seeing a new ad from Apple. I’ve never read an analysis of that reality. That might skew the Ace Metrix results a bit.

  • godofbiscuits

    To quote the legendary author Frank Herbert: Truth suffers from too much analysis.

  • Del Miller

    Actually he did. Back at the turn of the century (ok, about 2000) I wrote a column for MacOpinions and once published a story about Guy Kawasaki’s EvangeList. The next day I got an email from Steve that said simply, “I always liked that list.” I wrote back with, “Thanks.”

    I also received emails from Phil Schiller, but only when he was mad at me. He once thought I was dissing Apple ( I wasn’t) and he wrote me several angry paragraphs to set me straight. I tried to clarify things but he never accepted my clarification.

    My takeaway was that the honchos at Apple do read the Macweb stories. They’re busy of course, so they probably don’t read many but, then again you never now.

    Aside: I wrote a story back then about the MacOS background called “Quantum Foam,” I ended up getting a nice email from Kim Thorne, the physicist who co-developed the theory. Nice!

    By the way, Shawn, we met at a dinner back in 2001 at MacWorld. It was good to meet you.

    — Del Miller

  • Del Miller

    Er, make that Kip Thorne

  • Jeff Lebowski

    While I think your critique of Ace Matrix based on the nonsensical nature of their criteria for “creative effectiveness” provides a more than adequate takedown of these schmucks, there’s analytical reason to ignore these results as well.

    Where the hell are the error bars for the graph? Data collected in this fashion are always subject to statistical uncertainty yet the graph lacks error bars to show the uncertainty inherent in the results. It’s quite possible that the differences in ratings are not even statistically significant.

    Also notice that they used a sleazy and age-old trick to visually distort the results by only showing a portion of y-axis. The bottom of the y-axis starts at 350 rather than zero which effectively zooms in on the top of the bars. Re-graph those results with y-axis starting at zero and suddenly the differences are revealed as rather small.

  • I really love what you said! Surface Ads – who liked them? or who bought the product seeing those ads, or who got interested in Microsoft consumer products seeing those ads? If this A(ss)ce Metrix showed me the numbers I would have taken their lines.