Every year, an instructor at a large state university acquaints students to the Criminal Justice degree program. In the midst of the talk, a student enters the room, whispers a few words to the instructor and leaves, whereupon the instructor resumes his talk. After a while, he asks his audience to take a sheet of paper and write down the answers to a few questions. What color shirt had the student who had entered the room a few moments before been wearing? Had he been carrying anything in his hand? What color and length of hair did he have? Was he wearing eyewear?
The point of the exercise, in fact the point of the staged “exposure,” is to illustrate that eyewitnesses to events are unreliable and just how few details we recall from events that occur in our daily lives.
This point is well taken for the criminal justice system and for marketers. As we go through each day, we are bombarded with stimuli that we instinctively sift through, retaining what has meaning for us and discarding what does not. Whether we are sitting in a classroom listening to an instructor talk, driving along a city street listening to the radio and keeping an eye on signs, pedestrians, and other drivers, or sitting at home watching a favorite TV program, a plenitude of stimuli are there for our attention yet we internally process what to attend to and what to disregard, what to retain and what to dismiss: we focus our attention on an instructor and dismiss the message bearer; we stop for a school crossing guard and lose track of the radio program; we watch a TV program and divert attention when commercials come on.
How different our experiences would be if someone were directing our attention to aspects of our lives we might not otherwise consider important. What if someone in on the instructor’s ploy was sitting next to us and said, “Pay attention to this person.” With our focus honed in on the student, details we might not notice or might quickly dismiss would become items of scrutiny. We might not only remember the shirt color, but note whether it is a t-shirt, pullover, or a button-down type. We might notice not only that he had a few pages of paper in his hand but also that they were covered in notes written on yellow legal paper. Ordinary details that would otherwise be of no particular importance would attract our attention, worthy of some notice and perhaps consideration.
The story of the university lecturer came to mind when thinking about Michael Imperioli asking in a recent 1800 Tequila commercial, “Whatever happened to commercials? So many of them don’t make any sense and you can hardly tell what anyone’s selling.” Mr. Imperioli could be onto something because over the past few years, a sea change has occurred in the way companies evaluate the content of their advertising which embodies the contrast between how we normally observe events and how we might see things if our attention were abnormally directed to things we would otherwise dismiss.
Historically, companies relied heavily on day-after recall (DAR) methodologies to evaluate their advertising. DAR tests are designed to replicate, as closely as possible, the real-world environment in which people are exposed to an advertising message. In a DAR test, respondents watch a television program or read a magazine in their own homes, under natural conditions. The following day, a research interviewer contacts them and asks about the content of the program or publication, including the advertising. A DAR test provides insight into what people attend to and what is sufficiently important and meaningful to them that they retain it for at least 24 hours, despite all the other events going on in their lives that demand attention and occupy their minds.
In recent years the trend has been away from DAR testing toward more expedient diagnostic designs as marketers look to save money and time. In a diagnostic test, respondents are shown advertising, usually in a clutter environment of a few other advertisements and occasionally in a faux media environment. Testing is done in an unnatural viewing/reading context, often via the internet or occasionally in a research facility such as those located in shopping malls. Respondents are directed to look over the advertisements and often the test advertisement is shown more than once to provide respondents with an opportunity for further study and scrutiny. Interviewing about respondents’ thoughts about the ad takes place immediately after exposure.
The differences between these methodologies are dramatic.
- In a DAR test, respondents experience media in a normal environment under typical viewing/reading conditions. They are in their own homes, in the company of family members, in their favorite armchair by the lamp they typically use when reading or watching TV. A diagnostic test, by contrast, occurs in an environment in which the advertising appears out of context and in an artificial setting, either over the internet or in a research facility.
- In DAR tests, respondents see advertising in real time, as it occurs when media is purchased, within the context of other messages competing for attention and against normal household distractions. In some DAR tests, respondents know only that they have been asked to watch a TV program or read a magazine. In others, they are not pre-contacted to participate in research; the recruitment will not occur until the following day so that their viewing/reading behavior is entirely normal. In diagnostic tests respondents are recruited from a shopping mall or from an Internet panel and are fully aware they are participating in an advertising test. They are usually re-exposed to a test advertisement more than once and have the opportunity to focus their full attention on the message in a manner they may never do normally since they are anticipating that they are about to be interviewed about it and wish to appear knowledgeable.
- Respondents in a DAR test attend to media as they normally would, focusing upon things that interest them and ignoring or dismissing things that do not. In this regard, they are very much like the audience watching the university instructor who may or may not attend closely to an interruption (the instructor interrupted by a student is in some regards similar to the way an advertisement interrupts a program or a magazine feature). People in a diagnostic test have their focus deliberately directed to the advertising, just as though someone told them to what they should pay attention. If only companies had someone telling consumers to pay attention to the upcoming advertisement!
- About 24 hours pass before respondents are interviewed in a DAR test, allowing time for inconsequential events to fade from memory and for more lasting impressions that remain to inform buying inclinations. Diagnostic interviews occur immediately after exposure determining less what people retain and more what is recognized.
As a consequence, the perspective that respondents have of advertising based on their participation in DAR tests and diagnostic tests is as vastly different as how the student in the lecture hall is normally regarded and how he is seen when people are tipped off beforehand. It’s like the difference between seeing the world as it is and looking through a distorting lens. The lens changes the way people see things and these differences distort what companies learn about their advertising from the type of testing they do.
- Communication of the strategic message is usually much higher among those who prove recall of an advertisement in a DAR test than among participants of a diagnostic test despite the fact that respondents in a diagnostic test have just seen the advertisement. In a DAR test, those who recall the commercial have done so because the message has engaged them. When the advertisement’s message resonates among consumers, that advertisement has a better chance of forming a connection with the individual and of being remembered. Consumers for whom the message has little relevance dismiss the message and are not as likely to remember it. The DAR test effectively identifies the prospects most likely to be affected by an advertisement because recallers connect with the message. Communications effectiveness is being measured among this select group. The diagnostic test, by contrast, determines how well an advertisement’s message can be recognized and reiterated by a sample of consumers who have just seen the message and can only report on what they recognize the message to be. Since some proportion of respondents included in the diagnostic test are those for whom the ad has little meaning and who would ignore or dismiss it in a DAR test, they often fail to connect with the key message, leading to lower levels of message registration. While message communication levels in a diagnostic test has value, particularly during early message development phases, it tells marketers less about how well the message forms a connection with the reader since consumers are responding with what they have just seen, not describing an impression that has connected with them that they have chosen to retain.
- Measures of intrusiveness between the systems are very different and generally do not correlate. Aided brand recall levels in a diagnostic test approach 100%. Respondents have just seen the advertisements so the task of identifying what appeared in clutter is a snap. Recalling an ad in a DAR test of respondents is tougher since an ad must connect with the consumer in a meaningful and lasting manner. Claimed recall to an aided brand prompt is often in the 20-40% range, depending on the DAR methodology, and the percent that can accurately “prove” that they have seen the advertisement that actually appeared is somewhat less than that. Respondents in a DAR test do not know that they are participating in a test of advertising and yet must recall the ad and accurately describe it 24 hours later despite the tens of thousands of other stimuli that have clamored for attention in the meantime. For an ad to be accurately recalled in a DAR test, it must truly “break through.” In a diagnostic test, it must only be identified as something seen a few moments earlier.
- Not surprisingly, ads that recall well in a small diagnostic clutter environment may not do well in a rigorous DAR test. Some research has indicated that established, familiar brands are more highly recalled in diagnostic tests while no such pattern has been detected in the 60+ years DAR tests have been used and studied. While branding may be recognized in a diagnostic test, the linkage between a brand and the ad concept may not be sufficiently strong for respondents to recall the ad in the name of the advertiser after 24 hours have passed.
There is a place in communications evaluations for both DAR tests and diagnostic evaluations. Diagnostic tests help spot when key messages fail to communicate or when an ad’s tonality is not consistent with a brand’s image. These tests are useful for screening and refining alternative concepts early in the developmental process. But diagnostic tests do not assess how the ad will perform in a real world environment nor do they provide essential insight into whether the advertisement is capable of breaking through sufficiently with consumers that a meaningful impression has been formed that can last until some action can be taken. One research expert has estimated that close to 70% of CPG advertising plays out with near-zero ad effects and blames this largely on advertising that has failed to break through.
Advertisers have opted for diagnostic tests in recent years for a variety of reasons – low cost and quick turnaround prominent among them. By saving money and time, many have risked multi-million dollar media budgets on messages that leave both Michael Imperioli as well as their target customers wondering what that advertising was all about.
To learn more about G&R’s real-world testing solutions, go to Real-World Testing, and, to learn more about G&R’s better online system for measuring breakthrough, go to Web24.
Comments are closed.