However beautiful the strategy, you should occasionally look at the results.

Winston Churchill

Think of a pilot as a “test run,” an opportunity to work out the bugs before committing to carrying out a strategy broadly. As noted in the chapters on developing strategies, the tools you elect to use in your strategy should be selected based upon the barriers and benefits that you have identified. To pilot the strategy introduced in the previous chapter, store managers of two supermarkets would be approached and asked if they would be willing to participate. The two stores would need to be similar both in the demographics of their shoppers as well as in the products available (two stores of the same chain would be good candidates). By the flip of a coin, one of the stores would be randomly assigned to receive the community-based social marketing strategy, while the other would serve as a comparison or what is referred to as a control, and would not receive the strategy.

Prior to piloting the strategy, the rate of purchase for recycled-content products would be determined by examining the computerized inventory records for these items. Note that it is important to collect this data from both stores, since they may differ initially from one another in the rate of purchase for recycled-content items. Also, it is important to obtain this baseline data for a sufficient period of time to ensure that it is indicative of normal purchase habits, rather than seasonal fluctuations. Following this baseline period, the prompts, posters, stickers, pamphlets and video kiosk would be introduced in the intervention store. After introducing the strategy, the rate of purchase of targeted items would be monitored for several months at both stores to ascertain whether the strategy increased the purchase of these products.

To determine whether the strategy alters consumer purchases, the purchase of recycled-content products during the baseline period is compared to purchases during the intervention (seasonal adjustments may need to be made to these numbers to control for increased purchases around events such as Christmas). However, the success of the strategy cannot simply be determined by comparing the purchases of the recycled-content items for the two stores.

The following example clarifies how to correctly determine the impact of the strategy. Imagine that after implementing the above strategy, the intervention store had sold 5000 units of recycled-content toilet paper, while the control store had sold only 3000. On first glance, it appears that the community-based social marketing strategy has brought about a 67% increase in sales for this one item. However, such a conclusion assumes that the stores initially sold an equal amount of recycled-content toilet paper, which is very unlikely. To determine the real impact of the intervention, the sales of toilet paper during the baseline period for both stores needs to be considered. Imagine that baseline data revealed that the intervention store had sold 2500 units of recycled-content toilet paper in the month prior to the implementing the strategy, and the control store had sold 2000 units. As shown in the preceding table, the real increase in sales that can be attributed to the intervention is 50%.

Intervention: 5000-2500 = 2500

Control: 3000-2000 = 1000

Real Impact: 2500-1000 = 1500 (50% increase)

If, when comparing inventory records prior to and following the implementation of the intervention, little or no change in consumer purchases was observed, then the pilot would need to be revised until significant changes in behavior were found. Since in this strategy the prompts were a central aspect of the campaign, it is natural to start by investigating them. By conducting in-store surveys with a random selection of shoppers, awareness and understanding of the prompts could be probed. If low recognition and understanding of the prompts was found, then the prompts would need to be redesigned to be more prominent and clear. Further, the placement of the posters, that explain the purpose of the prompts, should be examined. Did shoppers recall seeing the posters? Did they know what the posters said? If the answer to either of these questions is “no,” it is possible that simply changing the location and/or number of posters might address this problem. Finally, a central element of this community-based social marketing strategy was to foster social norms that support the purchase of recycled-content products. Accordingly, the post-pilot survey should also address whether shoppers believed they should be purchasing recycled-content products. If this is not the case, then the social norm aspects of this program would need to be refined.

The point of a pilot is to identify and address problems before launching a campaign throughout the community. You should plan on there being problems and build into your plans the opportunity to refine your strategy until it works well. On one project, I revised a pilot six times before I was able to produce the desired behavior changes. While it was frustrating to have to make this many revisions, I was thankful that I was making the revisions to a pilot rather than to a larger project, for which the problems would have been much more difficult and expensive to rectify. Expect problems, plan for them; in the end, when you implement community-wide you will be rewarded for the time that you took to troubleshoot your strategy through piloting.

Piloting principles

Use the following principles as guidelines in conducting a pilot:

  1. Don’t mix barrier and benefit research with piloting: It is tempting to use the participants from your barrier and benefit research in a subsequent pilot, but avoid this temptation. The act of participating in focus groups or surveys often leads to greater awareness of the issues you are working on. If you include these people in your pilot, and your pilot is successful, you will not know whether the changes in behavior you observed were due to their participation in the barrier and benefit research, the strategy you employed, or a combination of the two. If the changes are due to either having participated in the barrier and benefit research, or a combination of participating in the barrier and benefit research and your strategy, your program will fail when you move from piloting to broad-scale implementation since these conditions will no longer exist. This concern is greatest when working with small numbers, such as local businesses or farmers in a watershed. In these cases, conduct your barrier and benefit research in similar watersheds or communities in order to avoid the problems noted above.
  2. Use a Minimum of Two Groups to Conduct your Pilot: When you conduct your pilot, you want to make sure that any changes you observe are the result of your intervention and not other events that are occurring in the community. To be certain that it is your intervention that is bringing about the changes you observe, always include a control group to which no strategy is delivered. By comparing your intervention and control groups, you can be much more confident that your intervention was responsible for any changes you observe. You may wish to have more than two groups. For example, as in many of the studies described in this book, you may wish to have one group receive a commitment strategy, a second receive feedback, a third receive a combination of the two, and a fourth act as a control. Keep in mind that pilots can often be quite inexpensive to conduct since the size of groups can be kept small. Including multiple groups in your pilot can help you determine the form that your strategy will take when you implement it across your community. For example, as a result of conducting a pilot on fostering car pooling, you may learn that obtaining commitments provides no additional benefit over assisting employees to identify others living in their neighborhood with whom they might drive to work. As a result, your subsequent program would drop commitment as part of the strategy.
  3. Use Random Assignment: When you conduct a pilot, you want to know that the group that receives your intervention is as identical as possible to the group that serves as the control. The only way that you can assure this is if the participants are randomly assigned to one group or another. To randomly assign farmers, businesses, individuals or households to the groups you plan to use, simply place all of their names or addresses in a hat and then pull them out assigning the first selection to the first group, the second to the second group, etc.
  4. Make Measurements of Behavior Change a Priority: In evaluating the effectiveness of a pilot, your primary concern should always be whether you have been able to change the behavior that you set out to change. Where possible, don’t rely upon people’s self-reports of their behavior; they can be unreliable. Obtain water records, ask to look in composters, examine weather-stripping, etc. You will also want to examine people’s perceptions and attitudes, but don’t substitute these for examining actual changes in behavior.
  5. Calculate Return on Investment: In conducting a community-based social marketing pilot, we want to know that not only have we effectively changed behavior, but that we have done so cost-effectively. To calculate return on investment, follow these guidelines from Nancy Lee:1

    A. Dollars Spent: Includes all costs associated with developing and conducting the pilot.

    B. Behaviors Influenced: Number of people who engaged in the targeted behavior.

    C. Cost per Behavior Influenced: Divide Dollars Spent (A) by Behaviors Influenced (B).

    D. Benefit per Behavior: Costs avoided, such as health care costs, by encouraging more active forms of transportation. Avoided costs can at times be difficult to estimate.

    To calculate return on investment:
    Step 1: Gross Economic Benefit = B * D
    Step 2: Net Benefit = Gross Economic Benefit - A
    Step 3: ROI = (Net Benefit/A)*100

    It is worth noting that the rate of behavioral change is not the only determinant of what program you might implement across a community. By calculating return on investment, you might learn that one behavior change strategy results in only a slightly lower adoption rate of the behavior than a more expensive strategy, but at only a fraction of the cost.

  6. Revise your Pilot Until it is Effective: It is tempting when a pilot is ineffective to assume that you know what went wrong and to move directly to community-wide implementation. Keep in mind that pilots can often be conducted very quickly. Take the time to run another pilot to confirm that you are actually able to change behavior before you implement across a community. The extra time that you take to run the pilot may save you hundreds of thousands of dollars and, possibly, your job.

Next Chapter » Step 5: Broad-scale Implementation