Kristen Weeks-Norton on "Planning/Monitoring"

As a program practitioner, the reason you want to monitor continuously is because you want, for one thing, quality control. You want to be able to look at all of this hard work that you've put into your program and make sure it's coming out on the other end the way you had planned it. So, for example, if you're creating print materials, you're creating posters, if distribution isn't happening according to plan, you're only going to find that out through your monitoring.

If posters are folded and they're not coming out in a way that someone feels good about putting the poster up, then they're not going to use the product. So we want to make sure that we continue with the quality of the products that we planned. Also you want to be able to make those mid-course corrections so that you can continue the good quality of your program, so that goes back to the example if your posters aren't being put up in the schools, then you need to know -- and your aim is to get them up in schools -- you need to be able to make those corrections so that they are getting out so the target audience is seeing them.

And you also have stakeholders that are involved in this process. They want to know what's happening with the materials, so you need to deliver to them information that's telling them what are the numbers of young people that are completing workshops successfully?

I think that as we're looking at using social marketing, part of the process is this whole continuous loop of feedback. When we've got a continuous loop, we don't want to stop necessarily because we've found out information that may not be as complimentary to our program. Instead what we want to do is take that information, and the positive spin to that is, how do we then increase the numbers or continue to maintain the program and increase the numbers? And so you don't look at the information that you get back that tells you something about your program isn't happening the way you want it to; you don't take that information and use it in a negative fashion, but rather you're spinning it into your process so that you can then use it in an effective manner and that is to make your program more successful, make your program more effective.

…if you have a hotline, we're going to count the number of calls and we're going to compare it to whether or not we've had exposure in certain areas and whether that has brought in more costs. So if we have radio ads playing at the same time, we watch the hotline numbers spike. But each one of these interventions really needs to have their own tool in order so that we can find out how that particular piece is doing. So we use that to pull back into the program to make that program piece stronger.

…we had a multi-component program. And so we have a number of interventions within one campaign. One was a workshop, so we evaluated a pre- and post-survey and a follow-up survey to find out what was the impact of that workshop on a young person and their intentions to behave. We also had a hotline that was part of the program, so we looked at the number of calls that people made to a hotline, the amount of time they spent on the phone, and how much time was spent in a particular area that they could select. In other words, what did they want to listen to? Do they want to have a little HIV 101, or do they want to have a Communication 101 or do they want to hear about condoms and where they can find them? Or do they want to hear their peers talking about where do they keep condoms, how do they carry them around? And then we also had, let's see, print collateral materials, and so we wanted to know how were those distributed? What was the number that went out? How many were placed? How many were taken down, and was that a bad thing or a good thing that people were taking the posters? So we have a very specific strategy for that as well, for the collateral materials. We could find out how each one of our interventions were doing, how relevant they were to the target audience, and how did we need to change those to even make them more relevant?

We are continuously pulling in information. Continuously pulling in information about the program is the key to making it the best program it can be. And making sure that the program is relevant to your target audience, so you never want to stop monitoring.

So you might identify a set of components or interventions that will be your social marketing strategy -- part of your plan that you're going to lay out -- and you might continue that for a period of time; but then you might find out a lot of the young people, for example, might be accessing the internet, so should you develop a website? So you might develop new interventions as a part of an overall strategy to reach a certain target audience, and certainly you wouldn't want to stop monitoring those pieces so you would need to add them into your evaluation plan and follow those pieces to make your program most successful.

When you talk about things that are hard to monitor or what's the hardest part about monitoring something, there are times when you have unintended consequences -- things that result out of a campaign that you never imagined would have resulted, but you don't want to leave that kind of rich data out of information that is the lessons learned in the evaluation of your program.

I think also I have a couple of examples that are probably shifting midstream, where you started off with one product and then had to change to another; and it wasn't necessarily -- it was one of those things that wasn't necessarily complementary. For example, we had a condom pack; we had a safe sex pack. It was made out of cardboard that folded up. What we didn't know was when it popped open, all the contents came popping out. Well, that didn't really lend itself to young people being comfortable about carrying these around and keeping condoms handy. In our campaign, the behavioral objective was to use condoms and part of that was keep condoms handy and be able to talk about them, be able to use them effectively. So if you have something that pops open like that, that's not going to lend itself to that. So we had to in mid-course change and come up with a different packaging idea.

And make sure that it was still something that we pre-tested and it was relevant to our target audience and it made sense and it was something that they could carry. And that change was pretty dramatic. It included a completely different way of packaging.

...one other example that comes to mind is the hotline. We set up a hotline for young people so that they could get information by calling a toll-free number and find out what they wanted to know about, and they could select what it is that they wanted to know.

What we didn't know was that parents also called these lines because they want to know what their teen is listening to, and they want the same information. They want to be able to talk to their son or daughter or young person that they're involved with to be able to participate in this conversation. And, as a result then, we put in a parent piece or component to the hotline so when you called you heard a teen talking to you and you could select going to the teen boxes of information, but if you were a parent you could select going to a parent menu as well. So we were offering something to our secondary audience to say "we want you to be involved in this campaign too."