Evaluation Relevant Steps in CDCynergy
(Click on the Phase or Step titles to return to the tutorial.)
|
CDCynergy Step |
Step Title |
Evaluation Relevance |
| Phase 1: Problem Description | ||
| Write a problem statement. | The initial health or safety problem you describe is often the sole or key long-term outcome in the evaluation. In rare cases, a social marketing program is so large and has such a direct, immediate impact that the secondary epidemiological data used to define the health problem can be used later as a pre-program baseline measure. This benchmark can then be compared to data from similar measures gathered after the program has been implemented to assess the impact of the program. But even if your program (like most) eventually targets a behavioral cause of the health problem instead of the whole problem, trends in the ultimate health problem will be important context for evaluation findings. | |
| List and map the causes of the health problem | This first draft of a complete causal model of the problem helps shape market research (or formative evaluation) which, in turn, guides intervention design. The causal model will also be reflected in the logic model (My Model) that shows how you expect your intervention to work. Eventually, the full problem analysis chart will help an evaluator determine whether your intervention really had an impact on its targeted outcomes by ruling out other causal factors that could be responsible for the observed effects. | |
| Identify potential audiences. | The identification of potential audiences narrows the universe of evaluation participants (e.g., to those who speak Spanish and have low literacy skills). Based on experience conducting research with the audiences, the evaluator may suggest some adjustments in the list. | |
| Identify models of behavior change and best practices. | Evaluators are well able to help with this task because they have behavioral science backgrounds. They can review literature for best practices and help select theories to guide market research. The models chosen will guide evaluators in choosing behavioral determinants for their assessments and research designs. | |
| Form your strategy team. | Include an evaluator on your team now if you don’t have one on board already. This individual can anticipate evaluation needs and issues and can offer relevant advice at appropriate points in the program planning process. For example, the evaluator can help you frame realistic, measurable program objectives. The other individuals on the strategy team could be viewed as evaluation stakeholders. | |
| Conduct a SWOT analysis. | This analysis is a form of evaluation in which you identify and gather information about program strengths, weaknesses, opportunities and threats, all of which could influence its implementation and acceptance. The SWOT can give evaluators insight into factors that may undercut or enhance the program’s effects and the potential for detecting them. | |
| Phase 2: Market Research | ||
| Define your research questions. | Basic social marketing concepts, best practices, and behavioral science theory should guide the selection of questions for market research (also called formative evaluation). An evaluator may have experience framing questions in terms of behavioral science theory. Also, in some cases, market research on behavioral determinants can provide not only formative information, but also baseline data for an outcome evaluation. | |
| Develop a market research plan. | Evaluators can assist you in identifying methods for the most useful market research and in writing a clear, logical research plan. | |
| Conduct and analyze market research. | Draw on the research background of the evaluator as you manage the market research. | |
| Summarize research results. | Evaluators are good at summarizing and reporting data. You may want to get comments from your evaluator on draft research reports, or have your evaluator help you write the reports. Your evaluator should review and understand market research findings. The evaluator can help you and your stakeholders understand the implications of the research findings for your market strategy and its evaluation. For example, the findings may support the selection of certain target audience segments or behavioral objectives, and may have the potential to guide the process of tailoring programs to audience needs and preferences. | |
| Phase 3: Marketing Strategy | ||
| Select your target audience segment(s). | The selection of target audience segments has major implications for the evaluation, ranging from interview methods to cost. Evaluators can help you define audience segments not only in terms of demographic variables, but also in terms of psychosocial or “psychographic” variables. They can also tell you whether a potential segment can be distinguished from others, an essential element in such evaluation tasks as measuring the proportion of the target audience segment that is eventually exposed to program components. | |
| Define current and desired behaviors for each audience segment. | The desired behavior is the major program outcome to be measured by the evaluation. | |
| Step 3.3 | Describe the benefits you will offer. | Benefits should be measured in the evaluation as a behavioral determinant. |
| Write your behavior change goal(s). | The behavior change goal for the entire program is a general directional statement about behavior. It is at the top of a pyramid of more specific standards against which program performance can be compared. The goals of the component intervention activities must add up to the overall behavior change goal. Then each of the subgoals will be translated into exposure/reach and outcome objectives which are measurable and quantified. An evaluator will determine whether the program has a detectable effect in the intended direction, as stated in this overall program goal. | |
| Select the intervention(s) you will develop for your program. | Decisions made here must be documented and shared with evaluators so that they understand the rationale for the intervention activities that they will evaluate. Evaluators may point out that some “candidate” interventions can be evaluated more credibly than others. This consideration may factor into your intervention choice. The selection of interventions has enormous bearing on the design of the outcome evaluation. | |
| Write the goal for each intervention. | Evaluators will compare the effects of the program against each of these subgoals to determine whether each component intervention is having a detectable effect in the intended direction. They can help you articulate clear subgoals that add up to your overall program goal. | |
| Phase 4: Interventions | ||
| Select members and assign roles for your planning team. | Having contributed to program strategy, an evaluator continues to be a valuable member of the intervention planning team. At this point, you may want to make other staff aware that they are likely to play a role in the evaluation and/or recruit additional partners that will be needed for this effort. | |
| Write specific, measurable objectives for each intervention activity. | Outcome objectives convey specifically how much your program must accomplish to be considered successful. Specific, quantified outcome objectives provide the evaluator with standards against which to judge the effects of various intervention activities. Your evaluator can help you set realistic objectives, thus helping to build in a positive evaluation result. | |
|
|
Write program plan, including timeline and budget, for each intervention. | When you have planned your intervention processes, SMART process objectives can be set. They will be the standards against which process evaluation data is judged. You will estimate the time and resource requirements of the evaluation in the next phase, so your budget and timeline should be considered provisional until then. Some of your program resources may be able to do double duty by supporting the evaluation, and this overlap should be described in your budget narrative. Your evaluator can anticipate some of what will be needed before final decisions about the evaluation are made in the next phase. |
| Pretest, pilot test, and revise as needed. | The evaluator can assist with designing relevant tools, such as recruitment screeners or interview/moderator guides with questions to assess performance of activities, concepts/messages, products or materials. He or she should help to ensure fidelity to program strategy, particularly with regard to the planned emphasis on particular behavioral determinants like the perceived benefits of your program offering. | |
| Summarize your program plan and review the factors that can affect it. | It is very important for your evaluator to understand how program components are supposed to hang together to bring about the goals and objectives you set for your program. This will affect the way overall program exposure is measured. If certain intervention components account for the bulk of program investments, they may warrant special evaluation attention. As mentioned before, the evaluator should understand the factors that may affect the program so evaluation findings are interpreted within context. | |
| Step 4.6 | Confirm plans with stakeholders. | Stakeholders should understand that program plans will be updated after the evaluation is designed. Let them know that they will be consulted at some length about their information needs in the next phase, and ask them to begin to gauge their levels of commitment to evaluation activities. |
| Phase 5: Evaluation | ||
| Identify program elements to monitor. | See "What to do" and "How to do it" sections of each step. | |
| Step 5.2 | Select the key evaluation questions. | |
| Determine how the information will be gathered. | ||
| Develop a data analysis and reporting plan. | ||
| Phase 6: Implementation | ||
| Prepare for launch. | Final baseline data should be collected before programs are rolled out. | |
| Execute and manage intervention components. | Evaluators should document any unexpected events that may affect the launch of the program, and the programs initial impact. | |
| Execute and manage the monitoring and evaluation plans. | See "What to do" and "How to do it" sections. | |
| Modify intervention activities, as feedback indicates. | Evaluators should make note of when program elements are scheduled for change and, if possible, collect data before major changes are made. These data can serve as baseline information for assessing the effects of the modified program. | |