Step 5

How To Do It

Excerpts from CDC's Framework for Program Evaluation in Public Health discuss data sources for evaluation purposes:

Sources of evidence in an evaluation are the persons, documents, or observations that provide information for the inquiry. More than one source might be used to gather evidence for each indicator to be measured. Selecting multiple sources provides an opportunity to include different perspectives regarding the program and thus enhances the evaluation's credibility. An inside perspective might be understood from internal documents and comments from staff or program managers, whereas clients, neutral observers, or those who do not support the program might provide a different, but equally relevant perspective. Mixing these and other perspectives provides a more comprehensive view of the program. The criteria used for selecting sources should be stated clearly so that users and other stakeholders can interpret the evidence accurately and assess if it might be biased. In addition, some sources are narrative in form and others are numeric. The integration of qualitative and quantitative information can increase the chances that the evidence base will be balanced, thereby meeting the needs and expectations of diverse users. Finally, in certain cases, separate evaluations might be selected as sources for conducting a larger synthesis evaluation.

Information collected for answering the evaluation questions and for comparison with the activity standards can be derived from a number of qualitative and quantitative data collection methods:

Key informant interviews
Focus groups
Community surveys
Morbidity, mortality, or disability statistics
Behavioral risk factor surveillance reports
Empirical findings reported in scientific literature
Statistics on diseases for which notification is required
Disease registry statistics
Hospital admission and discharge reports
Life-expectancy tables
Data on years of life lost
Police accident reports
Information about health services, including eligibility restrictions and service capacity

What is most important to remember is that your data collection methods should be suited to answering the evaluation questions.

The chart below indicates some major differences in qualitative and quantitative data:

CORRESPONDING DIFFERENCES BETWEEN
QUALITATIVE AND QUANTITATIVE DATA COLLECTION METHODS

QUALITATIVE METHOD	QUANTITATIVE METHOD
Primarily inductive process used to formulate theory	Primarily deductive process used to test pre-specified concepts, constructs, and hypotheses that make up a theory
More subjective: describes a problem or condition from the point of view of those experiencing it	More objective:provides observed effects (interpreted by researchers) of a program on a problem or condition
Text-based	Number-based
More in-depth information on a few cases	Less in-depth but more breadth of information across a large number of cases
Unstructured or semi-structured response options	Fixed response options
No statistical tests	Statistical tests apply
Can be valid and reliable:largely depends on skill and rigor of the researcher	Can be valid and reliable:largely depends on the measurement device or instrument used by researcher
Time expenditure lighter on the planning end and heavier during the analysis phase	Time expenditure heavier on the planning phase and lighter on the analysis phase
Less generalizable	More generalizable

Qualitative methods typically include direct observation, key informant interviews, and focus groups discussions (See Tools for Research). Listed below are some examples of qualitative methods that can be used to collect information pertaining to why people in affected areas are not adhering to mosquito control measures during a West Nile Virus epidemic:

Interviewing public health and public works officials in the affected area
Interviewing individuals at random in a public place in the affected area (e.g., a shopping mall) for short, intercept interviews
Tracking conversations in Internet chat rooms on West Nile Virus
Conducting focus groups among home and business owners in the specific areas where cases have been diagnosed

Qualitative methods may be especially helpful under certain circumstances:

A need to add depth, detail, and meaning to empirical findings
A need for information to help develop the evaluation plan
A need for descriptive information to document how a program in the formative stage is developing or progressing
A need to delineate the program's theory of action (e.g., logic model)
A need to emphasize individualized outcomes that are expected to be different for each program participant
A need to analyze the content of text-based information like newspaper articles
A desire to capture the context of the setting where a program is implemented
A desire to ask open-ended questions
A desire to obtain the perspectives of a variety of individuals who have different relationships to the program (e.g., patients, caregivers, students, teachers, victims, perpetrators, community leaders, nonparticipants)
A desire to know more about the problem or condition that the program is designed to address
A distrust of prevailing theory or a desire to generate a new theory about how to address a particular problem or condition
A desire to corroborate one method with another
A desire to document how program inputs/actions (i.e., the means) relate to outputs/effects (i.e., the ends) (e.g., how program staff believe their actions lead to certain outcomes)
A desire to gather in-depth information on unique cases (e.g., unusual successes or failures, dropouts) of an individual person, program, organization, or community.
Decision-makers' desire to understand the dynamics of a program's operations without personally observing the program in operation
Program staff's desire for detailed descriptive information to help improve the program
A desire to document differences (e.g., input, implementation, content, politics, context, outcomes, and program quality) across different sites included in a multi-site program
Concerns about program quality that requires description
A need for unobtrusive observation
Program goals written vaguely, generally, or nonspecifically
Concerns about unanticipated outcomes or unexpected side effects
A study focuses on process (i.e., how something happens) rather than on outcomes

Adapted from Herman, JL, Morris, LL, Fitz-Gibbon, CT (1987); Patton, M.Q. (1990); Crabtree BF, Miller WL ( 1992) Doing Qualitative Research.

Data can be collected on public health communication activities by using numerous quantitative methods, including conducting community surveys, abstracting clinic or hospital records, and analyzing data derived from vital records. Listed below are other examples of quantitative data collection methods:

Administering a survey to determine risk behaviors
Administering telephone surveys before and after airing paid or public advertising to determine differences in targeted attitudes, perceptions, or behaviors
Inspecting clinic encounter forms to determine reasons for visits
Examining vital records to determine causes of death
Using a surveillance system to determine demographic characteristics of individuals with a specific diagnosis
Examining clinic records to determine incidence/prevalence of a particular disease or to determine reinfection rates

The use of both qualitative and quantitative methods will provide a means of cross-validation that results when two or more maximally different measures of the same construct (i.e., trait) agree (i.e., have a similar response pattern). In addition, the methods used to collect more qualitative data (e.g., key informant interviews, focus groups, and direct observation) will provide background (i.e., contextual) information helpful in interpreting quantitative data collected with a survey (See <Tools for Research> for additional information on both qualitative and quantitative data collection methods).