Statement of Need | Outcomes and Evaluations | Strategic Communications Audit | Opportunities and the Road Ahead
A Designer's Foundation | History | Mission | Vision | Board of Trustees | Advisory Panel | Contact
Knowledge Development: There is an acute need for knowledge development and dissemination about how to approach this work . There is also a very real possibility that the demand for this information could increase quickly in the near future.
As Gary Henry at Georgia State University said,
I think we could see a very strong ramping up foundations want to test their effectiveness, because the evaluation community will respond to that market. But we are in the very beginning stages and our tools are vastly deficient.1
I think we could see a very strong ramping up foundations want to test their effectiveness, because the evaluation community will respond to that market. But we are in the very beginning stages and our tools are vastly deficient.1
Cross-Sector Field Building: This work brings people working on similar issues a reason to come together to share ideas, strategies and action-items. The consortium we have already put together represents different nonprofit sectors in different markets at all levels if funding. It will continue to encourage and support collaboration with the nonprofit sectior and public-at-large, encouraging greater awareness of needs and access to services.
Theory Building: Theory is critical in these endeavors. A lot is already out there, but because it comes from many different disciplines, most evaluators do not have a good grasp of what is available. In addition, there is a need for more theory development and theory integration, particularly for public will campaigns. One way to do this is to borrow from the example set by the public health field.
In 1991 while trying to battle the growing AIDS epidemic with prevention campaigns focused on behavior change, the National Institute of Mental Health organized a "theorists workshop" that brought together the developers and/or leading proponents of five of the most seminal theories on behavior change. Their task was to come up with a finite set of variables that should be considered in any behavioral analysis.
Their purpose was to facilitate the development of theory-based interventions and campaigns that would prevent the behaviors that expose people to and spread HIV. They accomplished their task and came to consensus on eight variables that appear to "drive" or account for most of the variance on any deliberate behavior (Fishbein, et al., 2001).
Campaign Theory, Outcomes, and Methods Education: Part of the problem with public communication campaign evaluation is a lack of awareness among campaign practitioners, evaluators, and their sponsors about what outcomes and methods are appropriate and available. For evaluators, unless they have a communications background (and most do not), they would not encounter outcomes like media exposure or measures like media impressions in their other evaluation work. For campaign practitioners and sponsors, it is critical to understand the difference between process measures, outcomes, and impacts and what it is reasonable to expect from campaigns and from their evaluations.
As John Bare at the Knight Foundation put it:
Program Directors, who are trying to develop support for [campaigns], speaking of foundations specifically, need to realize that they can't treat these as separate from regular program interventions. It's one thing to say that we are doing a teen pregnancy program, let's look at how research says these things work. When you look at a communications campaign it's easy to assume that you don't need to reflect upon the science because it's just advertising. We just put the messages out there and good stuff will happen. But there's just as much science to consider on these kinds of things. And so we need training and/or materials targeted specifically to people who shepherd grant programs forward. So at the front end of the development they seek out and are not threatened by seeking out the kind of assistance that will strengthen [the campaigns].
Program Directors, who are trying to develop support for [campaigns], speaking of foundations specifically, need to realize that they can't treat these as separate from regular program interventions. It's one thing to say that we are doing a teen pregnancy program, let's look at how research says these things work. When you look at a communications campaign it's easy to assume that you don't need to reflect upon the science because it's just advertising. We just put the messages out there and good stuff will happen. But there's just as much science to consider on these kinds of things. And so we need training and/or materials targeted specifically to people who shepherd grant programs forward. So at the front end of the development they seek out and are not threatened by seeking out the kind of assistance that will strengthen [the campaigns].
Knowledge development for education purposes might take the form of a series of working papers or short briefs. It might also mean being opportunistic with meetings or conferences where interested parties gather. The same principles that apply to campaigns would apply here – the point would be not only to raise awareness, but also to give people the tools they need to act and put the ideas into practice.
Evaluation Tools: Evaluation tools are aids for campaign practitioners and evaluators who are struggling with evaluation design and implementation issues.
Decision Aids or Principles:
Evaluators, campaign designers, implementers, and sponsors would
benefit from the development of aids or principles that can help with
decisions about the possibilities available for evaluation and the
consequences or opportunities for making certain choices. They would
not provide definitive answers, but would help guide choices for
questions like: What utility should the evaluation serve ( e.g.
accountability, continuous improvement)? Where should most of the
evaluation effort go (formative, process, outcome, impact evaluation)?
What level of resource investment (money, time, staffing) will be
needed to get the job done? What are reasonable expectations for the
campaign and what are short-term and intermediate measures of progress?
Planning and Design Tools: Ideally a campaign and its evaluation should be designed at the same time. Too often the evaluators are called in once the campaign has been set up. This allows them to establish together the important theory that underlies the campaign and the outcomes of interest, and for evaluators to inform the design process with formative research. Campaign designers and evaluators need planning tools that can assist them in this process and help them speak a common language. They need tools that respond to John Bare's concern that "We need stronger theory-of-change models connecting the activities to the desired change."
Data Access and Analysis Sophistication: Nonprofits typically cannot afford the market data that for-profits or advertising agencies use. Also, data sets typically do not meet the needs of nonprofit campaign planners and evaluators. More funding and resources can help nonprofits access better data. Finally, while more campaigns are using quantitative data for planning, there is much room for improvement in data analysis design and sophistication. For example, audience tracking and segmentation are commonplace in the private sector, but rare among nonprofits.
Planning and Design Tools: Ideally a campaign and its evaluation should be designed at the same time. Too often the evaluators are called in once the campaign has been set up. This allows them to establish together the important theory that underlies the campaign and the outcomes of interest, and for evaluators to inform the design process with formative research. Campaign designers and evaluators need planning tools that can assist them in this process and help them speak a common language. They need tools that respond to John Bare's concern that "We need stronger theory-of-change models connecting the activities to the desired change."
Data Access and Analysis Sophistication: Nonprofits typically cannot afford the market data that for-profits or advertising agencies use. Also, data sets typically do not meet the needs of nonprofit campaign planners and evaluators. More funding and resources can help nonprofits access better data. Finally, while more campaigns are using quantitative data for planning, there is much room for improvement in data analysis design and sophistication. For example, audience tracking and segmentation are commonplace in the private sector, but rare among nonprofits.
Exploratory Evaluation Investments: These ideas channel the exploratory spirit of research and demonstration projects, without the non-participatory or thoroughly academic tendencies that go along with them. The appeal is for evaluation investments that lead to new and valued learning and implementation.
Experiments or Quasi-Experiments:
While they can be a challenge to set up and can be expensive,
experimental or quasi-experimental evaluation designs offer the most
promise for learning more about what works. Investments in
quasi-experiments that would allow campaign designers and evaluators to
vary their models and learn from different approaches. One example
could be marketing for symphony orchestras to get people to buy
tickets. One approach would be to randomly mail two incentives to two
groups of potential buyers to see which yielded the most ticket buying
at the least amount of cost. That approach can be particularly
effective if theory or past practice helps program designers learn how
to "match" treatments with individuals most likely to be moved by them.
Participatory and Learning Evaluations: This is a call for sponsors to be open to diverse models of evaluation, including participatory evaluations that develop along with the campaign, stick with it throughout its implementation, and regularly have a learning exchange with campaign designers, implementers, and funders. It requires that the evaluation and campaign plan be done at the same time. And it requires that the evaluation team stay with the campaign over time.
Participatory and Learning Evaluations: This is a call for sponsors to be open to diverse models of evaluation, including participatory evaluations that develop along with the campaign, stick with it throughout its implementation, and regularly have a learning exchange with campaign designers, implementers, and funders. It requires that the evaluation and campaign plan be done at the same time. And it requires that the evaluation team stay with the campaign over time.