Assessing impact on participants takes place throughout a program, not just at the end. Indeed, some measures such as attendance or enrollment and methods such as experimental design have to be in place before the program begins. Plan an evaluation by following these steps:
This decision drives the evaluation. What you want to know is whether participants achieved any of the outcome goals. Review these goals as you plan and implement your evaluation so you can maintain focus on the bottom line: participants’ physical, mental, and social health. Craig Dreeszen lists these key questions:
Assessing outcomes also informs evaluation of how the program affected personnel, budget, and partnerships. If the program is a success for the participants, it softens the blow of higher-than-anticipated expenses or mitigates the impact of grumpier-than-usual staff members or partners. Questions include:
To delve deeper into the partnership dynamics at the conclusion of your program, revisit the checklist.
Another answer to the question “What do you want to learn?” is the impact of the community sharing of the art created during the program. If external validation of accomplishment is beneficial to older adults’ experience of mastery, then the audience’s reaction to a performance or exhibit is important to measure. Questions should explore basic reactions to the art, perhaps by asking respondents to circle descriptive adjectives. More important, assess the change in audience members’ perceptions about the capabilities of older adults.
If you are working with other organizations, determine the specific questions in collaboration with the partnership team. Remember to raise this topic with the major funders. Many of them have a particular interest in evaluation and a penchant for certain kinds of information.
A common pitfall in figuring out what you want to learn is focusing solely on the outcomes that pertain directly to your field. Aging services organizations need to evaluate older adults’ skill development and learning in the arts; arts organizations need to evaluate older adults’ physical, mental, and social development.
Don’t devote more time, energy, and money to evaluation than you did to implementation. Reflect on these issues:
Be clear about the purpose of the evaluation. Ask where does information already exist. Consider how easy is it to get and use the information. Do you have to reorganize the data? Would less-straightforward measures work as well? Is there some information that we can get at easily enough, well enough to make competent decisions about the program? 96
With respect to cost, some experts suggest devoting 5 to 7 percent of the total program budget to evaluation. Others say 10 to 15 percent. Expenses include:
Obviously, spending this amount of money isn’t always possible. Fortunately, there are options. Start modestly with some basic quantitative measures and a few qualitative questions on a survey form. Or conduct a pilot evaluation that utilizes a variety of more sophisticated measures to assess the impact on a small number of participants. In subsequent years, increase the scope of your evaluation as you gain experience (and funds).
Music for All Seasons conducted a pilot project in collaboration with the New Jersey Neuroscience Institute at JFK Medical Center to measure the quality of life among 23 nursing home residents with dementia. Staff tracked the number of falls, number of times sedating medication was given, and quality of life using the Quality of Life in Dementia scale. Using a repeated measures design, the participants were tested before and after two live music performances given six weeks apart. Each set of measures was taken at two-week intervals. The principal result was a measured increase in quality of life. Scores increased significantly following each performance and maintained this increase for as long as six weeks.
Typical audiences are:
Not every one will be interested in the same type of information. Indeed, each group has different responsibilities and interests. Focusing on the needs of partners, Stephanie Golden notes:
Hospitals and nursing homes . . . need process notes documenting whether a patient’s ability to interact improved and whether the workshop stimulated [his or] her creative imagination. Schools require a portfolio of each student’s work, including both art projects and their journals. The teacher uses the portfolio to assess the students’ learning in the program. Senior and community centers may only want a report based on the survey information, or they may request a record of attendance, skills learned, and works produced. 97
Participants are most likely to be interested in qualitative information—the anecdotes and images that provide tangible evidence of a productive and happy experience. Members of the media, too, will want photos and stories with a human interest “hook.” Funders and your board of directors, who are more interested in the program’s success, may prefer quantitative data. Community leaders will probably want both to provide content for speeches and show constituents that they support older adults. Catering to all of these audiences doesn’t mean conducting different evaluations or distorting the results. It simply means emphasizing different information in each report.
Don’t overlook the importance of broadening the circle of those who receive the results of your evaluation. Sharing information and knowledge with each other contributes to the growth of the arts and aging field. In addition, evaluation results provide the content for public awareness efforts and the ammunition you sometimes need within your own organization.
The basic document is a comprehensive, written, final report that includes all of the evaluation results and an executive summary. Adapt this report to produce:
Optimize your time by writing some of these pieces shortly after completing the final report so that you can respond quickly to requests for details or just follow up on any expressed interest.
You don’t have to wrestle with evaluation on your own. In addition to assistance from the partnership team or staff members, identify an evaluation partner such as a university, or hire a consultant. You are more likely to need external help if your evaluation is sophisticated. Sometimes an outside expert enhances the credibility of your results. It is important to decide early in the planning process whether to use an external evaluator because the evaluation may include surveying or interviewing participants before the program begins. Moreover, making arrangements with an external partner takes time.
Look for a university with a research center or an academic program in psychology, sociology, arts administration, aging services, or social work to be your evaluation partner. While you will probably pay for services, aim to establish a mutually beneficial partnership in which you exchange free evaluation advice and/or services for real-life experience for the university’s students. Designing an evaluation might be a good class project. If you intend to find an academic partner, invite a representative of the institution to serve on your advisory committee.
Many consultants specialize in planning and conducting evaluations of all shapes and sizes for varying fees. You may be able to contract with a consultant for a short period to give you specific advice, but chances are that he or she will want to be involved for the entire program. This arrangement can be the best investment. Like a partner in higher education, a consultant not only designs the evaluation, but also helps ensure that your objectives are measurable; you are clear about what you want to learn; and you know—and know how—to communicate with various audiences.
Look for a consultant who
Each partner has a unique perspective and valuable skills to contribute to the evaluation process. Staff members of senior centers, adult day programs, and long-term care facilities are well positioned to assess the impact of arts and aging programs on participants. Not only do they observe older adults on a routine basis, but they also can administer surveys. Teachers and coordinators at a school or in an out-of-school program are similarly equipped and able to evaluate students who participate in an intergenerational program. A challenge in schools, out-of-school programs, and aging services facilities is lack of time. Activities directors, certified nursing assistants, and teachers are underpaid, overworked, and underbudgeted. This argues for involving them in designing the evaluation: if they are invested early, they are more likely to follow through, and their input will strengthen the process.
Appendix 7: Evaluation Tools