What Are the Best Ways to Measure the Effectiveness of Afterschool Programs?

In October, the Reference Desk received this question about measuring the effectiveness of afterschool programs. It was one of a number of recent questions inquiring about program evaluation in public schools. The resources listed below not only describe research on afterschool programs; they also illustrate some of the methodology and best practices for program evaluation in general.

Question

How can one measure the effectiveness and outcomes of afterschool programs, and what are some examples of quality evaluations of afterschool programs?

Research Synopsis

Reference Desk researchers found numerous resources that provide data collection and analysis methods from studies on afterschool programs. Although there is no consensus on which methodology is best for evaluating afterschool programs in particular, a number of studies cite meta-analysis as a rigorous and comprehensive research methodology. According to one resource, “Meta-analysis is a collection of systematic techniques for resolving apparent contradictions in research findings. Meta-analysts translate results from different studies to a common metric and statistically explore the relations between study characteristics and findings” (Afterschool Alliance, 2008; see below).

Publicly Available Resources

The Evaluation of Enhanced Academic Instruction in After-School Programs. Black, A. R., Somers, M-A., Doolittle, F., Unterman, R., and Grossman, J. B.; Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education; NCEE 2009-4077; September 2009; 305 pages.

This report evaluates the effectiveness of enhanced academic instruction in afterschool programs. The evaluation draws on multiple data sources: afterschool program attendance; data on the number of skills assigned during the school year; afterschool staff surveys; structured interviews with afterschool instructors, regular afterschool program group staff, and afterschool district coordinators; structured protocol observations of the implementation; student surveys; regular-school-day teacher surveys; student achievement tests. Data include characteristics of the school, such as school setting, student body demographics, and student-to-teacher ratio.
Evaluations Backgrounder: A Summary of Formal Evaluations of the Academic Impact of Afterschool Programs. Afterschool Alliance; July 2008; 35 pages.

This article summarizes numerous studies of large-scale, state- and local-level afterschool programs and includes the research methods used in these studies as well as findings on the impact of these programs on students’ academic performance. Among the studies analyzed in this article are: “The 2006 McREL Meta-Analysis of Effects of Out-of-School Time Programs for At-Risk Students…[the] New Hampshire Study of Academically Focused Afterschool Programs …[and the] Mahoney and Lord Ecological Analysis of After-School Program Participation.”
When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program: New Findings. Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., & Mansfield, W; Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences; U.S. Department of Education; October 2004; 159 pages.

“The evaluation team collected student outcome data in five areas: after-school supervision, location, and activities; academic performance and achievement; behavior; personal and social development; and safety.”

The Reference Desk also found these organizations and resources to be helpful in learning more about afterschool programs:

Afterschool.gov

Afterschool.gov provides a single location for the public to access afterschool resources that are located across the federal government, including information about starting and operating an afterschool program, federal funding sources, and research and reports on afterschool and youth issues.
Afterschool Alliance

The Afterschool Alliance works with the Administration, the U.S. Congress, governors, mayors, and advocates, and has more than 25,000 afterschool program partners. Its website contains a series of briefing papers, report,s and fact sheets used widely by media, policymakers, concerned organizations, and individuals.

Download the Full Response

If this material brings up questions of your own, please submit a new request here. If you’re interested in learning more about the Reference Desk, read about us here. And finally, share thoughts, experiences, and resources of your own in our Feedback section below.