Posted by Sheila Rodriguez on November 21, 2016
As REL Northeast & Islands at EDC began its work with research alliances in 2012, our staff soon discovered that many district- and state-based alliance members wanted to learn more about how to use logic models to improve program or policy design, implementation, and evaluation. A logic model is a visual representation of a theory of action that can guide individuals who are working together to design, implement, and/or evaluate a program or initiative. There is no “one way” a logic model should look, but all models include certain elements, such as a clearly identified problem to be addressed, desired outcomes, proposed strategies for achieving the desired outcomes, and assumptions about the inputs or actions that lead to the desired outcomes. Logic models help to ensure that a proposed education program or intervention truly addresses the problem identified by the group and that measures for evaluating success are integrated into the program design and roll out.
Creating logic models, however, should be understood as a process that requires deliberate and thoughtful conversations among all stakeholders invested in the new program or intervention. A logic model is typically not a static document but a guiding framework that can and should be revisited and revised as needed.
To help alliance members more fully understand and utilize logic models, a team of REL researchers, including Karen Shakman and me, created a workshop to build stakeholders’ capacity to develop and use logic models in their work. The workshop was delivered to three alliances in 2013.
For the alliances—which in most cases convene members from across states, organizations, and disciplines around a shared common interest—the logic model workshops were an opportunity to not only learn how to develop logic models but, in some cases, also build members’ capacity to begin the process of designing a logic model for their own alliance projects. Dr. Shakman, fellow REL researcher Jenny Scala, and I delivered the first logic model workshop in June 2013 with the Puerto Rico Research Alliance for Dropout Prevention, which is a broad stakeholder group trying to develop an Early Warning System (EWS) for dropout prevention in Puerto Rico. Alliance Facilitator Sandra Espada-Santos invited us to lead a two-day, face-to-face workshop that not only explored what logic models are but also led the alliance members in discussing and defining the elements they wanted to include in a dropout prevention logic model for their EWS.
“The workshop helped the alliance members to identify that there were several needs to be addressed,” Espada-Santos said. “Because of the discussions in this session, we realized we needed closer communication and collaboration between the key stakeholders. They had different definitions of what a dropout student is, and issues like that needed to be discussed with more intense communication. They also identified that studies were needed to better understand the statistics and data that were available about dropouts in Puerto Rico.”
The other two alliances that requested the logic model workshop were the Northeast Educator Effectiveness Research Alliance (NEERA) and Urban School Improvement Alliance (USIA). These alliances were particularly interested in learning how to use logic models as a vehicle for building effective evaluation designs. NEERA has focused much of its research agenda on educator evaluation, and alliance members were interested in thinking more deeply about using theories of action to guide educator evaluation models. USIA is comprised of research directors at mid-sized districts in New England and New York, and they wanted training in how to help their instructional leaders think about setting up a proper evaluation prior to starting a new program or initiative. To address these needs, Rodriguez and Shakman presented three logic model workshops as webinars, inviting people both inside and outside the alliances to participate. Over 100 people attended these virtual workshops, which were held in October and November 2013, and January 2014. (Visit REL Northeast & Islands’ Skill-Builder Archive to view the webinars.)
Constituent groups outside REL Northeast & Islands alliances also requested logic model workshops. These groups include the following:
The logic model workshops were such a success that Rodriguez and Shakman used the materials as a starting point to build a toolkit that could be used by stakeholders across the region, and indeed the country, who want to conduct workshops with their colleagues in order to develop a logic model. The goal is for the toolkit is to support and build the capacity of education researchers and practitioners to develop and revise logic models on their own, without the help of people like REL researchers or other consultants, who are experts in this process. To keep the toolkit grounded in real-life needs and scenarios, a small group of regional stakeholders served as an advisory committee in developing the toolkit, reviewing materials and providing feedback along the way.
In 2015, the Institute of Education Sciences published REL Northeast & Islands’ Logic Model Workshop Toolkit. The toolkit includes a complete facilitator workbook, participant workbook, and slide deck that can be used by education researchers, district leaders, and others to lead two separate workshop sessions—the first focused on the elements of a logic model and the process for developing a logic model for a program or policy, and the second focused on how to use logic models to develop effective program evaluation questions and indicators of success. The toolkit also includes case study examples that can be used throughout the workshop to illustrate the logic modeling process.
To further disseminate knowledge about logic models, REL Northeast & Islands published in September 2015 a self-paced, online professional development course focused on using logic models to develop program evaluations to support improvement. Since then, 33 people have completed the course. The REL also hosted a Skill-Builder Webinar last December on using the toolkit to facilitate the logic model workshop. Over 80 people participated, and survey feedback included:
REL Northeast & Islands continues to field requests for these workshops and materials, and often uses these tools in its work with constituents in the region. In September 2016, Dr. Shakman used the toolkit to work with a member of the Connecticut State Department of Education to help her plan for an evaluation of the state’s teacher induction program. And on October 3, Dr. Avery led a team of REL staff who traveled to Augusta, Maine, to work with staff at the state department of education in developing a logic model and evaluation design for a new K–3 literacy pilot.
Looking ahead, the published logic model toolkit, coupled with the facilitator’s guide and the archive of the Skill-Builder Webinar, will allow state, district, and school leaders—as well as other regional and national stakeholders—to continue to use the tools even after the current REL contract ends on December 31.