Transformimg Jewish Lives
by Renae Cohen, Ph.D., Director, and Shira Rosenblatt, Ph.D., Associate Director, JESNA’s Berman Center for Research and Evaluation
This article was adapted from an article originally published at eJewishPhilanthropy on September 8, 2011.
The importance of using data to inform has been at the forefront of the work of JESNA’s Berman Center for Research and Evaluation for nearly 20 years. In addition to basic data, we also emphasize the importance of quality evaluation research, which can help us know the impact of programs on participants. The results of evaluation at different stages in the life of programs will help inform and improve program planning and decision-making, and may facilitate promising directions for funding. It is our responsibility to continue to make the evaluation data we collect relevant, meaningful, and accessible.
Our key point is that while good empirical data is crucial, it is strengthened by an accompanying commitment to evaluation. There are a number of key ways that evaluation can be implemented, among them the following:
- Evaluation from the design stage: Optimally, evaluation is considered by key stakeholders even before a program is off the ground. Thinking “evaluatively” from the inception of a program’s design provides organizations with the potential for greatest impact. By beginning the design phase in conjunction with outlining what one really wants to learn and how one is going to learn, and ensuring a feedback loop for incorporating the learnings, an organization capitalizes on the benefits of evaluation. Evaluation commencing at this stage may include working to identify and articulate outcomes: immediate, short-term, and even long-term. It can also set the stage for designing feedback forms and/or conducting focus groups/interviews that will allow the organization to understand the immediate take-aways of its program, what’s working well, and/or what can improve. Ultimately, this design hopefully would include measures to evaluate impact on participants at critical junctures post-program.
- Evaluation using capacity-building: There are many times that a foundation or a funder offers small- to mid-size grants, and includes an evaluation requirement for the program- or year-end report. Many of these organizations may lack staff with the skills necessary to conduct meaningful evaluation and/or the organization may not have sufficient funds to hire an external evaluator to conduct the evaluation. A model that can be particularly effective in this situation is to offer workshops in combination with a limited number of coaching hours per grantee. In this way foundations/funders can feel comfortable that they will get more useful data about program impact. They also will know they are advancing the knowledge, skills, and capabilities of their grantees to be able to have more tools to design and conduct evaluations, to think more evaluatively overall, and to have many more resources/tools on hand for evaluation in the future.
- Evaluation of community initiatives: Most of the time evaluators focus their attention on individual programs and their impact. However, sometimes there is the desire to go beyond evaluation of individual program impact and broaden the question to focus on the complexity of a community initiative made up of multiple programs all aiming to affect one target population, such as families with young children. This innovative approach may involve frequent and collaborative conversations to better understand the data being collected, the way in which the parts of the whole are interacting with one another, and the impact on the whole with the understanding that the impact is coming from multiple directions.
These are just a few examples of the various ways in which evaluation can be applied to help programmers, funders, foundations, communities, and/or organizations capitalize on the evaluation process. Let’s collect strong, reliable empirical data, but let’s also undertake empirical evaluation. And let’s not just do evaluation because we are required to do so—let’s maximize our learning along the way.
Evaluation 101:
Evaluating Jewish Youth Programs
Evaluation is critical to assessing the effectiveness of Jewish educational and communal programs and services. Evaluation need not be daunting or ominous. Ideally, it is a valuable and manageable process of reflective learning to explore different aspects of a program by asking and answering carefully considered, relevant questions. The following brief article originally appeared, in edited form, in Torah at the Center (Union for Reform Judaism. Volume 12, No. 1. Winter, 2009). The authors consider evaluation in the context of Jewish youth education, camp, enrichment programs, and other youth programming. However, the ideas and guidelines presented here are applicable to and valuable for many other types of programming.
How can a funder, federation, central agency for Jewish education, or program provider begin to address the myriad questions about improving and/or enhancing a Jewish youth program and its impact on participants? Step one is defining the “right” research and evaluation questions. While specific questions must be tailored to the learning goals of each program, JESNA’s Berman Center has identified foundational questions that are helpful to making any evaluation process a useful and value-added endeavor:
Ø What indicates success, and for whom?
Ø Which aspects of program implementation worked well and which need improvement?
Ø How can a program improve to better meet the needs of the program’s key stakeholders: program providers, students, parents, community leaders, funders, etc.?
Ø How will a program change participants in terms of the way they think, what they do, and/or what they know?
Ø What kinds of information are most helpful to gather for this learning?
Ø What resources are available – financial, personnel, and other – to gather this information?
Ø How will the information gathered through evaluation be used and disseminated to enhance the program and the field of Jewish education/youth programs?
As noted previously, many factors, including setting, target audience, and desired program outcomes, will help the evaluators develop specific questions. For example, typical evaluation questions that could arise in a formal school setting might focus on the impact of curriculum changes (e.g., In what ways are the goals of the curriculum being met in terms of new or improved ways students might be thinking or understanding?) or on the impact of teacher training (e.g., Is professional development for teachers making a difference in classroom interactions, student participation, and/or teacher motivation? How does professional development for teachers impact student learning?). In an informal/experiential education setting (e.g., a camp or youth group), evaluation questions might lead evaluators to investigate short-term and long-term impacts of Jewish summer camp experiences on the Jewish engagement or attitudes of counselors/advisors, campers, and their families.
Once these and other questions are thoughtfully considered, the next step is to begin to think about specific evaluation methods and tools that may be used to answer the questions. Depending upon relevant organizational (e.g., budget, resources, timelines, etc.) and environmental (e.g., funders’ priorities/needs, availability of participants, etc.) factors, the evaluator or evaluation team will employ different methodologies for gathering and analyzing data from key sources. Examples of evaluation methodologies include interviews, surveys or questionnaires, focus groups, and direct observations. Often, an evaluation will employ more than one methodology. Following are some basic elements of each approach to data gathering that funders and program providers should consider when planning program evaluation.
Interviews – either by telephone or in person – allow a program provider to ask probing questions to understand why and how a program made a difference, and the process by which participants gained knowledge and/or skills. Because interviews are designed to provide in-depth learning, they require adequate resources of time and staff to gather and analyze data. Interviews are helpful in particular when evaluators desire a narrative report of evaluation findings that includes direct quotations.
Focus groups are a useful way for program providers more interested in patterns and themes among program participants and/or key stakeholders and less concerned with individual responses. In educational settings, focus groups are constructed to include participants who share many common characteristics (e.g., age, experience in program) related to the Jewish youth program. A focus group is an especially productive approach to data collection when the lessons learned through dialogue likely will be more valuable than lessons learned through one-on-one conversations. This approach also is helpful when the program provider has limited financial resources for data gathering and/or analysis.
Surveys and questionnaires are most useful when the evaluator, funder, and/or program provider want to ensure that they obtain feedback from a large group of participants. This approach requires appropriate resources to field the survey, enter data (if the survey is not administered online), and analyze responses to open-ended and/or closed-ended questions. Surveys also enable a program provider to “test” differences between the way a participant rated his/her behaviors, attitudes, skills and/or knowledge before and after a program. Surveys are efficient tools to gather information about individuals’ backgrounds and attitudes about aspects of the Jewish youth programming. In addition, surveys allow evaluators to learn how respondents prioritize the impact of certain aspects of a program.
Direct observations of programs in action help evaluators better understand the context, personalities, content, and/or interactions in a given program, such as a retreat for Jewish youth. They also allow the evaluator or evaluation team to assess relationships among faculty and participants (and among participants) in formal and informal/experiential Jewish youth program settings (e.g., school or camp).
Each of these approaches to evaluation data collection allows funders and program providers to engage in a serious process of reflection through evaluation. Among other outcomes, the results of an evaluation may indicate the need to make mid-course program corrections or to rethink the alignment of goals/outcomes with the actual program as it evolves. To maximize learning through evaluation, funders and program providers must be open to constructive criticism and to reworking existing assumptions and/or programs. Once a funder or program provider has delved seriously into evaluation as a regular tool for reflection, it can become integral to the success of a program and the culture of an organization.
Effective evaluation is a challenging endeavor. Many funders and program providers want to learn everything at once from a single evaluation. However, responsible evaluation cannot be rushed; it requires patience, an appreciation of the small steps necessary to achieve greater learning, and the willingness to take the evaluation process one step at a time. The importance of securing “buy-in” from program staff and/or lay leadership cannot be underestimated. Key stakeholders should be engaged from the start (as appropriate) to develop learning goals and explore key evaluation questions. This collaborative approach to evaluation will generate collective support and appreciation for reflective learning through evaluation. Other individuals who may contribute to or facilitate the evaluation process include interns, colleagues, written resources (online and others), and skilled evaluation consultants.
Even the smallest organization is capable of embarking on a path toward planning, implementing, and using evaluation effectively. It requires engaging a skilled professional evaluation partner, beginning to ask important questions, considering the proper approach, and applying evaluation findings to strategic program decision-making. These small, but important steps – and the willingness to adjust to what we learn along the way – will go a long way toward ongoing program improvement and regular reflective practice.
Renae Cohen, Ph.D., is the Director of JESNA’s Berman Center for Research and Evaluation in Jewish Education. Shira Rosenblatt, Ph.D. is the Associate Director of the Berman Center.
To learn more about how JESNA’s Berman Center for Research and Evaluation in Jewish Education can help you maximize the effectiveness of your central agency’s Jewish educational and communal sector programming, contact Renae Cohen, Director of the Berman Center, at 212-284-6516 or at This e-mail address is being protected from spambots. You need JavaScript enabled to view it .
Previously, the Berman Center demonstrated how the ABCs of effective evaluation can help organizations, and the Jewish educational and communal sectors, go from good to great. Read more...
by Renae Cohen, Ph.D., Director, and Shira Rosenblatt, Ph.D., Associate Director, Berman Center for Research and Evaluation
This article was adapted from Torah at the Center, Union for Reform Judaism, Volume 12, No. 1.
Evaluation need not be daunting or ominous; rather, it is simply a way to engage in reflective learning to explore different aspects of a program, essentially by answering relevant questions. How does one begin to address the myriad questions that can enhance a program and its impact on the participants? First, one needs to determine the questions to address. While evaluation questions might vary by program, there are some foundational questions that are helpful to consider in making any evaluation process a truly useful and value-added endeavor:
- What indicates success, and for whom?
- What aspects of the implementation worked well and what aspects need improvement?
- How can a program improve to better meet the needs of the program providers, students, parents, community stakeholders, funders, etc?
- How will a program change the participants in terms of the way they think, what they do, and/or what they know?
- What kind of information is most helpful to gather for this learning?
- What resources are available – financial, personnel, and other – to gather this information?
Other specific evaluation questions depend on the nature of the evaluation. In a school setting, for example, some typical inquiries focus on the impact of curriculum changes (e.g., In what ways are the goals of the curriculum being met, perhaps in terms of new or improved ways students might be thinking or understanding?) or on the impact of teacher training on the teachers and/or their students (e.g., Is the professional development work for teachers making a difference in classroom interactions, student participation, and/or teacher motivation?).
Once these and other questions begin to be thoughtfully considered, the next step is to begin to think about specific methods/tools that may be used for an evaluation – or to answer the questions. Different methodologies may be employed at different times depending on a number of factors, including those within an organization (e.g., budget, available resources, programmatic timelines, etc.) and those outside of an organization (e.g., funders’ priorities/needs, availability of participants, etc.). Examples of methodologies include interviews, surveys or questionnaires, focus groups, and observations, and what follows are some general issues that should be kept in mind when considering each approach.
Interviews – either in-person or over the phone – help a program provider understand why and how a program made a difference, and the process by which a participant gained knowledge or skills. Interviews provide in-depth learning, and they require the resources to devote time to gathering and analyzing data. Interviews are helpful when there is the desire to include many quotes and/or narratives in the reporting.
Focus groups are useful when the program provider is most interested in patterns and themes of participants or key stakeholders, and less concerned with individual responses. In educational settings, focus group participants often share many common characteristics (e.g., age, experience in program) related to your program. A focus group may be used when there is a thought that lessons learned through dialogue will be more valuable than lessons learned through one-on-one conversations. This approach is also helpful when there are limited financial resources for data gathering and/or analysis.
Surveys and questionnaires are often useful when one wants to ensure that feedback is received from a large group of participants. One needs the resources to field, enter data (if not online), and analyze open-ended and/or closed-ended questions. Surveys also enable a program provider to “test” differences between the way someone rated his/her skills before and after a program. Surveys allow for information on individuals’ background and attitudes about certain aspects related to programming to be gathered efficiently. In addition, surveys allow the investigator to learn how someone would prioritize the impact of certain aspects of a program.
Observations help a researcher better understand the context, climate, atmosphere, and/or interactions in a given program (such as a retreat). They also allow for the assessment of teacher-student relations in a classroom, and are helpful to explore the ways in which students or campers interact with one another.
Each of these methods allows program providers to engage in a serious process of reflection through evaluation. Often an evaluation will employ more than one methodology. To maximize learning through evaluation, one must be open to hearing criticism and reworking existing assumptions or programs. Among other things, the results of an evaluation may lead one to make mid-course program corrections or to rethink the alignment of goals/outcomes with actual programs. When one has seriously delved into evaluation as a regular tool for reflection, it can become integral to the success of a program and the culture of an organization.
Common challenges of evaluation include wanting to learn everything at once. However, evaluation cannot be rushed; it requires patience and an appreciation of small steps toward greater learning. Those interested in pursuing evaluation might consider what may be a first step – or a first question to address, a second step, and a third step in an evaluation process, and take one at a time. Staff and/or lay leadership should be engaged in developing and exploring key questions in this learning process so that there is input, “buy-in,” and a collective appreciation for an approach of reflective learning through evaluation. Interns, colleagues, written resources (online and others), pro-bono consultants, or consultants, are often helpful as one moves along in the evaluation process. Even the smallest organization is capable of embarking on this path – by taking the first step, thoughtfully considering the proper approach, and beginning to ask and address the relevant evaluation questions. These small steps – adjusting for learning along the way – go a long way toward program improvement and reflective practice.
The Berman Center provides a range of evaluation services for clients of all sizes, from large-scale national surveys to targeted marketing studies to individual consultation and coaching. In partnership with our clients, we craft the best evaluation approach for each project within our clients’ critical time and budget parameters. Some of the outstanding educational and communal service organizations and funders that recently have engaged the Berman Center’s evaluation services include:
The Harold Grinspoon Foundation (The PJ Library), West Springfield, MA
The PJ Library is a flagship program of the Harold Grinspoon Foundation that distributes age-appropriate Jewish children’s books and CDs (with guides to help families use the selections in their homes) to families with children from six-months through eight years of age in participating Jewish communities.
Jewish Education Center of Cleveland (JECC): Retreat Institute, Cleveland
The JECC’s Retreat Institute creates and implements text-based retreats and programs in the Greater Cleveland Jewish community in cooperation with Jewish educational institutions as its primary partners (i.e., Jewish day and supplementary schools and congregations).
MAKOM, New York, New York
MAKOM – the Israel Engagement Network - provides professional development and content for communities and organizations to stimulate innovative thinking; inspire new initiatives; and generate systemic, programmatic, and institutional change in Israel engagement within the North American Jewish community.
The Montreal Federation CJA, Montreal, QC Canada
The Montreal Federation CJA is the central funding, planning, and coordinating body for services within Montreal’s Jewish community.
The Jewish Community Relations Council (JCRC), San Francisco
The JCRC serves as the central public affairs arm of the organized Bay Area Jewish community, representing synagogues and Jewish organizations on issues impacting the rights and protection of Jews as individuals and as a community.
Remember Us: The Holocaust Bnai Mitzvah Project, Santa Rosa, CA
The Holocaust Bnai Mitzvah Project connects students preparing for bar/bat mitzvah with voluntary opportunities to remember individual children who perished in the Holocaust before they could be called to the Torah.
To learn more about how JESNA's Berman Center for Research and Evaluation can help you maximize the effectiveness of your Jewish educational and communal service initiatives, contact Dr. Renae Cohen, Director of the Berman Center, at 212-284-6516 or This e-mail address is being protected from spambots. You need JavaScript enabled to view it .
We have a long-standing, deep appreciation for JESNA and the work of the Berman Center. In the course of the recent evaluation of The PJ Library, the Berman Center staff was always very professional and very responsive. We were thrilled with the final report and disseminated it widely to our communities, funders, and stakeholders. The senior Berman Center staff member who worked with us was a hero in many ways, including in her excellent presentation of the evaluation findings. They are great colleagues you can count on.
— Director, The PJ Library®, The Harold Grinspoon Foundation
The Berman Center specializes in applied learning through program evaluations that makes us better at what we do. When I have a program that needs evaluation, I click on “J” for JESNA in my contact list. Our evaluation partnerships have worked well and provided actionable recommendations.
— Former Executive Director, Synagogues: Transformation and Renewal (STAR)
Our coaching and consultation work with the Berman Center put us on the path to greater effectiveness and efficiency. It translates into all areas of our work in the Jewish community. We are still seeing the impact of what we did with your senior staffer. She was very supportive and available. She convinced me that evaluation was not scary and pushed me to make it a priority. At the same time, she made it really enjoyable.
— Associate Director, JEFF and Informal Education, Alliance for Jewish Education of Metropolitan Detroit
When it comes to my evaluation experiences with JESNA’s Berman Center, I am a really satisfied customer. Their team has shaped the way we do evaluation – both on our own and in partnership with Berman Center staff. Through our collaborative work, the Berman Center has helped us build our evaluation capacity and approach so that the results of the evaluation process can take root in our local setting.
— Director, Planning and Evaluation, The Jewish Education Center of Cleveland
The JCRC had an exceptionally positive experience with the Berman Center. Its staff was very professional and provided the expert insight and wisdom that we needed. As the result of our work with JESNA’s Berman Center, the JCRC will be more effective in its current programs, as well as the programs that it innovates and implements in the future. In addition, we will be able to demonstrate to our stakeholders – including funders - that we are constantly learning, and that the organization effectively achieves the goals we share. We would highly recommend the Berman Center to any Jewish nonprofit.
— Associate Director, Jewish Community Relations Council
To learn more about how JESNA's Berman Center for Research and Evaluation can help you maximize the effectiveness of your Jewish educational and communal service initiatives, contact Dr. Renae Cohen, Director of the Berman Center, at 212-284-6516 or This e-mail address is being protected from spambots. You need JavaScript enabled to view it .