Step-by-Step Guide

This guide is intended to be used by organizations that have intentionally designed programs to cultivate virtues. The guide provides instructions on using a post-then-pre survey to conduct a program evaluation for collecting data to address the following questions:

Is {PROGRAM} effective at increasing {VIRTUE}?

The guide provides five steps: (1) Establish a program description, (2) Select the virtue measures and method, (3) Facilitate data collection, (4) Analyze and interpret data, and (5) Disseminate results via a report.

Step 1. Establish a Program Description

When embarking on a program evaluation, it is crucial to establish a program description. The program description should outline the program’s purpose, including its goals and objectives, and provide a detailed understanding of how the program is intended to function. Additionally, it should reflect the necessary resources for the program and adequately describe the expected benefits or outcomes. While there are various approaches to crafting a program description, such as writing a program narrative (i.e., written text), the most effective format is a logic model. Logic models provide a condensed one-page overview of the resources, activities, deliverables, and impact; logic models also offer a concise and comprehensive representation of the program’s design. Such a model can provide evaluators with a clear roadmap of the program’s intended operations, which can help them identify better areas of success and areas needing improvement.

Step 2. Select the Virtue Measure(s) & Methods

Virtue Measures. The program description should help you quickly identify the virtues your program intends to impact. Chapter 11 – Tool: Virtue Scales provides psychological scales (i.e., measures) that have been reformatted using the retrospective pre-test approach for the virtues listed below. Select the virtue(s) that align with your program goals (i.e., which virtues do you expect the program to impact).

Method to Administer Survey. Two options for administering the survey are paper and pencil or online. Both options have pros and cons. Group size and setting are two critical factors to consider when making this decision. Paper-and-pencil requires more administrative work in the end (data entry), whereas online requires more administration in the beginning (creating the survey online). You are more likely to collect data from everyone in the program if you administer a paper-and-pencil survey during the program. We have created worksheets and instructions for paper-and-pencil methods. You can easily modify these instructions if you choose to use online platforms.

Step 3. Facilitate data collection

At the designated time and place, administer the surveys to students. It is recommended that you collect data on the last day of the program. Evaluations can be built into a facilitated reflection about the program (e.g., purpose, content covered, activities, etc.). It is vital to share with students that you are collecting data to determine the program’s effectiveness and that data will be collected anonymously so that they can be honest in their responses. Read the instructions aloud. Then, provide time to complete the worksheet. Time will vary based on the length of the worksheet, but on average, each survey will take about 5-10 minutes to complete. A facilitated reflection and evaluation could take 30-60 minutes.

Additional Resource: At the end of the chapter, we provide an example of a facilitator script for implementing the survey in-person setting.

Demographics. To ensure quality data collection, it is recommended that you do not collect demographic information at the same time as administering a program evaluation survey. For program evaluations, students should not provide identifying information that would give away their identity—we commonly think of this as name, birth date, or student ID. However, in program evaluations for small groups, reporting race or biological sex and other demographic variables may provide the details to identify a specific student. Collecting demographic data will likely result in students not feeling that they can provide honest feedback and thereby reduce the quality of the data.

Demographic information can be collected on a different day or separately to describe the program participants. We provide an example demographic survey here: Demographic Survey.

Step 4. Data Analysis & Interpretation

In this step, the guide provides the simple steps accompanied by videos to assist you in entering your data, completing simple statistical models (means, t-test), and interpreting the data to meaningful information you can share with others.

Scoring. The surveys provided in Chapter 11 – Tool: Virtue Scales have scoring instructions on the second page. Each survey has different scoring instructions; read carefully. For scoring the surveys, consider these options:

• • If you use an online data collection system, you can download your data and use functions to calculate the score.
  • If you use paper-and-pencil, you (or a staff member/student worker) can choose to score these surveys yourself. If you choose to do it yourself, you may want to only print page 1 for data collection with students.
  • If you use paper-and-pencil, you can also choose to have the students score their own survey before turning them in. In this case, be sure to print page 2 of the worksheet and ask students to follow the instructions. It is best to have someone double-check the student’s work.

Reverse Coding. Some scales may have items that require reverse coding. Reverse coding is used in surveys to reduce response bias. It involves changing the wording or scoring of some items so that they are opposite or negative to the original ones. For example, a positive statement like “I am a patient person” can be changed to “I am not a patient person.” If an item is reverse coded, then the scoring is reversed, meaning a 1 becomes a 5, and a 5 becomes a 1. Tables 9.1 and 9.2 below illustrate reverse coding for scales of 1-5 and 1-7. These tables have been embedded into worksheets to assist in scoring.

Table 9.1. Reverse Coding for a 1 to 5 Scale

Table 9.2. Reverse Coding for a 1 to 7 Scale

Data entry. Data should be entered into an Excel sheet or Google Sheets, as these software programs allow you to conduct simple data analyses. Additionally, storing data in this manner will enable you to keep track of data year after year and make comparisons over time.

Calculate the grand mean. ‘Grand mean’ provides a simple summary measure of the average for the group. First, calculate the mean for the pre-test and then repeat for the post-test. The mean is the average of a set of values, which is the sum of all values divided by the number of values. Therefore, you will divide the sum of all the student’s scores by the number of students.

Link to Mean & Standard Deviation calculator

Calculate the change score. ‘Change scores’ provide a simple summary measure of the average change in a variable between two-time points. To calculate, subtract the grand mean for the post-test from the grand mean for the pre-test score.

Calculate effect size. The change score indicates whether a change occurred and the magnitude of the change. However, it can be difficult to interpret just based on the value if there is a significant difference between the pre-test and post-test. In many assessments, t-tests are commonly used to analyze the data and determine if the difference between the pre and post-test is statistically significant. However, we will stay away from this analysis because there are many problems with significance testing, especially for small groups^[3].

Instead, we will examine practical significance, which is referred to as real-world relevance or importance of the findings. We will do this by determining the effect size^[4]. Effect size is the magnitude of the difference between the pre and post-test. Specifically, we will calculate Cohen’s d to determine effect size. There are many online, free calculators that are able to calculate effect size.

Link to Effect Size calculator

Interpretation. Next, you will make meaning of the data for yourself and others. Below is a template paragraph for making sense of the data from the analyses. This template can be used in your evaluation report. As you interpret these data, it is important to remember that the simple methods that we used for this evaluation do not permit us to generalize the information beyond the one program that was evaluated. There is a possibility that the data could be influenced by cohort, instructors, semester, and other factors.

Results

On {DATE}, an evaluation was conducted of {PROGRAM NAME} to determine if the program was effective in cultivating {VIRTUE}. The program had ## participants, and ## participants completed the program evaluation (##%). The evaluation was provided to students on the last day of the program using a post-then-pre survey. {VIRTUE} was measured using {NAME SCALE} (Citation/Footnote/Endnote). The virtue was measured using a scale of 1 (strongly disagree) to 5 (strongly agree) {Double check this sentence to ensure the scaling (1-5 or 1-7) and words (strongly agree, etc) are correct—this will change per survey}.

On average, students rated themselves on {VIRTUE} before the program at #.## {GRAND MEAN PRE}, and after the program at #.## {GRAND MEAN POST} — a difference of #.## {CHANGE SCORE}. The analysis demonstrates that the observed change in pre-test and post-test scores {is/is not} practically significant, with a {small, medium, large} effect size (Cohen’s d = #.##).  This finding suggests that {PROGRAM NAME} {has/ does not have} a meaningful and impactful effect in cultivating {VIRTUE}.

Step 5. Disseminate Results via a Report

After completing the data analysis and interpretation, it is important to create a simple report (1-2 pages) of your findings. The report will provide an overview of your evaluation and interpret the results for stakeholders. The following outline is recommended.

Program Evaluation Report Outline

• Program Name
• Program Description
• Purpose of the Evaluation
• Evaluation Methods & Results
• Implications/Recommendations

Implications and recommendations come from the culmination of your understanding of the current context of the program and the weight of the data in helping make informed decisions. In essence, based on what you know now about the program, what recommendations would you make to senior leadership? You might provide many different implications and recommendations to your stakeholders.

ADDITIONAL RESOURCES

If you are new to program design, then some of this might feel a bit overwhelming. We highly recommend checking out Jen Urban’s work. She has created a FREE platform called The Netway that helps practitioners develop program models and evaluation plans. You will need to create a login to use these resources. Once you have logged in, then click on Resources ->  Protocol. This manual will provide you with a wealth of information about designing effective programs and how to evaluate them.