Virtues may not always be the goal of a program; instead, virtues may be the by-product of communal, shared knowledge and practices that are aimed at other programmatic goals. In our research on CCTs, we noted that programs were often designed to facilitate intellectual outcomes. Virtues, if considered at all, were sometimes secondary outcomes (or the fruit) in the lives of those who might choose to apply the intellect to their own life. Thus, virtues were not always a programmatic goal that needed to be assessed using program evaluation. Instead, such organizations may be better served by organizational monitoring.

While program evaluations and organizational monitoring may share similar methodological elements, their focus differs significantly. Program evaluations typically concentrate on individual programs, such as fellowship programs. In contrast, organizational monitoring takes a broader perspective, studying the organization as a whole. This approach considers the collective impact of a program and the community’s social environment. Thus, in this approach, we engage all participants across the organization.

This chapter offers a guide for annual monitoring of an organization’s virtue formation milieu (i.e., social environment). Monitoring involves collecting and analyzing data on an ongoing basis to keep track of an essential aspect of an organization’s culture. For many organizations, monitoring ensures productivity and/or a positive workplace environment (e.g., preventing racism, sexism, ableism, ageism, etc.). With the help of this monitoring assessment, organizations can 1) identify if they are providing the support and opportunities to facilitate character formation they want to provide to participants and 2) monitor if any new practices, programs, or policies would impact the virtue milieu.

Step-by-Step Guide

In this guide, we provide the steps for conducting an assessment to identify the current status of virtues and trends in virtues of participants utilizing any of the organization’s offerings (e.g., programs, services, and supports). The assessment can include surveys like the Character Index that provide quantitative data and/or interviews about character formation that provide qualitative data. Therefore, below, we provide two guides, one using a quantitative approach and one using a qualitative approach. We recommend using both approaches together. While this guide focuses on using these approaches with student participants, they can also be used with center staff, faculty, community participants, or other stakeholders.

Quantitative Approach: Character Index

Step 1. Determine the Assessment Period

Consider your operational year. When would be the best time to conduct an annual assessment? Select a two-week period to target data collection. You will want to keep this roughly consistent from year to year. Avoid times when there is a lot on the calendar or when students are experiencing burnout (e.g., midterms and final exams).

Step 2. Create an Online Survey

For an organizational assessment, it is often best to use online methods to reach everyone associated with your organization. Organizations may use Google Forms (which is free), Qualtrics, or other survey platforms. Create your online survey using the Character Index and demographic survey listed below.

Character Index. We have developed a short index, the Character Index, that allows you to assess all the virtues in this toolkit in less time. We recommend using this tool to capture your organization’s virtue profile quickly. The index contains 16 statements– each statement is the best statement representing each virtue scale in this toolkit. Chapter 12 – Tool: Character Index provides the index, scales, and more information on using this index.

Note: To keep this guide simple, we focus exclusively on using the Character Index to measure virtues. However, it is possible to use one or two full scales separately or in combination with the index. We provide a few notes about this at the end of the chapter in Additional Resources.

Demographics. Collecting demographics can make this data even more meaningful. At a minimum, we recommend collecting data on current student status (e.g., first year, second year, etc.) and the programs participated in this year (or the number of programs). We provide an example demographic survey here: Demographic Survey. However, there are some caveats to consider when collecting demographics. To ensure quality data collection, it is recommended that you do not collect demographic information at the same time as administering a survey for small groups, or do not request demographics that would allow your team to identify the respondent—we commonly think of this as name, birth date, or student ID. However, in assessments for small groups, reporting race, sex, and other demographic variables may provide the details to identify a specific student. Collecting demographic data for small groups will likely result in students not feeling that they can give honest feedback, reducing the data quality.

Step 3. Identify Participants

Identify a list of current participants utilizing your organization’s offerings (e.g., programs, services, and supports) for the current operational year. How many students have you served? How can you contact these participants (e.g., by email, text, or social media)?

Step 4. Data Collection

Administer the survey to students during your annual assessment period. For online assessments, you should promote about two weeks before launching. On launch day, it is best to send an email or digital communication explaining the survey and why you are collecting data, along with the survey link. Tell students that data will be collected anonymously so that they can be honest in their responses. Continue to promote during synchronous events. Send a reminder each week until the assessment period ends.

If you are considering an incentive, be mindful of how you will keep the data anonymous. For example, some organizations have incentivized students’ participation using a raffle or drawing for a special prize (money, book, reduced program fee, etc.). If students are required to put their email addresses on the form with their data, then their data is no longer anonymous. There are many workarounds—sometimes, you need to be creative. For example, you can have students scan a QR code in the building, take the survey, show the “thanks for completing the survey” page to a staff member, and then redeem a raffle ticket.

Step 5. Data Analysis & Interpretation

If you use an online system for data collection, you can download your data and use functions to calculate the score. Chapter 12 – Tool: Character Index provides the scoring instructions and interpretation guide for the index. You have two options for scoring. Option 1 will allow you to calculate a total character score for both individuals and your group. Option 2 will allow you to calculate which virtues are strengths (i.e., highest levels) and which are growth areas (i.e., lowest levels) for your group.

Step 6. Disseminate Results via a Report

After completing the data analysis and interpretation, you will create a simple report (1-2 pages) of your findings. The report will provide an overview of your assessment and interpret the results for stakeholders. The following outline is recommended.

Organization Monitoring Report Outline

• Organization Name
• Overview of Organization
• Methods & Results
• Implications/Recommendations

In the report, you will want to provide an overview of the organization. Information should include (a) current programs and offerings in the last year, (b) any new policies or changes made in the last year, and (c) the number of students served in the last year. Then, identify the methods and results of the assessment. If you have additional years of data, you will also want to include this information (e.g., perhaps by providing graphs so that you can identify any trends in the data over time).

Finally, you will want to identify any implications and recommendations based on what you know now from the data. Recommendations can help senior leadership make data-informed decisions. It is worth noting that the Character Index can only provide reliable information on the virtues of the aggregate of your participants, not directions of change or growth for individuals or causal explanations for high or low results. For example, a center that identifies that its Total Character Score is a 3 may choose to make the goal of increasing this score to 4 and identify the specific strategies that will be used in the next year to improve the score. Or, perhaps a center identifies that they are scoring a 2 on the virtue of patience and makes goals and strategies for increasing this score to 3 in the next year. They may also choose to use the full virtue scale for patience in addition to the Character Index in order to assess this particular virtue more holistically (see Additional Resources at the end of this chapter).

Quantitative Approach: Character Interview

Step 1. Create a Strategy for Interviews

Interviews are a great way to enrich your quantitative data. By collecting students’ stories and experiences, you can have more data to make meaning of the quantitative results. In conducting interviews for an organizational assessment, you will want to create a strategy for who should be interviewed. If this was a program evaluation, we would only want individuals from a particular program. But as we consider the entire organization, we may want to intentionally interview students across diverse class standing (first year, second year, etc.), majors (business, humanities, sciences, etc.), sex, race, etc. As you get started, start small. Aim to interview 6-8 people per year. You could interview one-on-one or conduct focus groups (e.g., small groups of 3-4 and never more than 8). As you grow in your ability and capacity, interview more. Work with other staff as you create your strategy and brainstorm which students to recruit.

Step 2. Make an Appointment

For these interviews, you will want to reserve a space that is comfortable for you and the student. If you plan to record the interview, you will want a very quiet space. Send an invitation to the student explaining the purpose of the interview. Keep interviews to 20-30 minutes.

Step 3. Prepare for the Interview

Chapter 13 – Tool: Virtue Interview provides a general overview of interviewing along with 18 potential questions that you can use to guide your conversation about your organization; Chapter 5 –  Personal Reflection via Character Conversation provides additional questions that may be of interest. Identify which of these questions are a priority to you. Edit and personalize these questions to sound like they are coming from you. Your interview time will go by quickly, so at the end of each interview, you may want to consider what you still want to learn from students as you move into the next interview.

Step 4. Conduct the Interview

As you get started, remind the students of the purpose of the interview and ask if they have any questions. Then, ask your prepared questions. It is best to record the interview so that you can go back later and grab specific quotes from it. If you do the meeting online, tools like Zoom or Teams can record and transcribe the interview. Be mindful of time; you may want to set an alarm.

Step 5. Analyze and Interpret

You will analyze the data by identifying themes and selecting quotes to share with stakeholders. Chapter 13—Tool: Virtue Interview provides instructions on theming and interpretation. Try to provide 1 to 3 supporting quotes for each theme in your report.

Step 6. Disseminate Results via a Report

Like the quantitative data, you will create a simple report (1-2 pages) of your findings. The report will provide an overview of your assessment and interpret the results for stakeholders. If you also conducted a quantitative assessment, you will want to integrate these two data sources into one report. The following outline is recommended.

Organization Monitoring Report Outline

• Organization Name
• Overview of Organization
• Methods & Results
• Implications/Recommendations

In the report, you will want to provide an overview of the organization. Information should include (a) current programs and offerings in the last year, (b) any new policies or changes made in the last year, and (c) the number of students served in the last year. Then, identify the methods for the interviews (i.e., # of students interviewed, interview questions). The results will share the themes you identified from your interviews and the supporting quotes for each theme. Finally, you will want to identify any implications and recommendations based on the data. Recommendations can help senior leadership make data-informed decisions.

ADDITIONAL RESOURCES

During organizational monitoring, a center may desire to focus on just one or two virtues. If this is the case, use the virtue’s full scale. The full scale provides more reliable data but can take more time if you want to assess multiple virtues. You can use full scales separately or in combination with the index. Chapter 11 – Tool: Virtue Scales provides more information on full scales. Each full-scale survey provides scoring instructions on the second page of the handout. Additionally, references are provided to compare your center score with the average score of centers for Christian thought and/or university students across North America.

Scales are formatted to measure one-time in the present moment:

1. Intellectual Humility
2. Expressed Humility, Teachability
3. Interpersonal Generosity
4. Gratitude
5. Gratitude to God
6. Transcendent Indebtedness to God
7. Transcendent Indebtedness to Humans
8. Patience
9. Self-Control
10.  Courage
11. Meaning in Life
12. Beyond-the-Self Orientation