Formal assessment of virtues is a process that requires the use of rigorous tools and methods for collecting and analyzing information. In the following chapters, we will provide guides and measurement tools necessary for completing simple formal assessments on program evaluation and organization monitoring. Before delving into these tools, we will first explain the why and how of formal virtue assessment.

Specifically, in this chapter, we will grapple with why defining and assessing virtues is important, as well as approaches to empirically assessing virtues. If you are a numbers person and love the idea of being able to quantify abstract concepts like virtues, then this chapter is for you. If you are a measurement skeptic and are wary of the idea of trying to quantify something as incalculable as virtues, then this chapter is for you. We will explore both the strengths and limitations of quantitative and qualitative assessment of virtue.

Formal Assessment of Virtues: WHY

The formal assessment of virtue development is not just an academic exercise, but a practical endeavor driven by our commitment to positive character formation. We believe that by formally assessing virtues, practitioners can ensure that all their efforts, be it activities, programs, initiatives, or organizations, are effectively contributing to character formation rather than being a mere waste of time or, worse, contributing to the formation of vices.

Perhaps the best-known case of a program that accidentally contributed to the formation of vices is the anti-drug program D.A.R.E. (Drug Abuse Resistance Education).^[1] The D.A.R.E.  program, which was promoted by U.S. Presidents and schools, was not assessed until after 20 years of being implemented in schools all across North America. After millions of dollars and countless hours being spent on the program, the findings from several formal assessments determined that D.A.R.E. was not effective; one study even found that some kids were even more likely to try drugs after participating in D.A.R.E.

It is from stories like this that funders are now requiring organizations to conduct formal assessments to provide evidence about the impact of their programs/organizations. Specifically, formal assessments of virtue provide information about the impact programs and organizations have on virtues. While informal assessments discussed in Part 2 of this toolkit are helpful to students’ growth, formal assessments discussed in Part 3 of this toolkit can be useful to centers who desire to identify the impact their programs and organization are having on students’ virtue formation.

Strengths and Limitations of Formal Assessment

There are three main reasons why we formally assess virtues. First, measurement allows for objective comparisons. Relying on quantitative measurement, we can compare observations of people and groups to each other or some standard (e.g., compare our students’ levels of patience with other college student samples in the US). Second, measurement allows for tracking of change over time. If we want to be sure that we are positively contributing to virtue development, then we need to assess the growth (or reduction) of virtues. Measurement gives us a reasonably objective means to assess changes in people over time (e.g., changing levels of religious identity between the first and fourth year of college). Third, measurement allows us to make predictions (e.g., if a person is involved in this program, then their gratitude will increase) and test for how other abstract concepts relate to virtue development (e.g., the greater a person’s gratitude, the more likely they are to contribute back to the center by volunteering or mentoring new students).

The limitations to formal assessment of virtue include (a) the relative subjectivity of measurement, such that observations do not necessarily reflect reality, and (b) the limited scope of assessment, such that it cannot account for all types of knowledge. We acknowledge these limitations and conclude that an interdisciplinary approach to studying virtues that combines theory with quantitative measurement and qualitative understanding will give us the fullest picture of what virtues are and how they are cultivated. For the purpose of this chapter, we will further examine formal quantitative and qualitative approaches to assessing virtues.

Conceptualization and Definitions: WHAT

Now that we have grappled with why we formally assess virtues, we turn to what it is that we are trying to assess: the virtues themselves. Before we can assess and measure a thing, we need clear boundaries delineating what that thing is. Thus, the work of honing a clear definition of a virtue is an important prerequisite. Who gets to decide how a virtue is defined? Ideally, a collaborative and interdisciplinary group of experts.

In this toolkit, we provide established definitions and measures of a variety of virtues, spanning intellectual (e.g., humble teachability), prosocial (e.g., generosity), and performance (e.g., patience) types of character virtues. See Table 8.1 below for concise definitions of the virtues in this toolkit. No doubt, these definitions do not encompass the full richness of these concepts, but they are a useful starting point.

Table 8.1. Definitions of Virtues

Quantitative and Qualitative Approaches: HOW

So far, we have described why it’s worth formally assessing virtues and what virtues this toolkit examines. Now, we turn our attention to how we empirically assess virtues. We will briefly discuss two empirical approaches to virtue assessment: quantitative and qualitative.

Quantitative

Quantitative measurement refers to assigning numerical values to amounts of a thing so that, as described earlier, we can compare cases, track changes, and make predictions.

Foundational to the practice of quantitative measurement is the understanding that scientists are not measuring virtues directly. Instead, we measure indicators of the virtue that are directly measurable. Indicators are not the virtue itself but are presumed to be caused by or indicative of the presence of the virtue. For example, the virtue of patience is not directly measurable. However, scientists can observe how long a person waits (i.e., behavior), how much their heart rate increases (i.e., physiological arousal), and how much frustration a person reports (i.e., emotions). Thus, scientists rely on a variety of behavioral, physiological, and psychological indicators to capture the thing they want to measure. In this way, scientists are not claiming to perfectly capture every angle of a thing; rather, scientists are capturing the thing as well as they can by using observable indicators and assigning numerical values to indicators.

Psychological science of virtue development relies heavily on scales or questionnaires. With these kinds of measurements, participants self-report their agreement with a variety of statements (also sometimes referred to as items), and each self-reported response is assigned a corresponding numerical value (e.g., 1 = strongly disagree, 2 = disagree, etc.). Likewise, participants can report how well they believe a statement reflects another person’s virtue, serving as an informant report. The same statements can be used both ways; for example, students could rate their own patience as well as their roommate’s patience.

The strength of quantitative measurement is that scientists can more objectively make comparisons and track changes over time. A significant limitation of quantitative measurement is that some portion of the numerical score assigned to a person is assumed to be erroneous (i.e., variance in the score due to measurement fallibility). However, at the end of the day, scientists would rather capture 70-99% of a thing than not try to measure it at all.

Qualitative

Qualitative assessment refers to the interpretation of words, texts, and images to better understand human meaning-making and experiences. The use of qualitative research can thus capture how people define, understand, and even practice virtue. Here, researchers often rely on interview methods to solicit thoughts, reflections, and experiences by asking individuals a set of pre-determined questions. For example, a qualitative researcher could ask a college student about a time they volunteered for something on campus – an experience reflective of the virtue of generosity. The same researcher could also ask the students how they define generosity; that is, what this virtue means to them.

Qualitative researchers do not assume that an individual’s or a group’s perceptions and experiences are generalizable to all people. Rather, qualitative assessment can reveal how a person or group of people’s particular circumstances and situations shape their thoughts and behaviors.

The strength of qualitative assessment is that researchers can more deeply understand how people enmeshed in certain settings (e.g., college) make meaning about their experiences and environment, as described by the very people having those experiences. While qualitative research is not easily able to demonstrate cause-and-effect relationships or the strength of a correlational relationship, qualitative measurement can show changes in a person’s thinking or practices over time or the association between a situation and a way of thinking. Ultimately, qualitative research can provide thick narrative accounts of how people (e.g., college students) think about and practice abstract concepts (e.g., virtue).

Conclusion

The best approach to studying virtues will be interdisciplinary and involve quantitative, qualitative, and theoretical methods. Both quantitative and qualitative approaches to formal assessment of virtues have their affordances and drawbacks and can be used in tandem to capture the most detailed picture of a thing. Measurement of virtue is all about observing measurable indicators of a directly unmeasurable thing (virtues) so that we can make (relatively) objective comparisons, assess changes in character over time, and predict other things related to virtue development.

additional resources