By Howell Baum
Urban Studies and Planning Department
University of Maryland
In previous issues, the Voice has published about the Effectiveness and Efficiency 2.0 (E&E2.0) initiative of the Board of Regents, and also reactions from various faculty members. Below is another of the reactions.
“Efficiency” and “effectiveness” are favored terms in much of contemporary culture. Offered a choice, we typically prefer greater efficiency or effectiveness to less. However, much talk about “efficiency” and “effectiveness” ignores their precise meanings, and “rational” language often cloaks judgments made intuitively or on the basis of “common sense” or political interest. Nevertheless, even with careful attention to the meanings of “efficiency” and “effectiveness,” there are three difficulties in relying on them to make decisions about programs: (1) they can be difficult to operationalize in real-life situations; that is, it can be hard to say what these concepts mean with regard to real programs; (2) even if the concepts can be operationalized, it can still be difficult or even impossible to collect data that actually measure efficiency or, particularly, effectiveness; and (3) in any case, efficiency and effectiveness can conflict in real situations, supporting different, possibly opposing decisions and providing no definitive guidance.
“Efficiency,” taken from engineering through economics, refers to using relatively few resources to produce relatively many programs. “Effectiveness,” which has a more everyday meaning, refers to the ability of programs to bring about desirable changes in people and their lives. Thus in the university a class that enrolls the same number of students as another at a lower dollar cost or with less of a teacher’s time may be considered more efficient than the other. However, if the class that costs more is conducted in a way that enables students to learn more (and perhaps become more self-confident, set higher ambitions, and pursue more successful careers), that class would be considered more effective than the first. A third concept, “cost-effectiveness,” links the first two by considering the relationship between resources invested and results in people’s lives. For example, cost-effectiveness in a university could be formulated in terms of how much students learn per dollar or teacher time invested in their education, and classes or programs might be compared in terms of which leads to greater learning in relation to the resources invested in students.
Clearly, effectiveness is the crucial concern for universities, as for any institution. Unless programs have verifiable desirable effects on those whom the institution aims to serve, nothing else matters. This said, it is important to recognize the operational and practical difficulties of assessing the cost-effectiveness of programs. As examples of effects a university might want to bring about, we could consider enabling students to interpret fin de siècle novels or enabling students to analyze contemporary political debate in terms of liberal and conservative theories of human nature. To evaluate the cost-effectiveness of educational programs we would need a clear theory about how teaching works, which actions affect students’ learning in which ways, beginning with what things dollars should be spent on and proceeding through a sequence of activities to the learning of literary or political understanding. In reality we lack such knowledge; for better or for worse, professors teach on the basis of incomplete pedagogical theories, and they lack control over many important influences on their students’ learning. However, even if we could articulate a plausible sequence of events leading from dollars and professors’ time to students’ learning, we would still have difficulty identifying what data to collect to assess how programs are going and identify which programs are proceeding more or less cost-effectively than others. Not only is it difficult to identify good indicators of complex learning , but it is hard to collect data that realistically measure such learning. Moreover, students’s learning is influenced by both what professors do and what happens outside the university, and it is unclear how to collect data about both university and non-university influences so as to measure the effects of programs alone. Further, it is uncertain when data about students’ learning should be collected, that is, how long it takes for students to be fully influenced by professors’ efforts. Finally, there is no agreement about what standards to measure programs against: even if one program is more efficient, effective, or cost-effective than another, is it good enough, does it deserve a “passing grade”?
The implication of these difficulties is not to give up on thinking about the effectiveness and efficiency of programs, but to approach the challenges of assessing and choosing among programs with the recognition that programs are amorphous and poorly bounded, that we know less about how programs work than we would like, and that setting priorities is unavoidably a matter of making judgments in which, at best, calculations of effectiveness and efficiency can offer illumination and support. This, in fact, is how institutional actors, including university officials, normally proceed. More often than not, calculations of efficiency or effectiveness are used to rationalize decisions made through judgments based on interests, habits, values, “common sense,” and the like. There is nothing wrong with basing decisions on judgments, but decisions are better when decision makers recognize that they are making judgments and try to make their judgments on the basis of considerations they regard as most important. Further, their decisions are more likely to seem reasonable and legitimate to others insofar as they acknowledge what they are doing, consult with others along the way, and present their decisions clearly as products of judgments on the basis of criteria that have mattered to them.
Decision makers often feel vulnerable to challenge if they admit that they have simply made certain judgments, and they may adorn their decisions with language about “efficiency” and “effectiveness” to give apparent scientific weight to their judgments. A problem with proceeding in this way is that institutional members who have practical responsibilities–for example, professors who attempt to teach students critical thinking–see little connection between precise discussion of “efficiency” and “effectiveness” and the messy reality of their educational work, and they are likely to wonder about either the clear-sightedness or the intentions of decision makers.
Institutional decisions are about priorities–specifically, which effects on which people the institution should desire to bring about. There are familiar debates in the university about purposes. One concerns whom the university should aim to serve and influence: students, teachers, perhaps staff, and then whom else, such as businesses, communities, or the university itself as a corporation. Another concerns students in particular: in what ways universities should try to influence them: whether to aim to make them productive workers, creative thinkers, good citizens, good family members, some other things, or some combination of things. No calculations of efficiency or effectiveness will produce unambiguous decisions about either set of questions. With regard to the second question, more likely than not, professors will invoke tacit notions of effectiveness in arguing for support for certain academic programs, while some administrators will summon up notions of efficiency in arguing for increasing certain programs and diminishing or ending others. Such arguments illustrate conflicts between efficiency and effectiveness, but, more important, they show conflicts about what the university should be effective at. It is better to engage that discussion directly, rather than to avoid or conceal it with language of “efficiency” and “effectiveness” that doesn’t get at what people care about and what matters to them. Decisions emerging from such discussion will be more broadly understood and perhaps more widely endorsed.