13 September 2006

Bad Reasoning

One of the real problems in policy analysis concerns the misuse of sweeping generalizations. Here's an example, from the ordinarily better-conceived conservative magazine Commentary:

Even though [the book's author] has scant interest in the issues that preoccupy the most perceptive of the critics—a politicized faculty, threats to freedom of expression, the absence or the actual suppression of a balanced exchange of ideas—when it comes to "how much students are learning," and "what is actually being accomplished in college classrooms," he too sees trouble, and plenty of it, in the beautiful groves of academe:

Many seniors graduate without being able to write well enough to satisfy their employers. Many cannot reason clearly or perform competently in analyzing complex, non-technical problems, even though faculties rank critical thinking as the primary goal of a college education. Few undergraduates receiving a degree are able to speak or read a foreign language. Most have never taken a course in quantitative reasoning or acquired the knowledge needed to be a reasonably informed citizen in a democracy. And those are only some of the problems.

It seems, in short, that our colleges are "underachieving" after all—and that even their supposedly happy clients know it. Fewer than half of recent graduates, according to [the book's] ever-ready statistics, think they have made significant progress in learning to write, and some think they have actually regressed. Employers confirm this self-assessment, complaining that the college graduates they hire are inarticulate. As for critical thinking, "The vast majority of graduating students are still naïve relativists who 'do not show the ability to defensibly critique their own judgments' in analyzing the kinds of unstructured problems commonly encountered in real life." In the area of foreign languages, fewer than 10 percent of seniors believe they have substantially improved their skills and fewer than 15 percent have progressed to advanced classes. Nor are the results any better in general education, the great battleground of the critics. According to one study, only about a third of seniors report gains in the understanding or the enjoyment of literature, art, music, or theater. [the book's author] goes so far as to quote Daniel Bell's judgment of the typical curriculum as "a vast smorgasbord" amounting to "an admission of intellectual defeat."

Donald Kagan, "As Goes Harvard…" (emphasis added).

This passage—which is representative of Professor Kagan's comments as a whole—itself represents many of the problems being "diagnosed" in American university education.1 Note, first, that some of the blame being heaped on the university is not the university's fault:

  • "Few undergraduates receiving a degree are able to speak or read a foreign language"—perhaps for some value of "few." This, however, is an indictment of earlier schooling; college is too late to begin learning a second language. In fact, it doesn't apply at all to the better schools, which require several years of foreign language study in high school as an admission requirement.
  • "Many seniors graduate without being able to write well enough to satisfy their employers"—and it doesn't improve after three more years of graduate school, whether in law or in most PhD programs. This is a problem with all of American society (including the employers, who—on the basis of who gets promoted and who does not—don't value this skill themselves), not just the university.
  • "According to one study, only about a third of seniors report gains in the understanding or the enjoyment of literature, art, music, or theater"—I'd love to see that study, and smack its authors around a little bit, as it treats a self-perception of students as definitively true. Perhaps this is just a consequence of learning enough to realize how much one doesn't know. It sure as hell was for me, and for many of my classmates.2 Then, too, this is rather at odds with the purported emphasis on pure critical thinking, isn't it?

The most damning self-indictment, though, comes through by considering Professor Kagan's juxtaposition of an unveiled assault on "naïve relativists" (just think "activist judges") and his invocation of "absence of quantitative reasoning." That last is an area I find particularly amusing, because both Professor Kagan and Professor Bok (the author of the book Professor Kagan is discussing) rely on invalid, or at best dubious and reflexive, statistics in support of their arguments. Statistics such as "ten percent of ___" are valid only when they are drawn from a statistically congruent population. There is almost no population less congruent than that of "American college seniors"; the only one I can think of off-hand is "American employers". One cannot draw conclusions about an "average" when one is studying self-responses from a group of students who variously graduated from Hahvahd, Surfboard Tech, and the University of Southern North Dakota at Hoople. Neither can one lump the engineers, nurses, physical therapists, elementary educators, B-school grads, and "general studies" majors in with purportedly "elite" liberal arts students and then draw conclusions on the inadequacy of a liberal arts education.

Then there's the quantitative reasoning issue lurking underneath this reliance on survey statistics: How much did the questions influence or determine the answers? Oops. I guess that's some "naïve relativism"3 showing there: I do not accept the premise that there is a certain, preordained, measurable educational objective/outcome that properly informs the university experience of every graduate. Maybe, though, that's just a consequence of my own training in quantitative reasoning. Or, more likely, it's the several years I've spent wrestling materials put out by JDs and PhDs who don't learn quantitative reasoning in graduate school, either. On the evidence of his essay, that may well include Professor Kagan (and Professor Bok). It certainly includes whoever selected its headline.


  1. As an aside, perhaps the perceptions stated in the book in question are skewed by the stupidity of allowing—indeed, encouraging—faculty members to treat undergraduate education as a chore instead of as part of their job at some "elite" universities. The "elite" university I attended had the opposite attitude: There were no undergraduate classes entirely taught by graduate students. That Hahvahd can't say the same is not an indictment of Johns Hopkins, or Washington University, or other prominent "elite" universities… no matter what the article title implies.
  2. Perhaps, too, this conclusion follows from having some classics debunked by actually reading, seeing, or hearing them. For example, much of the canon of "American Literature" simply does not hold up under close examination—and those who don't study literature seriously will nonetheless treat those works as "classics." In that way, I enjoy Melville, and Hawthorne, and James (Henry, not William), and the so-called "Lost Generation," less now than I did in high school—which would fit the profile of that question. Just as I am not in a position to criticize an educator's choice of what constitutes a "classic" in the field of education…
  3. How "relativism" is "naïve" after the US experience in Vietnam—or, more broadly, everyone's experience with unconventional warfare—escapes the military officer in me.