Assessing Assessors

Another facet of the “college isn’t really worth it” mindset has shown up in an op-ed by Molly Worthen in the New York Times. The op-ed itself complains that assessments of learning in colleges are, as implemented at present, a waste of time and effort. It is in the comments that you see a lot of people arguing quite strongly that some proof of learning is of value to those paying the bill. The irony is that these two sides are probably not disagreeing.

Years ago, the assessment was pretty obvious.  Got an “A” in a course–you demonstrated mastery of that subject.  Got an “F”? You didn’t. Note this didn’t necessarily measure learning in that class–if you waltzed in having already passed such a course elsewhere, you might have learned nothing, while the F student might have learned a lot relative to a poor base. While on occasion the instructor caught flack for doing a poor job, generally it was the student who discovered he or she was not up to the task.

What has shifted is the expectation of a university education. What it still is is an opportunity to access scholars who are teaching you and evaluating your progress. But as costs have risen and job opportunities without a college degree declined, the expectation is that you are purchasing an education instead of access to education. Just as you would complain if McDonalds didn’t give you the hamburger you just paid for, many now complain that students did not learn though money passed hands. Furthermore, the emphasis on financial gain from college has brightened a focus on getting a well-paying job with that degree. The response from the educational community has been to evaluate learning to show that something useful is going on.

Now a good assessment is helpful.  It can help instructors see what works and what does not. It can reveal successful schools from less successful ones. The challenge is in the assessment.

Here’s the thing.  It is really hard to assess learning beyond the trivial recitation of factoids. A lot of the people complaining have, most likely, never tried to make questions that can reveal just what was learned. We’re beyond “name the state capitals” or “what year was the War of 1812 fought in?”

Let’s consider something GG teaches, field geophysics.  Now one way to look at this is training students to operate specific equipment in the field–the same stuff they’d do for an environmental firm or an oil company or such not. You could ask a question like “On the Geowonder XZ-1000, in order to get the correct polarity of the observed waveform, what is the correct position of the “Input link” switch?”

Utterly, completely, unquestionably a waste of everybody’s time. Why? A company might use the competitor’s product, the UltraGeo AA-120.  Geowonder might introduce a new machine. Or the graduate might need to work with different data.

OK, so what might be worthwhile? One thing GG tries to hammer on is learning how to recognize problems while in the field.  You do this by looking to see whether your data makes sense, and for that to work you have to understand the physics underlying the measurement to some degree. While the specifics of how you do that will vary from experiment to experiment and from device to device, the basic idea of not just accepting as gospel every number the machine kicks out, of having some way of considering the possibility that a mistake was made-that is important. Questions to evaluate this are harder to write, but these are on GG’s exams.  But probably not on assessment surveys built to see how universities compare.

Another example comes from the use of clickers in classrooms.  (These are devices allowing students to answer multiple choice questions in class from their seats). This was a big part of the Science Education Initiative here at CU, but GG’s observations are that good clicker questions are hard.  Physics has managed to develop some curricula that work well with clickers, but when individual geologists have tossed in clicker questions, they more often than not are regurgitation questions to see if people were listening.

So assessments tend towards the lowest common denominator. Prof. Worthen’s complaint is that the data is poor and so by consuming large chunks of time and effort, this process is a waste. Presumably those clamoring for outcome assessments are wanting good solid information. The challenge then is to figure out what good assessments look like.

Here’s problem number one.  Say we assess something and we find that only 20% of the students have mastered that ability.  The knee-jerk reaction is to say that the school failed in teaching that ability. If you see this across many schools, odds are that particular assessment will be removed. The assessments that will be featured prominently are ones where numbers appear quite successful, at least in some places. These will be more basic (and generally more worthless) results: can a student add a row of numbers or read instructions on a bottle of aspirin?

What you want to know is how much did the school help a student reach her or his potential? It could be that only 20% of the students are capable of some high-level ability, and so having 20% master that ability means the school did a great job. But to make this evaluation, you need to know what ability was latent in the students to start with.  Best of luck with that.

Problem 2 was alluded to above. A good assessment is hard–and this can even be true at more basic skills. How the question is phrased, avoiding testing for other effects, inadvertently making assumptions about background knowledge–these are things plaguing tests like the SAT. GG got to watch as postdoctoral fellows here tried to make a post-course concept inventory questionnaire, and it took a lot of effort and even then was only dealing with a few concepts.

Problem 3 is assessing what actually matters, which includes how well education is ingrained in students long term.  This is not done often and this can be important.  Some years ago GG’s home department surveyed current students and alums about which courses were most useful.  Current students hated the required physics and chemistry.  Alums lauded those same requirements. Determining what to assess is harder than you might think: certainly students as they emerge might mistake missing vo-tech knowledge (like how to run the Geowonder XZ-1000) as a big failing while not recognizing the other knowledge that carries them through their careers.

Certainly more assessing than was done in, say, the 1950s is worthwhile. But have we passed that point because the assessments being done are not worthwhile? That would seem the point of the op-ed, and GG is inclined to agree.