This is the time of year when tenure and promotion cases start getting going, so this seems a timely moment to consider the question. The answer at the most basic level is twofold: (1) there is more to the job than teaching and (2) evaluating teaching is difficult. Many times this is simply misinterpreted as “teaching doesn’t matter,” which while true in some institutions is not really true.
Take the second part first. How do you measure success in classroom teaching? Most universities have relied on student surveys at the close of class. These are notoriously weak measures. For instance, it is trivial to skew the numbers by providing students higher midterm grades than final grades. These surveys usually coincide with the peak of student anxiety over their grades and so tend to accumulate a fair bit of vitriol.
A second measure that is reasonably common is peer review of teaching. To do this well would require a lot of effort on the part of the peer reviewer; frequently this is little more than examining a syllabus and watching a lecture or two. Even then, the perspective of a faculty member will often differ greatly from a student. The Feynman Lectures on Physics was once described as being a wonderful textbook so long as you already understood physics.
Efforts are being made, particularly in core-type classes, to identify material students should have mastered and have a standard assessment of their success. This is, for instance, a key part of the Science Education Initiative’s program here at CU, and this can help separate successful from unsuccessful approaches for courses where the material is clearly documented and the assessment is well-vetted. Even here, though, one can ask what the goal should be: is this to assure that the greatest number meet the lowest bar? That the class average be as high as possible? That the most successful are engaged and succeed beyond their initial position? Toss in the reality that developing valid course assessments is hard and any variation between classes of the same course in material covered and you find that this isn’t going to be a frequently useful tool.
But let’s return to that first part: classroom teaching is only part of the job.
Yet another small random gripe from having been in Europe and back in the States…
Why do Europeans use the yield sign everywhere? Well, yes, there are some stop signs (more than in New Zealand), but when getting on a motorway/freeway/autobahn, most of the time the yield sign really means that you are to merge–you get an acceleration lane for a short time. The US merge sign makes a lot more sense, especially as sometimes you get on these kind of roads and there is *no* acceleration lane–you had better stop unless there is a gap in traffic right there.
On the flip side, why are we in the US so enamored of the stop sign? Virtually every intersection in the US has a stop sign or two or four, yet in many cases what is more appropriate is, ahem, the yield. This is why the rolling (or “California”) stop is so common–the driver can see there is no oncoming traffic and losing all momentum only to crank the car up again is a waste.
Seems everybody could improve some by looking at what everybody else is doing (a truism across many fields, actually).
[8:40 MST 8/15/15–updated to add some clarification and replace the figure with a more interpretable one].
OK, not quite do it yourself, but it is looking like in the coming year armchair geophysics nerds can take their own read on an interpretation of vertical uplift rates in the Sierra Nevada.
The background is this: We are now accumulating enough of a recording history of permanent GPS monuments that vertical uplift rates might be emerging from the noise. (Vertical rates are far harder to measure than horizontal ones because (1) there are no GPS signals coming from below a station and (2) atmospheric/ionospheric effects tend to be more significant in the vertical and (3) vertical rates of mm/yr are much lower than cm/yr rates typical of horizontal motions). In the Sierra Nevada, a Geology paper by Hammond et al. in 2012 argued that the vertical signals in the Sierra were now out of the noise and showed that the range is currently rising in the 1-2 mm/yr range. A 2014 paper in Nature by Amos et al. reinterpreted much of the same data quite differently, arguing that what was being seen was an elastic expansion of the lithosphere as water was being pumped out of aquifers in the Central Valley just to the west.
[Addendum as some of this was unclear. The elastic response is like that of a spring. Imagine you have a mattress with a weight on it–say a heavy suitcase. When you pick up the bag, the top of the mattress rises up as the springs in it are no longer compressed. Also the areas around the bag rise up some. This elastic response is different from the slower response of inflowing asthenosphere that produces post-glacial rebound such as is seen around Hudson Bay.]
It is possible (indeed, likely) that both processes are active, but untangling the two isn’t trivial. Read More…
Thinking a bit more on post-publication review and what not and it occurs to GG that the term “publication” is in flux. In the past, a publication was a fixed thing. The publication I read when the journal first came out 20 years ago is identical to the one you might read 20 years from now. But if we take the premise of post-publication peer review seriously, just what is a publication?
This might be getting to the heart of what peer review is good for. GG’s impression of post-publication review is that it is nearly always focused on “is this paper correct?” or occasionally, “is it fraud?” (since there is little such formal post-publication review in earth science, this might be a misunderstanding). As such, there is certainly value in this activity. But what action, if any, should the author take upon receiving such review? The extremes–leave the paper as is or retract it–exist in the non-post-publication-review world. But what if an author changes a figure, or adds a new section?
Many of us are already familiar with the need to not only cite a source on the web but to indicate the access date as some such materials change over time. Currently, though, this refers to archived datasets or documentation for software or some such material. Is this where we want the scientific literature to go? Are we to cite “Smith et al., The Great Paper, v2.1”? Might we end up citing different versions of the same paper in a single work? Are all the versions to be archived? Scientists change their minds: Einstein, for instance, viewed the cosmological constant in some of his earlier work as a huge mistake. Would he have edited it out and made it vanish from the literature? Or to take an earth science example, Joseph LeConte was an early advocate for a glacial origin for Yosemite Valley, but later in life he decided he was wrong and Yosemite was a graben. Would it have benefited the field for him to have removed or replaced the set of observations he made earlier? Or in these cases, was it better to have both their early thoughts and their later ones available for consideration?
If post-publication review doesn’t allow for changes to a paper–just a sort of global thumbs up or down–then how is it really peer review? It does not provide the author an opportunity to refine the paper to better present its points, or to clarify text that was murky. It is basically a social media version of the old-school chat over beer at a meeting leading folks to decide to ignore a flawed paper, or the writing of a formal comment. Evaluating the merits of a published paper is not a new thing. Are we gaining enough by having this kind of review as we lose the permanence of “publication”?
And even if post-publication review journals allow for changes, how common are reviews that seek to improve the communication the author sought? It seems like a lot of the discussions GG has seen focus on disputes, not on clarifying the paper. Is this a useful replacement for pre-publication, traditional peer review? After all, post-publication reviewers are self-selected and unburdened by the need to consider the paper as a whole.
Some journals are exploring an open review arrangement, where a paper exists as a pre-publication open to all while still soliciting formal reviews. This arrangement has several merits: somebody will definitely look at the paper, and if it is of broad interest, lots more feedback will be accumulated (perhaps allowing the author to better refute misunderstandings of the formal reviewers). The author presumably is expected to respond to feedback so obtained, but what if the decision of the editor is rejection? Was the pre-publication a publication? Was it being cited even as it was reviewed? Does this arrangement obviate the desire for post-publication review? For controversial work, will such papers be covered by the popular media before they are fully vetted? Could we end up generating more anti-vaccine type nonsense by having such an arrangement?
Peer-review and scientific publication is at a crossroads, and there are profound challenges across the board. But the full implications of all the solutions need to be considered. We need to identify what we need the scientific literature to look like and then work back to how it can be made to be so. Adding thumbs-up and thumbs-down buttons and a comments section isn’t necessarily a positive development….
Yosemite Valley in California in the Sierra Nevada was the very first piece of land laid aside by the federal government for use as a public park; it was the first place in the U.S. where the term “national park” was applied. (Some other day we can discuss the more peculiar situation at Hot Springs National Park, which had some land reserved even earlier). It remains one of the most popular parks. Yet later John Muir tried to use Yosemite as a common noun, terming large trunk valleys that had been glaciated “yosemites.” This never took off, but was he right in contending that the other valleys were pretty much the same deal?
The short answer seems to be no, and the difference between Yosemite and those other valleys seems to be a big part of the reason that Yosemite is unique. Consider how this was described by Frederick Law Olmsted:
There are falls of water elsewhere finer, there are more stupendous rocks, more beetling cliffs, there are deeper and more awful chasms, there may be as beautiful streams, as lovely meadows, there are larger trees. It is in no scene or scenes the charm consists, but in the miles of scenery where cliffs of awful height and rocks of vast magnitude and of varied and exquisite coloring, are banked and fringed and draped and shadowed by the tender foliage of noble and lovely trees and hushes, reflected from the most placid pools, and associated with the most tranquil meadows, the most playful streams, and every variety of soft and peaceful pastoral beauty.
This union of the deepest sublimity with the deepest beauty of nature, not in one feature or another, not in one part or one scene or another, not any landscape that can be framed by itself, but all around and wherever the visitor goes, constitutes the Yo Semite the greatest glory of nature.
What exactly is the crux of this?
Looking at a piece recently about peer review and you get the impression that this is the most worthless exercise imaginable, full of selfish reviewers damning papers for no good reason other than to preserve their own place in the world, failing to fix the multiple problems in manuscripts, and generally just taking up space and time. Let’s blow the whole thing up!
GG gets the feeling that these articles (and they seem to show up about once a week somewhere; Retraction Watch tends to cite them in their weekly roundup) are things some editors at some websites and publications sit about and try to solicit. So it seems like people are encouraged to write these by being told: “Have you had problems with peer review? Tell us your horror stories! Blow the lid off this sleazy operation! Get the revenge you’ve been yearning for against all those ungrateful bastards who made you add useless prose and cite irrelevant papers! We’ll share it with the world!”
GG would like to suggest a different question, and one far more appropriate and to the point. “Have you ever learned of a major failing within one of your manuscripts and been able to correct it because of peer review?” If the answer is truly and honestly no, you can feel free to blare your trumpet about how worthless peer review is. But if the answer is yes, would you have been happier making that mistake in print (or in publicly available electrons)? And yes, GG has gotten feedback in review that, in one paper, resulted in some errors in equations being fixed, and in another in a section being added that really did need to be there for the points being advocated to make sense (that from a review recommending rejection, by the ways). GG thinks he helped two authors avoid publishing papers with actual errors by reviewing material carefully (but you’d really need to ask those authors out of GG’s earshot). Peer review is misunderstood, not only by the public, which mistakes it for some kind of seal of approval, but by lots of scientists, who seem to view it as an opportunity to squash others’ opinions. What it should be is an opportunity to help other scientists present their observations and insights most clearly and effectively so that they can be the basis for new work.
In that piece that inspired this rant, one of the authors basically said that nobody else wrote useful reviews. Although it was a bit tongue-in-cheek, there was a sense that this was kind of true. GG has had the chance to see a lot of reviews over the years and this is either total BS or there are fields where all the practitioners need a good spanking. GG has seen inspired, careful, thorough reviews that arguably were worth publication themselves (these gems are perhaps the best argument for some kind of public version of peer review). Are they rare? Sure. But they show how positive peer review can be.