Sensitivity Testing (tap tap…1..2..3..)
No, this is not about being careful in what you say, or how quickly you jump if tapped on the shoulder. This is testing for how well an inversion can convince you of the presence of an anomaly.
Seismic tomography is one of those windows into the earth that is either a huge advance or a hall of mirrors. The single greatest challenge is to show that some high- or low-velocity blob is real. Sometimes you can do this by looking at raw travel time residuals, but most of the stuff we are looking at these days is lost in the noise in raw data–it takes the blending of tons of data to get to the anomalies in question. (Seismologists have been wading in big data for awhile now).
Probably the most convincing test is some kind of sensitivity test (or, if you do the full matrix inversion properly, an a posteriori covariance or resolution matrix–but with the numbers of degrees of freedom in most tomographic studies, these are few and far between). A simple form is a checkerboard, but let’s consider a better one, a hypothesis test. As we’ll see, there are unexpected pitfalls.
Suppose we put a body with some seismic anomaly into an otherwise spherical earth model and then calculate what observations this should produce. To be fair, we add in some noise. If we invert the synthetic observations and recover the anomaly, we pronounce it resolvable and so are happy to interpret it when it shows up in our inversion. [An alternative that is arguably better is to remove an anomaly from a final inversion result and then continue the inversion from the modified model–first to see how much the variance increased upon removal of the body and second to see if the inversion can fit the data as well with something else within the more complex final model. Ideally you keep the inversion from modifying the model where your anomaly was.]
There is a big, huge caveat, one rarely brought up in such papers. The caveat is that the physics of your forward model might be incomplete. Here’s an illustration:
Suppose we are interested in a blob some 450 km down and we are doing a teleseismic P-wave inversion using a geometric ray tracer. Let’s say our test shows this anomaly to be robust–is that fair? Well, maybe not. If the body is small enough, in the real earth the seismic waves will diffract around the body and heal up, leaving no anomaly to be measured. The synthetic test didn’t catch this because that physics wasn’t in the forward model.
OK, so now you use a finite difference ray tracer, say, that will in fact produce diffractions. We still are not out of the woods, for it might predict an arrival with such a low amplitude that you could not pick such an arrival. And then there can be issues where the gridding of a model might be too coarse for the numerical approximations to the physics to work properly (this can happen, for instance, with finite-frequency tomography).
It isn’t just body wave tomography that might suffer. Most “3-d” surface wave models are not constructed by propagating surface waves through a 3-D model but are in fact 1-d models stitched together. Would these structures reproduce observations if you actually made proper synthetic data? For the most part, we don’t know. (Arguably there are similar issues with magnetotelluric models as well).
Does this mean all such tests are nonsense? Well, no (at least GG hopes not, having done quite a few himself). But it does mean that you need to keep in mind the limitations of the forward model being used and the possible things it would miss. Usually these limitations don’t make it into the discussion section of a paper and so can be hidden from view.