| Story Views | |
| Now: | |
| Last Hour: | |
| Last 24 Hours: | |
| Total: | |
Dr Robert Brown: The pitfalls of extrapolating trend function fits to data
I picked up this comment from Robert Brown on a WUWT discussion of the latest UAH anomaly from Roy Spencer, who no longer adds his ‘for entertainment only’ polynomial trend fit to the data. Michele Casati’s friends would be wise to take heed.
rgbatduke says:
Fitting data with meaningful functions is a basic statistical technique. Fitting data with arbitrary smooth functions is indeed meaningless — literally — and produces nothing more than “a guide to the eye” even if the fit works. In particular, it has little predictive power — many functions will interpolate or approximate a finite data set decently but have very different behavior outside of the fit interval, and the actual data (as it continues outside of the fit) cannot agree with all of them and may not agree with any of them.
Roy [Spencer] understands this perfectly well, because Roy actually understands mathematics and statistical methods pretty well. I personally am rather an expert on predictive modeling methodology, and wrestle with these problems all of the time. There are a few modeling techniques that effectively fit with a large basis to obtain some extrapolative success — neural networks, for example — but methods based on harmonic series e.g. Fourier analysis are notorious for their problems.
That isn’t even an issue here. Roy is (was) fitting a trivial smooth nearly harmonic curve that we all know would not extrapolate on the far side to fit the existing past data, let alone future data. I’m guessing he left it out because its inclusion was too encouraging to folks like Henry P who think that because they can achieve a crude approximate fit with a few hand-selected fourier components that they have discovered the secret of the ages and can thereby predict the future progression of global temperature, to the point where he is “certain” that his model is correct and therefore UAH’s actual data must be wrong. It also distracts the observer from doing the obvious — letting the data speak for itself.
In a noisy, chaotic series like this where there is little reason to select any given analytic basis as being “meaningful”, even linear fits are pretty meaningless, especially when performed on a remarkably short data series. You can see this lack of meaning by observing the rather large variation in the “best fit” slope of a linear trend resulting from “cherrypicking” the end points within the series. Shift the end point a few years, or even months, at either end and you can make the slope vary by a large factor, implying that even the linear trend in the data is highly uncertain.
The Koutsoyiannis paper on stationarity vs nonstationarity hydrology — the one that caught my eye several years ago as a “playah” in this game — has one of the best discussions of this point that I’ve ever seen. It should be required reading not only for all climate scientists (although especially for them) but for all scientists, period. He presents a series of fits to successive windows on the same set of data to show how utterly misleading and meaningless they all are as to the actual (rather simple) functional behavior of a given data set generated from an analytic function plus noise. I commend it to you:
http://itia.ntua.gr/en/docinfo/673/
(click on the preprint PDF or download the possibly paywalled actual publication). Note also that this page has a number of his other papers on this general subject — which broadly speaking is the introduction of what he calls Hurst-Kolmogorov statistical analysis into climate science, a sort of punctuated equilibrium model that describes one particular aspect of global climate almost astonishingly well (I reiterate, Bob Tisdale needs to apply it to SSTs as they obviously are a “perfect fit”).
Henry P would also benefit tremendously from reading the introduction to this paper. Note well that Koutsoyiannis is keeping it simple and only illustrating three distinct windows onto the data. He is also keeping it functionally simple as none of the functions that appear to fit in a window are unique even in that window — a linear fit or exponential fit would clearly work nearly as well as his parabolic fit, a fifth order polynomial would fit the entire sequence pretty well (and if not fifth, sixth or seventh — Wierstrauss’ theorem after all). Koutsoyiannis himself points out that if the series continued, his beautiful cosine fit could turn out to be fitting nothing but noise on an even longer timescale meaningful trend!
The point of this — in case it eludes you — is that merely fitting a finite segment of data to a functional representation of any sort is right up there with Tarot and Tea Leaves as far as having predictive value is concerned. A fit that worked remarkably well — “convincingly” well to the uninitiated — in the first two windows proves to be completely and damningly wrong in the third, not only wrong but literally irrelevant — even as the third window seduces you to conclude that the cosine law is itself meaningful just because it works across this finite sample.
Fits like this have some degree of reliability and extrapolability under only two circumstances. One is when there is a sound physical argument to support the use of some particular fit function, one where the parameters of the fit themselves provide actual information about the physics or other dynamics of the system. The other is where, over time, empirical experience is that some particular fit scheme just works, at least so far, in a robust way over a very long series and works — so far (!) — to extrapolate the series as time continues to evolve. The latter is enormously dangerous — so much so that Nassim Nicholas Taleb wrote an entire book (The Black Swan) criticizing its widespread use in pseudoscientific modeling of essentially unpredictable series wherein “black swan events” are known to occur with some unknowable frequency. All too often they support some sort of Martingale system — doubling down to ride a comfortable linear trend. Sometimes, however, the model fit scheme does have meaning and correspond to physics, we just don’t know how (yet), or is so general that when built in a certain way the model itself can replace the human brain and make information-theoretic compressions that correspond to unknown but real internal dynamics that are empirically safely extrapolable.
The latter is one of my primary businesses — literally, as fits/predictive models of this sort are enormously valuable when you can build them — but they are not for tyros to undertake. I’ve often thought about building a really clever neural network to model the entire planet, but this is a ten million dollar five plus year sort of project even for me — the computer required to build and run and refine it would all by itself be pretty expensive — but if done very cleverly I think it could manage the integrodifferential evaluation back to timescales in the remote past, robustly, allowing for noise and missing information to give us perhaps the best possible model for global climate ever built (and one that by its nature would be utterly unbiased, as AFAIK it is literally impossible to introduce a meaningful and deliberate bias into a standard neural network build process).
rgb
2012-11-08 15:32:40
Source:
