Stats - See Updates Below

Lubos is back in the climate wars, and he appears to have a point. I haven't read the paper or done the calculations, but his Mathematica principal components calculation seem to show that there is something fishy about some recent Antarctica data. The data is represented as time series of temperatures from 5509 locations. However, essentially all the variance turns out to be captured by the equivalent of just four [Karhunen-Loeve transformed virtual] meaurement locations.

If you represent each time series as a vector, the 5509 measurement points produce a 5509 dimensional vector space - but all the vectors lie within a four-dimensional subspace.

This is quite odd, and suggests a degree of bogosity. Of course it's not odd at all that the temperatures at different locations in Antarctica should be correlated, but the enormous differences in scale of the eigenvalues do not appear compatible with that being the full explanation.

Anybody know what's really going on here?

UPDATE: OK, I looked through the Steig et. al. paper as well as Tapio Schneider's paper developing the statistical methodology they used: Journal of Climate, 14, 5 (March, 2001) "Analysis of Incomplete Climate Data: Estimation of Mean Values and Covariance Matrices and the Importance of Missing Values."

The problem they were dealing with: they wanted to find the best estimate of means and covariances and their behavior over time for the Antarctic Continent from irregularly spaced and sporadic weather station records. The frequent gaps in the record mean that those matrices are underdetermined (the time series vectors have unknown components). Consequently, the statistical estimation problem can be ill posed. That problem can be dealt with by "regularizing" the data (in effect smoothing it). It is a non-linear estimation problem, and Schneider's method is an iterative smoothing method.

The method has some features and effects in common with the more ad hoc method of truncation of principal components, but has more formal justification. The point is that you wind up with only a few principal components because the others would contribute only noise - their effects are smaller than the other known noise sources.

In any case, the methodology explains Lubos's results - he only sees a few principal components because the others have been filtered out in the RegEM estimation. You should try your method on raw data, Lubos. I predict a lot of large (and spurious) principal components due to the data gaps.

Arun's comments are very nearly correct, with the only qualification that the values in question are not so much modeled as the results of an estimation process.

Another virtue is that the principal components have natural interpetations in terms of familiar large scale weather or climate features.

UPDATE II: The Schneider paper is very clear and available online: http://ams.allenpress.com/perlserv/?request=get-document&doi=10.1175%2F1520-0442%282001%29014%3C0853%3AAOICDE%3E2.0.CO%3B2

Comments

Post a Comment

Popular posts from this blog

Anti-Libertarian: re-post

The most pointed critique of social security and liberalism in general is the libertarian critique. The problem with libertarians, for a liberal, is that we have too much in common. We both believe in individual rights, tolerance of individual differences, and dislike government prescription of religion. The basic difference, it seems to me, is the different answers we give to Cain's (with thanks to the Captain - my original version had the corpse asking the question!) famous question: "Am I my brother's keeper?" That's not really my basic bitch against the libertarians though. My real complaint is the same as my complaint against most religion - its premise is a fraud. For those who can't stand to wait for the punchline, I believe that trying to implement libertarian principles leads to tyranny or social disintegration. Demonstrating that takes some historical (and pre-historical) context. For all but the last 15,000 or so of the 100,000 years the human spec...
Read more

Uneasy Lies The Head

Uneasy lies the head that wears a crown ........... Shakespeare Henry IV, Part II. So it is with us and our slightly distant primate cousins , so it seems. A 9-year study of wild baboons in Kenya by Princeton University and the Institute of Primate Research may turn conventional assumptions of alphas males upside down. The study found that alpha males have high testosterone levels, which allows them to dominate other baboons, get access to more food, and attract better mates. This is expected. Surprisingly, though, the alpha males also have high levels of stress hormone glucocorticoid, which scientists measured from their feces samples. In fact, the stress of alpha baboons was on par with low-ranking baboons. The convention wisdom is that access to food and females provides life security and thus less stress for dominant males. Conversely, weak males who are constantly threatened by starvation and physical violence from stronger males are thought to be more stressed. The st...
Read more

We Call it Soccer

The US played its best, and maybe only good game of World Cup, but lost thanks to brilliant goalkeeping by the Swedish keeper and the bizarre method soccer uses to settle ties when three US stars could not put the ball on frame from the 12 yard spot.  At the end it came down to a blocked shot that spun in by maybe a couple of centimeters. Well, that's football - too much like life.   Which brings me to a modest proposal.  Replace the thirty minute overtimes with as many 7-player 15 minute periods as needed to settle it.  Scoring should be much easier with fewer players.
Read more