M&M Hockey Stick Audit Archive
ARCHIVE OF SUPPORTING MATERIALS:
Our dealings with Nature. In Fall 2004 we posted this page explaining our interactions with Nature over MBH98 and the Corrigendum of July 2004.
Climate2003.com. This was Steve's first companion web site which he uses to archive
supplmentary information on the data and computations. It includes
a step-by-step guide to the proxy data audit, an annotated discussion of the data sources and a step-by-step guide to the computations used to try and replicate the
Mann results and to compute the corrected results.
- Background to the 2003 paper (how the whole project got started)
- Results of the 2003 study
- Supplementary data information for M&M03
- Frequently Asked Questions (and Answers) about M&M03
- Back to Project Page.
ARCHIVE OF NOTES AND UPDATES PRIOR TO JAN 27 2005: (See also Steve's Climate2003 Site)
UPDATE: January 7, 2005:
As was noted below, over the next short while there will be some papers coming out which will complete the analysis of the differences between the MBH98
and the MM03 results, and we are reserving commentary as much as possible until post-publication. Steve is putting some preliminary material up at
The technical matters will be fully resolved at that time, but some
of the issues will require careful study for readers to appreciate the precise points of
disagreement. We will do our best in the papers themselves
and on our supplementary information sites to present clear, transparent explanations so interested readers can digest the arguments and form their own conclusions.
Go to publications page
In the interim, feelings are running high in some quarters and there is a certain amount of noisy sabre-rattling going on, such as the new posting by Professor
Mann on January 6 at
Realclimate. Reading this posting it is surprising how far the ball has moved down the field
since the fall of 2003. It is now conceded, for instance, that MBH98 used a decentering procedure that elevated the role of a small group of hockey stick-shaped
proxies from the low-order PC#4, right up into the dominant PC#1. This raises a host of important questions about the robustness of the
MBH98 results, and one unimportant question. Mann's posting focuses on the unimportant one and hammers at it with furious energy. Our papers will raise and address
the important questions, as well as the unimportant one.
In the meantime I would only add a personal note of regret that our debate with MBH98 has been the occasion
for such an ugly public insult to be thrown at a respected scientist named Richard Muller. I hope the arrogance and hostility of this latest entry on RealClimate is
not an example of the kind of writing they plan to publish on a regular basis.
UPDATE: December 13 2004:
Over the next 2-3 months there will be a series of papers coming out from both sides which, hopefully, will lead to a definitive resolution of this debate. The
material has already been exchanged between teams during the Nature correspondence, and now it's a matter of spelling it all out in various print forums. There is a
lot of material and it takes time to work through, so we caution readers against jumping to conclusions until everything is out.
There is a forthcoming paper by Mann et al. (+a few friends) in Journal of Climate as well as web postings at
their new blog; meanwhile Steve McIntyre is at the American
Geophysical Union meetings this week to present a poster on our work.
While we would like to reserve most of our comments for later, Steve has put up some initial reactions to the
JClim and realclimate material at his web page. We have 2 new papers under review.
We look forward to being able to publish our take on all the issues that have piled up over the past year.
The appearance of the new Mann paper and web site is helpful for us as
there are points made therein that we were having trouble convincing people of, especially concerning the effect of their unusual principal
components (PC) algorithm. We invite interested readers to examine our
fall update and then to have a careful read of Mann's new posting.
There is a discussion
here all about the PC issue. Some of the space is taken up with the old issue that in M&M03 we
did not apply the stepwise rostering changes that were used in MBH98. There was nothing nefarious about that, the stepwise scheme was simply not described in
MBH98. Once the rostering sequence was revealed last fall Steve implemented it, and in our
submission to Nature last January we used it, and have used it ever since. The differences between MBH98 and MM03 do not trace to this issue,
but instead to the nonstandard PC methodology in MBH98 and the treatment of the early portion of a Gaspe proxy.
The Mann essay contains a bit of jargony blather about how the MBH98 data decentering
"convention" does not influence (their emphasis) important properties of PC analysis, "such as the orthonormality of the EOFs or the completeness of the
eigenvector basis set." This is mere distraction--of course decentering the data doesn't undo basic theorems of linear algebra. What matters is that
the decentering affects the MBH98 calculations. This is shown
here. Under the MBH98 algorithm (Table a) a small group of less than 20 bristelcone
pines--which are the hockeystick shaped proxies--explain 99% of the variance of PC1, which in turn explains 38% of the variance of the NOAMER group. On this basis the
hockey stick shape was deemed the dominant pattern of climate variability in MBH98. Under a conventional
PC algorithm (Table b) the influence of the bristelcone proxies falls to PC#4, which explains only 6.6% of the NOAMER variance (the numbers are
roughly the same using the data up to 1980--in this example they terminate at 1971). This change is dismissed as a move "slightly lower down in rank".
Unfortunately they do not graph the corrected PCs 1-3, which are the truly "dominant" patterns of variability in the data set. If PC4 is removed from
the list of indicators the hockey stick shape in the final Northern Hemisphere temperature index goes with it. They insist that the Preisendorfer
rule mandates PC4 must be retained, though as Steve has noted on his web site we cannot verify that this rule was used to determine the number of PCs to retain in the
other proxy networks. Anyway, even granting the application of the test for the NOAMER roster, the issue is robustness: how
reliable are conclusions that hinge on the presence or absence of a PC4 that accounts for only 6.6% of the variance in one of the proxy
rosters, and which reflects the 20th century growth spurt in a set of bristelcone pines that is likely not a temperature signal. The
regression exercize for Figure 3 is an alternate way of doing the fitting procedure that still allows the bristelcones to dominate the
results. A better test of robustness would be to remove the 20 problematic bristlecone series and see if the results change. That was done
by Mann here but the results were not
reported in MBH98, nor are they discussed at realclimate.org. Finally, there are comments about the St. Anne river (Gaspe) series, which MBH98 used
in 2 places in their data set. By removing it from one place (they don't say which but I assume it's retained in the NOAMER roster and removed as an
individual proxy) they find the RE stat falls from 0.51 to 0.22, in their words "modestly lower". We'll have more to say about the Gaspe series and
the question of what a significant RE statistic is in the context of MBH98 in our next papers.
UPDATE: November 24 2004:
We have received permission from Nature to post the referee comments on our paper, so the
Fall Update page is back to its original form.
UPDATE: October 28 2004:
Portions of our
update page , including quoted segments from reviewer reports, were deemed confidential by Nature and we have had to remove them from this site. We are
working to resolve the problem.
UPDATE: October 25 2004:
A paper providing many of the details of our most recent analyses of MBH98 has been accepted for presentation (as a poster
session entrant) at the American Geophysical Union Meetings (Paper PP53A-1380). My coauthor Stephen McIntyre will attend the AGU '04 meetings.
A number of internet commentators have disputed the substance of our Monte Carlo analysis of the MBH98 principal component algorithm. Steve McIntyre has posted
some comments on his web page.
some people have asked for our Monte Carlo scripts. These will be posted
on Steve's web page shortly. No doubt the same people will be asking Professor Mann for the programs he used to calibrate and generate his hockey stick
graph--please notify us if and when these are made available.
UPDATE: October 18 2004:
Richard Muller of Berkeley has written
about the recent findings of this project and some of their implications.
For those who would like to visualize the problem with the PCA as implemented in Mann et al., consider the following graph:
One is the MBH98 temperature curve and the other is the result of feeding trendless random numbers into the MBH98 computer program. If you have
trouble seeing which is which, that, in effect, is the trouble.
UPDATE: September 13 2004:
publication last year and the response by Mann et al., we planned a 3-part reply. The
concerned the provenance of the data used for our analysis and was released in November
2003. The second part would itemize many additional lacunae and inaccuracies in MBH98
descriptions of data and methodology identified through examination of Mannís FTP site,
and the third part would show that two undisclosed and questionable methodological
decisions in MBH98 accounted for virtually all of the difference between our results and
MBH98. These projects went through a lengthy review process with Nature, with interesting results.
We can now provide a public update on this process. Because of its length the new material is on
A NEW PAGE HERE
and begins with the latter item first.
UPDATE: July 1 2004:
The Corrigendum in Nature today (July 1, 2004) by Professors Mann, Bradley and Hughes is a clear admission that the disclosure of data and methods behind MBH98 was
materially inaccurate. The text acknowledges extensive errors in the description of the data set. Even more important is the new online Supplementary Information
(SI) site, which concedes for the first time that key steps in the computations behind MBH98 were left out of (and indeed conflict with) the description of methods in
the original paper.
These items were published on the instruction of the Editorial Board of Nature in response to a Materials Complaint that we filed in November 2003. That our
complaint was upheld and the Corrigendum was ordered represents a vindication of our view that, prior to our analysis, there had been no independent
attempt to verify or replicate
this influential but deeply flawed study, something which was forestalled, at least in part, by inadequate and inaccurate disclosure of data and methods.
This is only the first step in resolving the dispute we initiated last fall. The Corrigendum and the SI contain the gratuitous claim that the errors, omissions and
misrepresentations in MBH98 do not affect their results. If this were true, then a simple constructive proof could have been provided, showing before and after
calculations. This is conspicuously missing from the Corrigendum and the new SI. We have done the calculations and can assert categorically that the claim is false.
We have made a journal submission to this effect and will explain the matter fully when that paper is published.
Further, detailed comments on the Corrigendum and new SI will be released shortly.
UPDATE: June 28 2004:
We are receiving daily inquiries from around the world about the status of this project, and we appreciate the continued interest. The previous update was, evidently,
premature in timing, though accurate in describing our intention to vindicate our original conclusions. We have had some
indication that a technical glitch related to manuscript production has been resolved. Sometime in the next month we anticipate being able to publicly resolve a
portion of the current debate.
UPDATE: May 3 2004:
We anticipate being able to make a comprehensive update within the next few weeks, which will include information about several forthcoming papers. We will provide a
detailed explanation of the differences between our results and those of MBH98. We will show why the explanation offered by Professor Mann (see notes below regarding
the three "key indicators") does not work, since we can get our results using the full NOAMER roster and the SWM roster (the latter doesn't make any difference
anyway). There are some core methodological differences between their results and ours which drive the changes. There is also another key series involved, but not
one that MB&H referred to. We stand by our original conclusions, and are now in a position to explain more exactly what went wrong in MBH98.
NOTE (MAY 3 2004):
The climate2003.com web site remains down for reasons we cannot ascertain. The site has been renewed but Steven McIntyre has not been able to access it to re-post the
contents, despite complaints to the ISP. We will keep trying. NB: July 5, climate2003.com is back up and updated.
NOTE (APRIL 7 2004):
The climate2003.com web site is temporarily down due to technical issues. (Well, an unpaid bill actually.) The domain has
been reactivated and the content will be reloaded in a few days.
UPDATE: March 19, 2004
In response to a couple of requests for an update, here is a quick one. A few papers are undergoing review at some journals, including Steve's and my
contribution(s), and there is not a dull page to be found in any of them. Unfortunately that's all we can report. While we are waiting for the review process to
complete, Mr. McIntyre has been winning squash tournaments and I have started bagpiping lessons.
UPDATE: January 22, 2004
Despite the long quiet on this page, the past 7 weeks have been very busy for us. A number of people have written to ask about progress on Part II, while others have
interpreted the 7 week gap as a sign that maybe we ran out of material. No, there is a lot of material, and the challenge has been to sift through it and put it into
coherent form. There are now some new journals involved in handling material that arose from our paper, and we have held back releasing any of the Part II contents
connected to these review processes. Professor Mann's response focuses on the role of 3 (out of 22) key indicators available in the 15th century portion of the data
base. His calculations show that without these series the MBH98 results would look like ours, and his assertion is that we improperly "omitted" the series in
question. Our response will establish that the series in question are in fact inadmissible. Of course the discovery that the 1998 conclusions rest so sensitively on
only 3 series already points to the lack of robustness of this famous graph. But there is much more to be said, when the time comes.
UPDATE: December 1
We are continuing to work on Part II of our response, which has required a detailed examination of Professor Mann's ftp site, hence the delay. We also traveled to
Washington DC on November 18, to present a briefing on Capitol Hill, sponsored by the Marshall Institute and the Competitive Enterprise Institute, on our work to
date. A transcript will be published in a few days, we're told. Some interested experts at a European climate lab had privately criticized us for what they regarded
as an insufficiently wide circle of reviewers for the E&E paper. We offered to them that they could review Part II before its release, on the condition that if they
found errors they could hold us to public account to rectify them, but if the document checks out they would have to issue a statement saying so. After considering it
for a week they declined the offer, saying they don't have time to do the review, and would prefer to follow the debate's progress in journals.
UPDATE: November 13
In the note posted below we comment on the deletion of pcproxy.mat and pcproxy.txt from the ftp site in question. We mistakenly thought pcproxy.txt was still at the
site because it appeared on-screen under an exact address call. That was a browser cache copy. The file itself has been deleted from the ftp site. Also, the time
quoted (Nov 11 1:46 PM) is local Toronto time.
UPDATE: November 11
Our response to the replies thus far from Professor Mann and his colleagues will be presented in three parts. Our overarching goal is to ascertain exactly what data
and what computational steps were used by MBH98, so as to focus in as quickly as possible on the real sources of differences between our results. But along the way
there are a few new isses that must also be dealt with.
available here in PDF format
responds to the claim that the data we audited was prepared in April 2003 in response to McIntyre's request to Mann, and that we ought to have gone to Professor
ftp site instead. We show that the data file we were sent was in existence long before April 2003 and had we gone to the ftp site we would have found it contains the
same data anyway. We also discuss some other pertinent file identity issues. This document, by establishing the practical equivalence between Professor Mann's ftp
site and the data file we were sent, returns our focus to the basic question of data quality and sets the stage for the subsequent parts in which we will
extend our existing critique.
Part 2 will present a detailed examination of the contents of Professor Mann's FTP site, in light of the claim that it is the official repository for the MBH98 data.
This document has been sent to some colleagues for their comments and will be made available shortly thereafter.
Part 3, now under way, will seek to resolve the outstanding differences between our computational methods and those of MBH. Completion of this part will be
contingent on our receiving the specific computer programs MBH used, and we are seeking this disclosure.
UPDATE: November 6 2003.
Mr. McIntyre has a cold. Mr. McKitrick is going to an economics workshop
in Manitoba for a couple of days, to discuss the question "Does the
possibility of climate change imply that we should wash our laundry in
cold water?" His presentation, if you are interested, is
here. So there won't be any updates until
UPDATE: November 4 2003.
Professors Mann, Bradley and Hughes have revised their reply to our paper, see
They have also corrected some errors in their goodness-of-fit calculations.
UPDATE: November 3 2003.
Professors Mann, Bradley and Hughes have made a more detailed reply to our paper, available
here in PDF.
We will have a response prepared shortly.
UPDATE: October 29 2003.
Professor Mann has made a preliminary reply to our paper. His reply and our response are available here
in MSWord and here
Go to Ross McKitrick's homepage