M&M Hockey Stick Audit Archive


BY

ARCHIVE OF SUPPORTING MATERIALS:


ARCHIVE OF NOTES AND UPDATES PRIOR TO JAN 27 2005: (See also Steve's Climate2003 Site)
UPDATE: January 7, 2005: As was noted below, over the next short while there will be some papers coming out which will complete the analysis of the differences between the MBH98 and the MM03 results, and we are reserving commentary as much as possible until post-publication. Steve is putting some preliminary material up at Climate2003. The technical matters will be fully resolved at that time, but some of the issues will require careful study for readers to appreciate the precise points of disagreement. We will do our best in the papers themselves and on our supplementary information sites to present clear, transparent explanations so interested readers can digest the arguments and form their own conclusions.
In the interim, feelings are running high in some quarters and there is a certain amount of noisy sabre-rattling going on, such as the new posting by Professor Mann on January 6 at Realclimate. Reading this posting it is surprising how far the ball has moved down the field since the fall of 2003. It is now conceded, for instance, that MBH98 used a decentering procedure that elevated the role of a small group of hockey stick-shaped proxies from the low-order PC#4, right up into the dominant PC#1. This raises a host of important questions about the robustness of the MBH98 results, and one unimportant question. Mann's posting focuses on the unimportant one and hammers at it with furious energy. Our papers will raise and address the important questions, as well as the unimportant one.
In the meantime I would only add a personal note of regret that our debate with MBH98 has been the occasion for such an ugly public insult to be thrown at a respected scientist named Richard Muller. I hope the arrogance and hostility of this latest entry on RealClimate is not an example of the kind of writing they plan to publish on a regular basis.


UPDATE: December 13 2004: Over the next 2-3 months there will be a series of papers coming out from both sides which, hopefully, will lead to a definitive resolution of this debate. The material has already been exchanged between teams during the Nature correspondence, and now it's a matter of spelling it all out in various print forums. There is a lot of material and it takes time to work through, so we caution readers against jumping to conclusions until everything is out. There is a forthcoming paper by Mann et al. (+a few friends) in Journal of Climate as well as web postings at their new blog; meanwhile Steve McIntyre is at the American Geophysical Union meetings this week to present a poster on our work.
While we would like to reserve most of our comments for later, Steve has put up some initial reactions to the JClim and realclimate material at his web page. We have 2 new papers under review. We look forward to being able to publish our take on all the issues that have piled up over the past year. The appearance of the new Mann paper and web site is helpful for us as there are points made therein that we were having trouble convincing people of, especially concerning the effect of their unusual principal components (PC) algorithm. We invite interested readers to examine our fall update and then to have a careful read of Mann's new posting. There is a discussion here all about the PC issue. Some of the space is taken up with the old issue that in M&M03 we did not apply the stepwise rostering changes that were used in MBH98. There was nothing nefarious about that, the stepwise scheme was simply not described in MBH98. Once the rostering sequence was revealed last fall Steve implemented it, and in our submission to Nature last January we used it, and have used it ever since. The differences between MBH98 and MM03 do not trace to this issue, but instead to the nonstandard PC methodology in MBH98 and the treatment of the early portion of a Gaspe proxy. The Mann essay contains a bit of jargony blather about how the MBH98 data decentering "convention" does not influence (their emphasis) important properties of PC analysis, "such as the orthonormality of the EOFs or the completeness of the eigenvector basis set." This is mere distraction--of course decentering the data doesn't undo basic theorems of linear algebra. What matters is that the decentering affects the MBH98 calculations. This is shown here. Under the MBH98 algorithm (Table a) a small group of less than 20 bristelcone pines--which are the hockeystick shaped proxies--explain 99% of the variance of PC1, which in turn explains 38% of the variance of the NOAMER group. On this basis the hockey stick shape was deemed the dominant pattern of climate variability in MBH98. Under a conventional PC algorithm (Table b) the influence of the bristelcone proxies falls to PC#4, which explains only 6.6% of the NOAMER variance (the numbers are roughly the same using the data up to 1980--in this example they terminate at 1971). This change is dismissed as a move "slightly lower down in rank". Unfortunately they do not graph the corrected PCs 1-3, which are the truly "dominant" patterns of variability in the data set. If PC4 is removed from the list of indicators the hockey stick shape in the final Northern Hemisphere temperature index goes with it. They insist that the Preisendorfer rule mandates PC4 must be retained, though as Steve has noted on his web site we cannot verify that this rule was used to determine the number of PCs to retain in the other proxy networks. Anyway, even granting the application of the test for the NOAMER roster, the issue is robustness: how reliable are conclusions that hinge on the presence or absence of a PC4 that accounts for only 6.6% of the variance in one of the proxy rosters, and which reflects the 20th century growth spurt in a set of bristelcone pines that is likely not a temperature signal. The regression exercize for Figure 3 is an alternate way of doing the fitting procedure that still allows the bristelcones to dominate the results. A better test of robustness would be to remove the 20 problematic bristlecone series and see if the results change. That was done by Mann here but the results were not reported in MBH98, nor are they discussed at realclimate.org. Finally, there are comments about the St. Anne river (Gaspe) series, which MBH98 used in 2 places in their data set. By removing it from one place (they don't say which but I assume it's retained in the NOAMER roster and removed as an individual proxy) they find the RE stat falls from 0.51 to 0.22, in their words "modestly lower". We'll have more to say about the Gaspe series and the question of what a significant RE statistic is in the context of MBH98 in our next papers.


UPDATE: November 24 2004: We have received permission from Nature to post the referee comments on our paper, so the Fall Update page is back to its original form.


UPDATE: October 28 2004: Portions of our update page , including quoted segments from reviewer reports, were deemed confidential by Nature and we have had to remove them from this site. We are working to resolve the problem.


UPDATE: October 25 2004: A paper providing many of the details of our most recent analyses of MBH98 has been accepted for presentation (as a poster session entrant) at the American Geophysical Union Meetings (Paper PP53A-1380). My coauthor Stephen McIntyre will attend the AGU '04 meetings.
A number of internet commentators have disputed the substance of our Monte Carlo analysis of the MBH98 principal component algorithm. Steve McIntyre has posted some comments on his web page. Also, some people have asked for our Monte Carlo scripts. These will be posted on Steve's web page shortly. No doubt the same people will be asking Professor Mann for the programs he used to calibrate and generate his hockey stick graph--please notify us if and when these are made available.


UPDATE: October 18 2004: Richard Muller of Berkeley has written a column about the recent findings of this project and some of their implications.
For those who would like to visualize the problem with the PCA as implemented in Mann et al., consider the following graph:

One is the MBH98 temperature curve and the other is the result of feeding trendless random numbers into the MBH98 computer program. If you have trouble seeing which is which, that, in effect, is the trouble.


UPDATE: September 13 2004: Following our publication last year and the response by Mann et al., we planned a 3-part reply. The first part concerned the provenance of the data used for our analysis and was released in November 2003. The second part would itemize many additional lacunae and inaccuracies in MBH98 descriptions of data and methodology identified through examination of Mannís FTP site, and the third part would show that two undisclosed and questionable methodological decisions in MBH98 accounted for virtually all of the difference between our results and MBH98. These projects went through a lengthy review process with Nature, with interesting results. We can now provide a public update on this process. Because of its length the new material is on A NEW PAGE HERE and begins with the latter item first.


UPDATE: July 1 2004: The Corrigendum in Nature today (July 1, 2004) by Professors Mann, Bradley and Hughes is a clear admission that the disclosure of data and methods behind MBH98 was materially inaccurate. The text acknowledges extensive errors in the description of the data set. Even more important is the new online Supplementary Information (SI) site, which concedes for the first time that key steps in the computations behind MBH98 were left out of (and indeed conflict with) the description of methods in the original paper.
These items were published on the instruction of the Editorial Board of Nature in response to a Materials Complaint that we filed in November 2003. That our complaint was upheld and the Corrigendum was ordered represents a vindication of our view that, prior to our analysis, there had been no independent attempt to verify or replicate this influential but deeply flawed study, something which was forestalled, at least in part, by inadequate and inaccurate disclosure of data and methods.
This is only the first step in resolving the dispute we initiated last fall. The Corrigendum and the SI contain the gratuitous claim that the errors, omissions and misrepresentations in MBH98 do not affect their results. If this were true, then a simple constructive proof could have been provided, showing before and after calculations. This is conspicuously missing from the Corrigendum and the new SI. We have done the calculations and can assert categorically that the claim is false. We have made a journal submission to this effect and will explain the matter fully when that paper is published.
Further, detailed comments on the Corrigendum and new SI will be released shortly.



UPDATE: June 28 2004: We are receiving daily inquiries from around the world about the status of this project, and we appreciate the continued interest. The previous update was, evidently, premature in timing, though accurate in describing our intention to vindicate our original conclusions. We have had some indication that a technical glitch related to manuscript production has been resolved. Sometime in the next month we anticipate being able to publicly resolve a portion of the current debate.

UPDATE: May 3 2004: We anticipate being able to make a comprehensive update within the next few weeks, which will include information about several forthcoming papers. We will provide a detailed explanation of the differences between our results and those of MBH98. We will show why the explanation offered by Professor Mann (see notes below regarding the three "key indicators") does not work, since we can get our results using the full NOAMER roster and the SWM roster (the latter doesn't make any difference anyway). There are some core methodological differences between their results and ours which drive the changes. There is also another key series involved, but not one that MB&H referred to. We stand by our original conclusions, and are now in a position to explain more exactly what went wrong in MBH98.

NOTE (MAY 3 2004): The climate2003.com web site remains down for reasons we cannot ascertain. The site has been renewed but Steven McIntyre has not been able to access it to re-post the contents, despite complaints to the ISP. We will keep trying. NB: July 5, climate2003.com is back up and updated.

NOTE (APRIL 7 2004): The climate2003.com web site is temporarily down due to technical issues. (Well, an unpaid bill actually.) The domain has been reactivated and the content will be reloaded in a few days.

UPDATE: March 19, 2004 In response to a couple of requests for an update, here is a quick one. A few papers are undergoing review at some journals, including Steve's and my contribution(s), and there is not a dull page to be found in any of them. Unfortunately that's all we can report. While we are waiting for the review process to complete, Mr. McIntyre has been winning squash tournaments and I have started bagpiping lessons.

UPDATE: January 22, 2004 Despite the long quiet on this page, the past 7 weeks have been very busy for us. A number of people have written to ask about progress on Part II, while others have interpreted the 7 week gap as a sign that maybe we ran out of material. No, there is a lot of material, and the challenge has been to sift through it and put it into coherent form. There are now some new journals involved in handling material that arose from our paper, and we have held back releasing any of the Part II contents connected to these review processes. Professor Mann's response focuses on the role of 3 (out of 22) key indicators available in the 15th century portion of the data base. His calculations show that without these series the MBH98 results would look like ours, and his assertion is that we improperly "omitted" the series in question. Our response will establish that the series in question are in fact inadmissible. Of course the discovery that the 1998 conclusions rest so sensitively on only 3 series already points to the lack of robustness of this famous graph. But there is much more to be said, when the time comes.

UPDATE: December 1 We are continuing to work on Part II of our response, which has required a detailed examination of Professor Mann's ftp site, hence the delay. We also traveled to Washington DC on November 18, to present a briefing on Capitol Hill, sponsored by the Marshall Institute and the Competitive Enterprise Institute, on our work to date. A transcript will be published in a few days, we're told. Some interested experts at a European climate lab had privately criticized us for what they regarded as an insufficiently wide circle of reviewers for the E&E paper. We offered to them that they could review Part II before its release, on the condition that if they found errors they could hold us to public account to rectify them, but if the document checks out they would have to issue a statement saying so. After considering it for a week they declined the offer, saying they don't have time to do the review, and would prefer to follow the debate's progress in journals.

UPDATE: November 13 In the note posted below we comment on the deletion of pcproxy.mat and pcproxy.txt from the ftp site in question. We mistakenly thought pcproxy.txt was still at the site because it appeared on-screen under an exact address call. That was a browser cache copy. The file itself has been deleted from the ftp site. Also, the time quoted (Nov 11 1:46 PM) is local Toronto time.

UPDATE: November 11 Our response to the replies thus far from Professor Mann and his colleagues will be presented in three parts. Our overarching goal is to ascertain exactly what data and what computational steps were used by MBH98, so as to focus in as quickly as possible on the real sources of differences between our results. But along the way there are a few new isses that must also be dealt with.
Part 1, available here in PDF format responds to the claim that the data we audited was prepared in April 2003 in response to McIntyre's request to Mann, and that we ought to have gone to Professor Mann's ftp site instead. We show that the data file we were sent was in existence long before April 2003 and had we gone to the ftp site we would have found it contains the same data anyway. We also discuss some other pertinent file identity issues. This document, by establishing the practical equivalence between Professor Mann's ftp site and the data file we were sent, returns our focus to the basic question of data quality and sets the stage for the subsequent parts in which we will extend our existing critique.
Part 2 will present a detailed examination of the contents of Professor Mann's FTP site, in light of the claim that it is the official repository for the MBH98 data. This document has been sent to some colleagues for their comments and will be made available shortly thereafter.
Part 3, now under way, will seek to resolve the outstanding differences between our computational methods and those of MBH. Completion of this part will be contingent on our receiving the specific computer programs MBH used, and we are seeking this disclosure.

UPDATE: November 6 2003. Mr. McIntyre has a cold. Mr. McKitrick is going to an economics workshop in Manitoba for a couple of days, to discuss the question "Does the possibility of climate change imply that we should wash our laundry in cold water?" His presentation, if you are interested, is here. So there won't be any updates until next week.

UPDATE: November 4 2003. Professors Mann, Bradley and Hughes have revised their reply to our paper, see here.
They have also corrected some errors in their goodness-of-fit calculations.

UPDATE: November 3 2003. Professors Mann, Bradley and Hughes have made a more detailed reply to our paper, available here in PDF.
We will have a response prepared shortly.

UPDATE: October 29 2003. Professor Mann has made a preliminary reply to our paper. His reply and our response are available here in MSWord and here in PDF.


Go to publications page
Go to Ross McKitrick's homepage