Statistical Methods in Theses: Guidelines and Explanations
Naseem Al-Aidroos, PhD,
Deborah Powell, PhD,
Harvey Marmurek, PhD,
Ian Newby-Clark, PhD,
Jeffrey Spence, PhD,
David Stanley, PhD,
Lana Trick, PhD
Date: May 16, 2016
The purpose of this document is to provide a series of guidelines for students when conducting their thesis research. The replication crisis in psychology has prompted increased examination by statistics/methods researchers of how research is conducted. These investigations are revealing that a large number of questionable research practices are quite common. Moreover, recent analyses suggest that these questionable research practices are a direct cause of the replication crisis (see Kunert, 2016). Therefore, in addition to identifying common statistical and design issues and providing solutions that can often address these issues, we aim to provide guidance that will steer students away from common practices that are ill-advised. As well, we must stress that this document is not comprehensive – conducting excellent research involves doing substantially more than following the suggested practices in this document. Finally, the document was not designed to cover qualitative research approaches.
The non-comprehensive list of questionable research practices provided by John et al. (2012) are listed/quoted below. These questionable practices will periodically be referred to by number.
Questionable Research Practices:
1. In a paper, failing to report all of a study’s dependent measures
2. Deciding whether to collect more data after looking to see whether the results were significant
3. In a paper, failing to report all of a study’s conditions
4. Stopping collecting data earlier than planned because one found the result that one had been looking for
5. In a paper, “rounding off” a p-value (e.g., reporting that a p-value of .054 is less than .05)
6. In a paper, selectively reporting studies that “worked”
7. Deciding whether to exclude data after looking at the impact of doing so on the results
8. In a paper, reporting an unexpected finding as having been predicted from the start
9. In a paper, claiming that results are unaffected by demographic variables (e.g., gender) when one is actually unsure (or knows that they do)
10. Falsifying data
How to use this document
This document was designed to correspond to the order in which students might encounter broad issues as they conduct their research. In addition to the broad main points presented, a number of very specific but important “points to consider” are presented following the references.
The sections below cover open science science considerations, when to stop collecting data, assessing the viability of planned sample sizes, and how to report your methods/analyses. As well, there is a tools section and an "Other Important Points" section that you may find useful.