Beyond Null-Hypothesis Testing

Participants will learn to apply open-science principles and cutting-edge statistical techniques that ensure maximally informative analyses. Contemporary issues - such as publication bias, questionable research practices, the statistical evaluation of (non-null) hypotheses, and methods for evaluating the same research question with multiple (replication) studies - are also addressed.

The course will be non-technical in nature and is targeted at PhD students and researchers who want to apply the presented approaches to their own data.

The evaluation of hypotheses is a core feature of research in the behavioural, social, and biomedical sciences. Over the last decade, the replication crisis has drawn a lot of attention to high rates of false-positive results of hypothesis tests in the literature, caused by problems such as publication bias (selective publication of positive results), ‘questionable research practices’, and statistical misconceptions. This course will give participants a non-technical introduction to the theoretical basis of hypothesis evaluation from different perspectives (frequentist, information theoretic, and Bayesian,) and teach how to apply hypothesis evaluation appropriately to avoid fooling oneself and others.

The course is targeted at students and researchers who want to improve their understanding of how to evaluate theory-based hypotheses.

The first day of the course will cover classic null-hypothesis significance testing (NHST), common problems with NHST (misconceptions, questionable research practices, publication bias), open-science practices to avoid these problems, and how to draw more informative inferences with equivalence testing.

The second day will focus on hypothesis evaluation using model selection. Model selection provides an alternative to dichotomous decisions that are the default in NHST and allows more nuanced inferences. Two types will be discussed: information theoretic model selection, that is, model selection using information criteria, and Bayesian model selection (BMS). For both types, the focus will be on informative, theory-based hypotheses (as opposed to null hypothesis testing). The methods that will be covered include the AIC-type criteria called the GORIC and GORICA, GORIC(A) weights, Bayes factors, and posterior model probabilities.

The third day of the course will address informative hypothesis evaluation for multiple (replication) studies, including both direct and conceptual replications. Attention will be paid to (among others) hypothesis updating and combining evidence from multiple studies addressing the same research question (using both GORICA and BMS).

Each day consists of lectures, including small hands-on sessions, and ends with a lab meeting, where there is also room to work on your own data.

Participants are requested to bring their own laptop. Software will be available online.