The Psychological Methods Lab is committed to avoiding ‘model myopia’ in psychology, or the idea that only one correct interpretation of research data is possible. Instead, they emphasise the power of diverse models and argue that methodological diversity is crucial for reliable science.
Doing justice to uncertainty in research
Does playing violent games make you more aggressive? Does cognitive behavioural therapy work against depression? Socially relevant questions that scientists like to find clear answers to. After research, crystal clear and unequivocal conclusions often follow: no, you do not become more aggressive from violent games; yes, cognitive behavioural therapy helps against depression. Yet such conclusions can be undermined by a subsequent study where results are not replicated. In other words, the methodological approach does not always do justice to diffuse reality.
For twenty years, the Psychological Methods Lab (PML) has been committed to promoting robust and transparent statistics that take more account of uncertainty in psychological research. The research group aims to break the status quo where one researcher reports the results of one specific analysis, conducted with only one model and often using outdated methodology. Their critical research shows that the idea that there is only one correct interpretation after analysing research data is a fallacy. To illustrate, the team asked four different research groups to analyse a simple dataset. This produced four completely different results, all of which differed from those of the original authors. The message: diverse models lead to diverse conclusions, and for reliable science it is essential to embrace and explicitly name this diversity in the social sciences.
"Diverse models lead to diverse conclusions"
The research group actively contributes to increasing diversity in research methods. For example, they are developing new network approaches that allow psychological constructs, such as IQ or mental disorders, to be understood as a complex of interacting variables. In traditional psychology, problems in mental disorders are often seen as isolated symptoms that can be explained from a single ‘cause’. For example, if someone sleeps poorly, broods, and becomes increasingly insecure, these may be considered signs of possible depression. In the network approach, however, all symptoms are put into a broader perspective to see how they interact. After all, if you sleep badly, you get tired and make more mistakes, after which you start feeling insecure and fretting and you may sleep even worse. In short, the network approach shows how these kinds of symptoms can cause and influence each other. PML’s innovative models make it possible to map this complex interplay and make more realistic predictions, for example about the course of mental illness.
The team’s next goal is to develop innovative methods that help prevent hindsight bias. Suppose a researcher tests a hypothesis suggesting a link between coffee drinking and the likelihood of health problems. After collecting data and conducting statistical analyses, it turns out that there is no significant relationship between coffee consumption and health risks. Hindsight bias occurs when the researcher subsequently decides to adjust the analyses to bring the results more in line with the original hypothesis. For instance, by analysing specific subgroups, using different statistical methods, or removing certain outliers, a significant association may still be found. Although common in scientific statistics, the PML team stresses the need to avoid such adjustments to ensure the integrity of psychological research.
"The team’s next goal is to develop innovative methods that help prevent hindsight bias"
Another special feature is that the team continuously integrates their state-of-the-art statistical modelling into JASP which is an open-source software package which they developed, and which is now used by more than 292 universities in 67 countries worldwide. Open science is therefore a guiding principle in the research group’s work. The new methods to be developed to prevent hindsight bias will also be made available to researchers worldwide through JASP.
All the different lines of research share a common goal: to develop accurate statistical models that enable researchers to extract more reliable information from (psychological) datasets. This goal is both ambitious and bold, given the tendency of most scientists to stick to existing methods. Typical of this research group is therefore their enormous determination: even if they meet resistance, they continue to try. If a method they develop is successful, it can significantly improve the reliability of a significant amount of empirical research.
Denny Borsboom (1973) is a professor in the Psychological Methods programme group at the University of Amsterdam. His research focuses on the conceptual analysis of psychometric concepts, the development of new psychometric techniques and the formation of formalised psychological theory. Borsboom is also the founder of the Psychosystems Project, which is devoted to the development of complex network modelling techniques in the context of psychology. This line of work has led to the novel field of network psychometrics, in which psychometric constructs are statistically modelled using network analysis. Borsboom is currently developing Theory Construction Methodology, a methodological approach for systematically constructing formalised theories in psychology.
Han van der Maas (1966) received his PhD in 1993 for research on methods for the analysis of phase transitions in cognitive development. After a KNAW fellowship, he joined the faculty of the developmental group of the University of Amsterdam, first as associate professor and in 2003 as full professor. In 2005, he became professor and chair of the Psychological Methodology programme group at the University of Amsterdam. He served as Graduate school director and Director of Research between 2007 and 2019. In 2009, he founded Oefenweb.nl, a research-based educational platform where students can improve their maths and language skills.
Dora Matzke (1982) studied Psychology at the University of Amsterdam (UvA) where she obtained her PhD in Mathematical Psychology cum laude in 2014. Since 2020, she has been Associate Professor at the UvA’s Psychological Methods programme group. Her research focuses on developing formal theories of cognitive processes involved in decision making. Matzke has won several research grants, including a Veni and Vidi grant from NWO. She is alumna of De Jonge Akademie and has received the Early Career Award from the Psychonomic Society and the Society for Mathematical Psychology. She has been editor in chief of the journal Behaviour Research Methods since 2024.
Eric-Jan Wagenmakers (1972) is professor of Bayesian Methodology at the Psychological Methodology programme group of the University of Amsterdam. His main research interests are open science and Bayesian statistics. Together with Dora Matzke, Wagenmakers wrote the course book Bayesian Inference from the Ground Up: The Theory of Common Sense, which is available for free at BayesianSpectacles.org. Since 2012, Wagenmakers’ laboratory has been developing the open-source software programme JASP for statistical analysis (jasp-stats.org). He is currently the chair of the Skepsis Foundation, an organisation that aims to subject extraordinary claims to critical examination.
