#replicationcrisis — Public Fediverse posts

Being connected to two fields that have been dealing with the replication crisis in science for a very long time now, it's fun to watch other areas getting drawn in and going through the stages of grief.

Apparently cell biology has entered the "it's just you guys that suck and we're the real scientists!" phase of the process. 🙄

Always fun to meet a True Scotsman. So few of those around these days.

(1/2)

#science #replicationcrisis #venting #noReallyJustVenting

Being connected to two fields that have been dealing with the replication crisis in science for a very long time now, it's fun to watch other areas getting drawn in and going through the stages of grief.

Apparently cell biology has entered the "it's just you guys that suck and we're the real scientists!" phase of the process. 🙄

Always fun to meet a True Scotsman. So few of those around these days.

(1/2)

#science #replicationcrisis #venting #noReallyJustVenting

Being connected to two fields that have been dealing with the replication crisis in science for a very long time now, it's fun to watch other areas getting drawn in and going through the stages of grief.

Apparently cell biology has entered the "it's just you guys that suck and we're the real scientists!" phase of the process. 🙄

Always fun to meet a True Scotsman. So few of those around these days.

(1/2)

#science #replicationcrisis #venting #noReallyJustVenting

Being connected to two fields that have been dealing with the replication crisis in science for a very long time now, it's fun to watch other areas getting drawn in and going through the stages of grief.

Apparently cell biology has entered the "it's just you guys that suck and we're the real scientists!" phase of the process. 🙄

Always fun to meet a True Scotsman. So few of those around these days.

(1/2)

#science #replicationcrisis #venting #noReallyJustVenting

Being connected to two fields that have been dealing with the replication crisis in science for a very long time now, it's fun to watch other areas getting drawn in and going through the stages of grief.

Apparently cell biology has entered the "it's just you guys that suck and we're the real scientists!" phase of the process. 🙄

Always fun to meet a True Scotsman. So few of those around these days.

(1/2)

#science #replicationcrisis #venting #noReallyJustVenting

Reproducibility Woes Plague Economics and Political Science

New studies show many findings in economics and political science cannot be repeated by other researchers. This affects trust in published research.

#ReplicationCrisis, #SocialScience, #Economics, #PoliticalScience, #ResearchIntegrity

https://newsletter.tf/economics-political-science-studies-hard-to-repeat/

The piranha problem: Large effects swimming in a small pond.
Christopher Tosh, Philip Greengard, Ben Goodrich, Andrew Gelman, @avehtari, @djhsu
2 Apr 2024
https://arxiv.org/abs/2105.13445

In a lot of social science research, small, random factors are reported as having large effects on social and political attitudes and behavior (social priming, hormonal levels,parental socioeconomic status, weather, ...). Studies have claimed to find large effects from these and other inputs.

The results show that it would be extremely unlikely to have all these large effects coexisting—they would have to almost exactly cancel each other out.

#socialScience #replicationCrisis #statistics #quantitative #research #science #scientificMethod

Only about half of published papers in social sciences, including economics and political science, can be independently repeated by other researchers. This is a big problem for science.

#ReplicationCrisis, #SocialScience, #Economics, #PoliticalScience, #ResearchIntegrity
https://newsletter.tf/economics-political-science-studies-hard-to-repeat/

There you have it:

"Together, these dynamics suppress replication and cumulative verification, distort the visible scientific record, and misdirect expert attention away from epistemically stabilizing review."

https://link.springer.com/article/10.1186/s41073-026-00193-3

#replicationcrisis #replication #reproducibility

Oktoberfest waiters were said to perform worse due to their expertise.

New preregistered replication, 400+ participants.

Result? No effect.

Even Oktoberfest is not immune to the replication crisis. 🍺

#replicationcrisis #psychology

https://theeconomyofmeaning.com/2026/02/14/even-oktoberfest-is-not-immune-to-the-replication-crisis/

@petersuber #ReplicationCrisis clearly has humbled the scientific community.

🧠💥 Oh, the irony! Cognitive psychology: the science of understanding minds that couldn't even understand itself! 🚫🔄 Turns out, "replication crisis" is just a fancy way of saying, "Oops, we goofed!" 🤦‍♂️ #ScientificFacepalm
https://buttondown.com/aethermug/archive/aether-mug-famous-cognitive-psychology/ #CognitivePsychology #ReplicationCrisis #ScientificIrony #MindUnderstanding #OopsWeGoofed #HackerNews #ngated

#xkcd: #ReplicationCrisis

https://xkcd.com/3117/

#Humour #Cartoon

"We’ve saved the funniest tool for last: the tortured phrases detector. Sometimes researchers copy paste text from other academic papers. To avoid accusations of plagiarism they use tools that automatically rewrite the text. But this doesn’t always go well. Sometimes it results in phrases that sound weird and no longer make sense in their context. ‘Artificial intelligence’ becomes ‘counterfeit consciousness’ while ‘deep neural network’ is changed into ‘profound neural organization.’ Nonsensical terms like these suggest a paper has been produced by a paper mill. Guillaume Cabanac and colleagues have pioneered the detection of tortured phrases in computer science, but others have used it in medicine and found some hilarious examples. ‘Anal canal’ became ‘butt-centric waterway’ while ‘breast cancer’ is often rephrased as ‘bosom peril’."

#Science #ReplicationCrisis #ScienceIntegrity

https://mecfsskeptic.com/how-many-scientific-papers-are-fake/

Before the experiment, they interview an expert to ask:

- What was this experiment testing?
- What's the most likely way we could screw it up?
- How is our setup different from the original, and how might have the original experiment been flawed?
- What are the stakes? If we fail to replicate the experiment, what might that mean for scientific knowledge?

Common experimental errors have muppet avatars, and if the expert mentions them, they come on screen and sing a song.

#ReplicationCrisis

Our presentation about #RegisteredReports from last week. We wanted to introduce this research and publication approach to those of our colleagues who may have been unaware of it. We gave three examples from projects we've recently been involved in.
https://osf.io/skdwj
#OpenScience #ReproducibiliTea #ReplicationCrisis #CognitiveDissonance #Personality #RiskPerception

Happiness research hit hard by #replicationcrisis, though some #happiness hacks seem to have consistent (small) effects.
https://www.theguardian.com/science/audio/2024/nov/14/secrets-of-happiness-the-happiness-hacks-backed-up-by-science-podcast

I trust the scientific method.

I do not trust humans to strictly follow that method.

Science is wonderful. Humans however can be less than awesome when incentives are misaligned from the public good.

The #replicationcrisis has a clear solution. Government scientists paid to replicate boring studies (the sensational ones will get plenty of eyeballs)… it’s the boring studies that slip through the cracks.

Government #science is better than cooperation science…
#leftist #progressive #politics

Finally got to read Yarkoni's "Generalizability crisis", and it didn't disappoint. I am not a psychologist, but it seems to describe some very hard problems that cannot be resolved by replication or open science
https://doi.org/10.1017/S0140525X20001685
#psych #replicationcrisis #OpenScience

so why are #students important as #actors of the #openscience movement? they are directly affected the the #replicationcrisis, they are the scientists of tomorrow, everyone at a university should by involbed, and students can support the cultural change towards open science.
more can be found in Maximilian's slides! #OSDMA24

If not anything, this paper has really good examples of various frameworks used in the context of analysing human behaviour: https://www.nature.com/articles/s41562-018-0522-1. It would be interesting to see if their argument of overcoming/avoiding replication crisis with an overarching framework holds true. The #openaccess version is available at https://eprints.lse.ac.uk/100335/1/Dual_Inheritance_Theory_Perspective_final.pdf
#replicationcrisis #replication

I’m part of the #EEGManyLabs project testing the #replicability of influential #EEG studies. We are using #PredictionMarkets as a tool in this effort and you are invited to take part, especially if you have some expertise in EEG research, no matter how little. See below for details.

You may well know about the success of “prediction markets” in forecasting the likelihood of replication (e.g., Dreber et al., PNAS 2015). We are delighted to announce that we have partnered with economists who led these seminal studies to test the wisdom of the EEG community.

From today (as we near the end of recruitment for this project - please see last calls below), we are opening a survey to ask you to vote on the likelihood of some hypotheses studied in the #EEGManyLabs project. Subsequently, you will be invited to bet on the likelihood of success through a stock market platform, where you will earn real money for you or a selected charity.

The success of this effort will become clear when we complete the full project in a few years time. But the results will immediately tell us about the degree of optimism/pessimism amongst our community.

So, please share this widely and place your bets now...

How can I sign up for the prediction markets? Registrations to participate in the prediction markets are administered via the sign-up form linked below. You must have experience of working with EEG (for example, through collecting and/or analysing EEG data, which may be evidenced by having published peer-reviewed articles or preprints with EEG or equivalent experience e.g. designing, collecting and analysing data from EEG experiments).

#neuroscience #psychology #replication #replicationcrisis #reproducibility #metascience

https://pavlovug-dot-yamm-track.appspot.com/2fDq4NW3JydtlOhg09G1yGVAVSU8W-Whs6kPML9_ZfizxufZcjwEmQPWP5AnyD8NUrSZsBDvk7fYcappAg1Sgo3_tXQFMEu2Cc91Q_7ZoUrnvmZiS20DZiHsSFcGXN3Cpu_h990jl35DhtizskaAYiFIXKDiVMD0arROUO4-wXStjr4hF_n39GH14z3uQM6NK9ioOU86MruRx

@wagesj45 @nothingfuture There is nothing morally wrong with that - unless one considers that the time of researchers a limited resource, or one feels that diluting the visibility of genuine research by inflation is a bad thing. This is already normalized in many research communities, and there is little to loose by accelerating it further. #science #replicationcrisis #sociologyofknowledge

Looking forward to today's #ReproducibiliTea, where we will this paper on positice consequences of the #ReplicationCrisis with first author Max Korbmacher. https://www.nature.com/articles/s44271-023-00003-2 #OpenScience

Are the criticisms of psychological measurement fundamentally flawed or are psychologists indeed not measuring anything, in spite of appearances?

https://www.tandfonline.com/doi/full/10.1080/09515089.2023.2300693

#psychology #replicationcrisis #iknowwhatyouarethinking

Is Psychology Self-Correcting? Reflections on the Credibility Revolution in Social and Personality Psychology

tl:Dr — "No."

https://spb.psychopen.eu/index.php/spb/issue/view/485

#science #psychology #replicationcrisis #handwavingandfingerpointing #SomedayMaybe

All the people who collate #science accounts to follow on Mastodon - is Data Colada on your list?

@datacolada keep on publishing new posts at https://datacolada.org/ , and they apparently are getting sued right now (by none other than Francesca Gino...).

Why would you be interested? Well, they're the ones who got the ball rolling on #ReplicationCrisis.

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

Just written a hopefully accessible introduction to the replication crisis in biology with focus on problems in statistics, and how Bayesian modeling could help. Based on a talk for a cohort of mostly experimental graduate students.
https://manywordsandnumbers.org/2023/09/16/better-reproducibility-in-biology-by-better-statistics/

#replicationcrisis #reproducibility #biology #bayes

Fascinating take on the #ReplicationCrisis - from the perspective of academic #history

Thanks to Stuart Richie's blog - https://www.sciencefictions.org - I found this great piece:

"Age of Invention: Does History have a Replication Crisis?"

The author, Anton Howes, takes inspiration from the replication and #reproducibility issues in psychology and other sciences to look at similar problems with history:

https://www.ageofinvention.xyz/p/age-of-invention-does-history-have

#ResearchIntegrity

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

Critical Metascience:

2022 has been a bumper year for what I’d call “critical metascience” - work that takes a step back and offers a critical perspective in the field.

My Top 10 papers of 2022 in this area are, in alphabetical order… 🥁 🧵👉

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#SociologyofScience
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon

1/12

Replicability and Theory:

“Our results suggest that many of the practices that have been proposed as a means to improve the replicability of psychological research—such as open data and methods…preregistration and Registered Reports…and basing conclusions on Bayesian inference…or p < .005 rather than p < .05…—do indeed improve confidence in replicability among our sample.”

Continued 🙂 🧵👉

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

@MarkRubin This is massively simplistic. Hypotheses include the criteria for delineating phenomena in need of explanation, satisfaction criteria for success, disciplinary standards and practices, and taxonomies of subjects under investigation. IMO.
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

What’s a hypothesis?

“A hypothesis is not simply a guess about the result of an experiment. It is a proposed explanation that can predict the outcome of an experiment. A hypothesis has two components: (1) an explanation and (2) a prediction. A prediction simply isn’t useful on its own.” (Haroz, 2014)

Blog post: http://steveharoz.com/blog/2014/mysterious-origins-of-hypotheses-in-visualization-and-chi/

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

A “quietist” response to the replication crisis:

“The quietist approach proposes that we should just accept that it is in the nature of science that we get things wrong, and that this is particularly true with sciences in early stages of development.”

Bird (2021). Understanding the replication crisis as a base rate fallacy.

🔒 https://doi.org/10.1093/bjps/axy051

🔓 https://kclpure.kcl.ac.uk/portal/files/95454096/Replication_base_rate_fallacy_REV2.pdf

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
@philosophy

Bad Stats / Poor Methods:

Qualitative study finds 39.8% of 548 psychology researchers believe that statistics and/or research methods are misused and/or misunderstood in the field.

Miranda et al. (May 2022). How do researchers in psychology perceive the field? A qualitative exploration of critiques and defenses. Collabra: Psychology.

https://doi.org/10.1525/collabra.35711

#Psychology
#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

No evidence of p-hacking in imaging research:

Analysis of 4,105 randomly sampled p-values finds no evidence of p-hacking in work published in over 100 imaging journals since 1972.

Rooprai et al. (2022): https://doi.org/10.1177/0846537122113941

#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

Looks like a great talk from Stephan Guttinger on Questionable Research Practices

“What should be abandoned is not the idea of questioning practice, but the idea that there is a class of questionable research practices.”

Slides: https://philstatwars.files.wordpress.com/2022/12/guttinger_draft_red.pdf

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#QRPs

“We are not only in a replication but an interpretation crisis, a crisis of theory building.”

Benjamin Krämer (@benjkraemer) (2022, November). Why are most published research findings under-theorized? In Questions of Communicative Change and Continuity.

🔓 https://www.nomos-elibrary.de/10.5771/9783748928232-23.pdf

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon
#Communication

“The appropriate conclusion based on significant social-psychology experimental findings could perhaps be characterized as ‘sometimes this happens.’….Although ‘sometimes this happens’ may be disappointing compared with establishing universal laws, perhaps the field should accept this with both humility and pride.”

Baumeister et al. (2022). https://doi.org/10.1177/17456916221121815

#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilSci
#SocialPsychology
@socialpsych

Replicating experiments:

New preprint considers the “minimum viable experiment to replicate.”

“In this paper, we introduce the idea of a minimum viable experiment that needs to be identified in practice for replication results to be clearly interpretable.”

Devezer & Buzbas. (2022). Preprint: http://philsci-archive.pitt.edu/21475/

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
@philosophy

On the questionable use of “Questionable Research Practices” (QRPs):

“Practices labelled as ‘QRPs’ can be both beneficial and problematic for research practice and targeting them without a sound understanding of their dynamic and context-dependent nature risks creating unnecessary casualties in the fight for a more reliable scientific practice.”

Stephan Guttinger. Talk to be presented on 1st Dec: https://phil-stat-wars.com/3-tentative-schedule/

#Statistics
#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis

🔹 "No field can grow without healthy critique" 🔹

(Berkeley Initiative for Transparency in the Social Sciences: https://twitter.com/UCBITSS/status/1557834242763079680)

Check out this list of over 80 articles that have a somewhat critical view on one or more issues associated with open science, science reform, and/or the replication crisis: https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-articles-critical-of-open-science

Includes a ResearchRabbit collection: https://www.researchrabbitapp.com/collection/public/V467G42MLX

#openscience
#metascience
#psychmethods
#replicationcrisis

Replication crisis:

“Don't get me wrong, I think there are real reasons to be concerned....But I'm not sure calling it a crisis is particularly helpful because I think that can just be a bit distracting.”

Interview with Marcus Munafo, chair of the UK Reproducibility Network

https://www.thenakedscientists.com/articles/interviews/are-reproducibility-rates-concern

#science
#openscience
#metascience
#psychmethods
#replicationcrisis

I am wondering, did anyone already try #CodeInterpreter or equivalents to replicate academic research?! This would be an awesome use case! #Ai #AIAcademicResearch #ReplicationCrisis