#metascience — Public Fediverse posts

Some recent #metascience work and news…

• Average replicability rates

• Theories don’t die!

• Is the replication crisis overblown?

• Postdoctoral Fellowships

• Partial replications

https://open.substack.com/pub/markrubin/p/critical-metascience-roundup-f3f?r=2bqjb7&utm_campaign=post-expanded-share&utm_medium=web

and if any one is wrong we can accept it.

In contrast, some big clinical trials maximize rigor at high cost.

A great science ecosystem has a mix of different approaches.

#metascience

Under the banner "The winding road to better science", we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA postdoctoral fellows. If you are interested in developing a project on or across the boundaries of #metascience and #STS, reach out! 1/

Under the banner "The winding road to better science", we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA postdoctoral fellows. If you are interested in developing a project on or across the boundaries of #metascience and #STS, reach out! 1/

Under the banner "The winding road to better science", we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA postdoctoral fellows. If you are interested in developing a project on or across the boundaries of #metascience and #STS, reach out! 1/

Under the banner "The winding road to better science", we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA postdoctoral fellows. If you are interested in developing a project on or across the boundaries of #metascience and #STS, reach out! 1/

Under the banner "The winding road to better science", we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA postdoctoral fellows. If you are interested in developing a project on or across the boundaries of #metascience and #STS, reach out! 1/

#metascience What if we could study all grant applications—not just the funded ones—without compromising privacy?

Introducing MIGA: a minimal standard for open funding data https://doi.org/10.31222/osf.io/9hguf_v1

Helen Pearson: Beyond Belief

In this Plutopia podcast episode, journalist and author Helen Pearson discusses her book Beyond Belief, which traces the rise of evidence-based decision-making in medicine, government, education, conservation, and other fields, arguing that evidence-based practice is both more recent and more fragile than many people realize. Pearson explains how pioneers of evidence-based medicine challenged “eminence-based” authority and helped build systems like randomized trials and systematic reviews, while also emphasizing that evidence is only one part of good decision-making alongside human values, experience, and compassion. The conversation explores how misinformation, influencers, political polarization, and poor communication of scientific uncertainty have eroded trust, especially in the U.S. — but Pearson remains cautiously optimistic, stressing the need to help people ask better questions, synthesize bodies of evidence rather than rely on anecdotes or single studies, and communicate science through engaging stories in the media channels where people actually get information.

Podcast: Play in new window | Download

Helen Pearson:

We have to understand where people are getting their information from. If science is failing, then it’s because other channels are providing better entertainment and — maybe we touched on this earlier — the idea that scientists need to be where people are. I teach a class in science communication and journalism, and I ask them where they’re getting information from. This is sort of top-level undergraduate students or MSc students. And when I last polled the class, it was an interesting mix actually. They were saying from academic papers and YouTube. Academic papers, I think the scientists have got covered, but YouTube — that’s where that’s where they need to be.

Related: Michael Marshall on Compassionate Skepticism

YouTube Video

This week I'm taking a break from #MetaScience / #OpenScience and getting back into #SCIENCE!

I'm at the Rank Symposium on "21st Century Colour Vision" 👁️🧠🌈🤓

Got Colour Vision questions? AMA!
(If I don't know the answer I'll have lots of people to pass the question on to...)

#Rank #RankSymposium #ColourVision #ColorVision #OpenResearch

Is a 55% replication rate too low, too high, or just right? Some thoughts on Tyner et al.’s (2026) recent study.

#Metascience

https://markrubin.substack.com/p/is-a-55-replication-rate-too-low

New work and events...

🔸 Workshop on the politics and finances of open science reform

🔸 Symposium: “Who critiques the critique? Toward a reflexive metascience”

🔸 Preprint encourages establishing phenomena before testing theories

🔸 Systematic review of questionable research practices

https://markrubin.substack.com/p/critical-metascience-roundup-b87

#Metascience

🤔 Gleiche Daten führen zu gleichen Schlussfolgerungen?

Nicht unbedingt, wie eine neue Publikation in "Nature" zeigt, an der auch der Innsbrucker Ökonom Felix Holzmeister beteiligt war. Eine Re-Analyse von 100 Studien durch 457 Wissenschaftler:innen zeigte, dass die statistische Interpretation der Daten erheblich variierte. Die Autor:innen sprechen sich daher für Mehrfach-Analysen aus.

🆕 https://www.uibk.ac.at/de/newsroom/2026/meta-science-mehrfach-analysen-starken-wissenschaftlichkeit/

📖 https://www.nature.com/articles/s41586-025-09844-9

#Metascience #Metawissenschaft #Sozialwissenschaften

#metascience - Mehrfach-Analysen stärken Wissenschaftlichkeit
Eine Studie mit Beteiligung der @uniinnsbruck zeigt, dass in den Verhaltens- & #Sozialwissenschaften selbst bei identischer Datengrundlage deutlich voneinander abweichende Schlussfolgerungen entstehen können, wenn unabhängige Forschungsteams die Analyse durchführen.
https://idw-online.de/de/news868504
#MetaAnalysen #Wissenschaft #Forschung

Any bright ideas out there? 💡 Here's an "essay competition inviting active scientists from any sector to share concrete research challenges that you hypothesize are caused by broad structural or systemic bottlenecks in science, and experimental strategies to fix them": https://astera.org/essay-competition/. The first prize is 30,000$!

EDIT: Please read @ced's reply below. It turns out "active scientists" don't include economists, sociologists, psychologists, historians, and presumably anyone else doing social sciences and humanities research! 🤬

#academia #science #AcademicChatter #metascience

Critical Metascience Roundup

Update #2 highlights work on:

- Barriers and enablers of reproducibility

- Cataloguing open research practices in the humanities and social sciences

- Reconsidering reproducibility and questionable research practices in animal-based biomedicine

- Naming individual studies in metascience work

https://open.substack.com/pub/markrubin/p/critical-metascience-roundup

#Metascience

A quick report by me on running a randomisation process to distribute emergency research funds, now on the @RoRInstitute substack

https://researchonresearchinstitute.substack.com/p/bingo-partial-randomisation-for-emergency

#ResearchFunding #Metascience

RE: https://fosstodon.org/@researchsoft/116245545044086743

If you are interested in any aspect of #ResearchSoftware, including #RSEng, #funding, #publishing, #training, #metascience, #policy, etc., please consider submitting a proposal to and/or attending #IRSC26 (immediately before #RSECon26 in Sheffield, UK, in Sept). We're trying to bring all elements of the #ResearchSoftware community together

RE: https://flipboard.com/@wsj/business-b1985f4jz/-/a-T6_YAnIFSTSuzZdNlac-EA%3Aa%3A248213600-%2F0

“Instead of paying humans to join focus groups and complete surveys, #Auru uses thousands of #AI agents, or bots, to simulate human responses. It feeds #demographic and psychographic information into its models to create human profiles that match clients’ needs, and the results those bots spit out are being used for product #development, #pricing, identifying new customers and political #polling.”

Researchers have warned about the inaccuracies of treating #LLMs as human proxies (https://doi.org/10.1007/s10462-025-11297-5), but I wouldn’t be surprised #languageModels beat #qualitative interpretation of non-representative focus groups.

#marketing #psychometrics #statistics #quantMethods #philSci #metascience #business

Are there any examples where it is defensible to combine observational and randomised data in a metaanalysis? #stats #metascience

Research collaborations often start out vague, and participants may have different expectations about issues like workload, authorship, publication form, and open science. We are developing a toolkit where project initiators can state how they foresee the collaboration, and potential collaborators can evaluate if the project is right for them.

Please try it out and give feedback!
https://docs.google.com/document/d/19hi7Diro6NuE5QExR8YZ8eozCCLTSE5hhEh-xvc2g9M/edit

#BigTeamScience #MultiLab #MegaStudy #ResearchCollaboration #ResearchPlanning #MetaScience #OpenScience #AcademicChatter

Following up for this, I've written a longer piece for the new @RoRInstitute #Metascience substack

https://researchonresearchinstitute.substack.com/p/the-point-of-no-return

My minimal claim is that, contra Schweiger, it is very hard to definitely claim any scheme a waste of time and money, since the full costs and benefits of awarding funding in a particular way (e.g. by competition) are difficult to account for

New substack from @RoRInstitute

https://researchonresearchinstitute.substack.com

#MetaScience

Metascience seminar 1: Improve our inferences from observational studies

https://events.humanitix.com/metascience-seminar-1-harrison-hansford

The event is free for all.

Date and time: March 12 at 11AM (GMT + 11; Sydney, Melbourne, Canberra time)

Speaker: Harrison Hansford

#MetaScience #GMT+11 from @aimos

RE: https://mastodon.social/@MetaArXiv/116098607342372256

Yet another example of how so-called "#AI" entrenches existing problems in academic research.

#metaScience #metaresearch #academia #academicChatter #NoAI #maihT3k

Recently had to write a short bio about my Open Science work. I've also been harshly reminded (again) that many well-funded Open/Meta Science projects still expect unpaid expert contributions. I don't think we should be working for free. Do you?

Cont. on LinkedIn
https://www.linkedin.com/posts/alelazic_openscience-share-7430198728864587776-tBjq

#openscience #openscholarship #metascience

👀 Aishwarya Khanduja and Stuart Buck make the case that research funding could be improved if it took lessons from venture capital.

https://goodscience.substack.com/p/venture-capital-has-lessons-for-government

fwiw, the VC comparison makes very clear the distinction that what is good for research may not be good for researchers

#Metascience

📄 "To Be FAIR: Theory Specification Needs an Update"

New paper in Perspectives on Psychological Science: https://journals.sagepub.com/doi/10.1177/17456916251401850

Semantic versioning of theories? Citable theory objects that are reproducible and published independently of classical papers?

Van Lissa et al. (2025) argue that psychological theories need clearer and more systematic specification. The paper adapts the #FAIRprinciples (Findable, Accessible, Interoperable, Reusable) – well known from open data – to the way theories are defined, documented, and communicated.

1/2

#OpenScience #Theory #PsychScience #MetaScience

RE: https://fediscience.org/@UlrikeHahn/115887508756592856

Check out this paper on different conceptions of #reasoning and how it impacts the debate about whether #AI models can do it. 👇

#computerScience #psychology #philosophy #tech #publicDiscourse #academia #philSci #metascience #conceptualAnalysis #interdisciplinary

Wir vergeben eine Juniorprofessur (W1-equivalent) im Bereich Psychological Metascience (mit Tenure-Track bis W2 LBesG-Äquivalent)! 🎓 Mehr Infos: leibniz-psychology.onlyfy.jobs/job/10kku5n7 #Wissenschaft #Metascience #Karriere #ZPID

Junior professorship (W1-equiv...

https://retractionwatch.com/2025/11/21/springer-nature-flags-paper-with-fabricated-reference-to-article-not-written-by-our-cofounder/

This is very meta... essentially someone who is in the business of writing fake papers has written a fake paper criticizing Pubpeer and post publication peer review as being vulnerable to “misuse” and “hyper-skepticism.

#RetractionWatch #AcademicIntegrity #ResearchMisconduct #ScholarlyPublishing #PostPublicationReview #PubPeer #FakeScience #MetaScience #PeerReview #ScienceFraud

"younger colleagues ...look down on those who do not deposit research data.... 'if you say ‘data is available upon request,’ they take it as a #middleFinger. ...it would be so easy to share the data—it takes just as much energy to write that statement.'”

https://ir.library.illinoisstate.edu/fpml/268

#openScience #ethics #publishing #science #philosophyOfScience #metascience

Looking forward to speaking at the JISC community event next week (10th September, 11am, online) about implementing Electronic Lab Notebooks at the Cancer Research UK Manchester Institute

I'll be discussing our activities in:

📱In-lab hardware
💻Data management
🥼User feedback
🧪#ResearchIntegrity
📑 #Metascience with a UKRI grant
🍞 (possibly) addressing the important question of what self-service checkouts can teach us about #ELNs

Booking👇
https://research.jiscinvolve.org/wp/events/

#ELN #ERN #DigitalNotebooks

A #Metascience Study of the #LowCode #Scientific #Field 📊📊

We aimed to:
☑️ Analyze the composition and growth (e.g., size, diversity, venues, and topics) of the emerging #LowCode #community; and
☑️ Explore how these aspects differ from those of the “classical” #modeldriven community.

This covers #ResearchQuestions such as:

❓Are Low-Code #publications affecting the number of traditional #Modeling publications?
❓How many Modeling conferences and workshops aim to explicitly attract Low-Code publications?
❓How are authors distributed across Low-Code and traditional Modeling publications?
❓ How many Low-Code open-source platforms and tools explicitly relate to traditional Modeling techniques?

Lots of interesting insights 🤯🤯🤯

Read the post for the full details ⬇️⬇️⬇️ and/or meet me at the next #STAF #ECMFA conference in #Koblenz next month

https://modeling-languages.com/metascience-adoption-low-code/

A #Metascience Study of the #LowCode #Scientific #Field 📊📊

We aimed to:
☑️ Analyze the composition and growth (e.g., size, diversity, venues, and topics) of the emerging #LowCode #community; and
☑️ Explore how these aspects differ from those of the “classical” #modeldriven community.

This covers #ResearchQuestions such as:

❓Are Low-Code #publications affecting the number of traditional #Modeling publications?
❓How many Modeling conferences and workshops aim to explicitly attract Low-Code publications?
❓How are authors distributed across Low-Code and traditional Modeling publications?
❓ How many Low-Code open-source platforms and tools explicitly relate to traditional Modeling techniques?

Lots of interesting insights 🤯🤯🤯

Read the post for the full details ⬇️⬇️⬇️ and/or meet me at the next #STAF #ECMFA conference in #Koblenz next month

https://modeling-languages.com/metascience-adoption-low-code/

A #Metascience Study of the #LowCode #Scientific #Field 📊📊

We aimed to:
☑️ Analyze the composition and growth (e.g., size, diversity, venues, and topics) of the emerging #LowCode #community; and
☑️ Explore how these aspects differ from those of the “classical” #modeldriven community.

This covers #ResearchQuestions such as:

❓Are Low-Code #publications affecting the number of traditional #Modeling publications?
❓How many Modeling conferences and workshops aim to explicitly attract Low-Code publications?
❓How are authors distributed across Low-Code and traditional Modeling publications?
❓ How many Low-Code open-source platforms and tools explicitly relate to traditional Modeling techniques?

Lots of interesting insights 🤯🤯🤯

Read the post for the full details ⬇️⬇️⬇️ and/or meet me at the next #STAF #ECMFA conference in #Koblenz next month

https://modeling-languages.com/metascience-adoption-low-code/

A #Metascience Study of the #LowCode #Scientific #Field 📊📊

We aimed to:
☑️ Analyze the composition and growth (e.g., size, diversity, venues, and topics) of the emerging #LowCode #community; and
☑️ Explore how these aspects differ from those of the “classical” #modeldriven community.

This covers #ResearchQuestions such as:

❓Are Low-Code #publications affecting the number of traditional #Modeling publications?
❓How many Modeling conferences and workshops aim to explicitly attract Low-Code publications?
❓How are authors distributed across Low-Code and traditional Modeling publications?
❓ How many Low-Code open-source platforms and tools explicitly relate to traditional Modeling techniques?

Lots of interesting insights 🤯🤯🤯

Read the post for the full details ⬇️⬇️⬇️ and/or meet me at the next #STAF #ECMFA conference in #Koblenz next month

https://modeling-languages.com/metascience-adoption-low-code/

A #Metascience Study of the #LowCode #Scientific #Field 📊📊

We aimed to:
☑️ Analyze the composition and growth (e.g., size, diversity, venues, and topics) of the emerging #LowCode #community; and
☑️ Explore how these aspects differ from those of the “classical” #modeldriven community.

This covers #ResearchQuestions such as:

❓Are Low-Code #publications affecting the number of traditional #Modeling publications?
❓How many Modeling conferences and workshops aim to explicitly attract Low-Code publications?
❓How are authors distributed across Low-Code and traditional Modeling publications?
❓ How many Low-Code open-source platforms and tools explicitly relate to traditional Modeling techniques?

Lots of interesting insights 🤯🤯🤯

Read the post for the full details ⬇️⬇️⬇️ and/or meet me at the next #STAF #ECMFA conference in #Koblenz next month

https://modeling-languages.com/metascience-adoption-low-code/

Excited to announce that we are recruiting a Research Associate for this #UKRI-funded #metascience research grant investigating the roll-out of Electronic Research Notebooks (ERNs)

For full details of the role, and information on how to apply, please visit https://www.jobs.manchester.ac.uk/Job/JobDetail?JobId=31714

This is a great opportunity to enhance research practice, support #OpenResearch and #FAIR data, and have a positive impact on researchers in #Manchester and across the UK

#ELN #ELNs #DigitalNotebooks

Meta-science 2025 UCL June 30-July 2 Proposals due Feb 7 I am thinking of sending something. See you there? #hps #metaresearch #metascience metascience.info/call-for-pro...

Call for Proposals - Metascien...

Will this one simple trick get people to think more reflectively? As usual, no.

In two experiments in multiple countries, a brief scientific article describing the benefits of intuitive or reflective thinking didn't impact performance on tests of intuitive and reflective thinking.

https://doi.org/10.1111/sjop.12961

#cogSci #decisionScience #psychology #priming #nudge #rationality #metaScience

Is there any #metascience research on #altac research?

"Productive scientists are associated with lower disruption in scientific publishing" in PNAS

The flip side of the title is perhaps "Deliberate, thoughtful scientists are more impactful". Kinda like the idea of slow food.

#science #metascience #productivity #impact #disruption #goals #slowscience

https://www.pnas.org/doi/epdf/10.1073/pnas.2322462121

I’m part of the #EEGManyLabs project testing the #replicability of influential #EEG studies. We are using #PredictionMarkets as a tool in this effort and you are invited to take part, especially if you have some expertise in EEG research, no matter how little. See below for details.

You may well know about the success of “prediction markets” in forecasting the likelihood of replication (e.g., Dreber et al., PNAS 2015). We are delighted to announce that we have partnered with economists who led these seminal studies to test the wisdom of the EEG community.

From today (as we near the end of recruitment for this project - please see last calls below), we are opening a survey to ask you to vote on the likelihood of some hypotheses studied in the #EEGManyLabs project. Subsequently, you will be invited to bet on the likelihood of success through a stock market platform, where you will earn real money for you or a selected charity.

The success of this effort will become clear when we complete the full project in a few years time. But the results will immediately tell us about the degree of optimism/pessimism amongst our community.

So, please share this widely and place your bets now...

How can I sign up for the prediction markets? Registrations to participate in the prediction markets are administered via the sign-up form linked below. You must have experience of working with EEG (for example, through collecting and/or analysing EEG data, which may be evidenced by having published peer-reviewed articles or preprints with EEG or equivalent experience e.g. designing, collecting and analysing data from EEG experiments).

#neuroscience #psychology #replication #replicationcrisis #reproducibility #metascience

https://pavlovug-dot-yamm-track.appspot.com/2fDq4NW3JydtlOhg09G1yGVAVSU8W-Whs6kPML9_ZfizxufZcjwEmQPWP5AnyD8NUrSZsBDvk7fYcappAg1Sgo3_tXQFMEu2Cc91Q_7ZoUrnvmZiS20DZiHsSFcGXN3Cpu_h990jl35DhtizskaAYiFIXKDiVMD0arROUO4-wXStjr4hF_n39GH14z3uQM6NK9ioOU86MruRx

I’ll be presenting at the Big Team Science conference Karli Nave’s multi-lab EEG replication project about musical beat perception and steady-state evoked brain potentials (aka frequency tagging) next Monday. Hope to see you there if you're into #music, #openscience, #replications, etc.

Spoiler alert: we actually did replicate Nozaradan et al.'s basic effect of auditory imagery on EEG activity, but the observed effect size was quite a bit smaller than the original study, as tends to happen.

There are lots of other good looking presentations too of course. Registration is free or inexpensive with suggested donation and completely online.

https://bigteamscienceconference.github.io/

#teamscience #openscience #reproducibility #science #neuroscience #psychology #metascience #replicationcrisis

New blog post about my journey with the editors-in-chief of the Journal of Cheminformatics (@rguha and @[email protected]) to pilot a reproducibility review as part of submission.

https://cthoyt.com/2023/08/27/jcheminf-reproducibility-pilot.html

#cheminformatics #chemoinformatics #reproducibility #asapscience #chemistry #openscience #jcheminf #metascience

The bad news: research typically visualizes *either* statistical test uncertainty (e.g., confidence intervals) *or* data variability (e.g., the distribution of data), but medical professionals, data scientists, and tenure track faculty reliably misinterpreted those.

The good news: Visualizing *both* more or less eliminated experts’ misunderstanding of the visualization.

#OpenAccess in PNAS: https://doi.org/10.1073/pnas.2302491120

#decisionScience #dataViz #metascience #QuantMethods #PhilSci #Epistemology

#OpenScience #metascience #reactiontime #preprocessing

Exploratory hypothesis testing:

Nice to see that our article “Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests” (https://doi.org/10.1080/09515089.2022.2113771), which was published 4 months ago, is the most read article in @PhilosophicalPsychology in the last 12 months: https://www.tandfonline.com/action/showMostReadArticles?journalCode=cphp20

#Psychology
#PhilosophyOfScience
#PhilSci
#MetaScience
#Preregistration
#HypothesisTesting

#Introduction I'm an #InternalMedicine physician and #HealthPolicy researcher who studies how new medical technologies including drugs, devices, and vaccines are used to improve the quality of health care. #RxPolicy #RxEpi #PharmEpi #HealthServicesResearch

Other interests include #GIM, #PrimaryCare, #Geriatrics, #Deprescribing, #EBM, #Metaresearch, #Metascience, #ScienceOfScience, #STS, #HistSci, #HistSTM, #HistMed

Looking forward to connecting with friends old and new! #TwitterMigration