#psychmethods — Public Fediverse posts

How can researchers overcome #AcquiescenceBias?

In a #questionnaire, acquiescence is a tendency to agree with statements or answer affirmatively regardless of survey content.

Alvarado-Leiton et al. report simple ways to mitigate it.

https://doi.org/10.1093/jssam/smaf022

#PsychMethods #xPhi

Do professors with less PC views self-censor?

Clark et al. reported only the linear relationship: the less PC their view, the more reluctant professors were to share it. Kudos to Clark et al. for publishing their data so Luke could detect a better-fitting non-linear, non-unified explanation: most professors were not self-censoring; they were either uncertain or else unreluctant to share.

https://doi.org/10.31234/osf.io/ab34v

#edu #higherEd #psychMethods #logic #replicability #manyAnalysts #metaScience

Surprised this conclusion survived #peerReview: a "program succeeded in promoting positive attitudes and beliefs" about "#implicitBias #education ...among ...police" (N = 145).

The 1st survey was online, but the 2nd was in-person. And the 1st survey's questions weren't about the same trainings as the 2nd survey's.

So any differences in answers are as explainable by differences between the surveys as they are by one #education program.

https://doi.org/10.1080/09515089.2023.2296585

#psychMethods #logic #psychology

When people ask me how to estimate the sample size needed for their research question, my answers fall broadly into two buckets: power analysis and precision for planning analysis. But there seem to be other options as well.

What's your preferred method?
Preferred software? (Or software package?)

https://qr.ae/pKnFql

#Stats #QuantPsych #PsychMethods #R #TheNewStats

Are you more likely to fall for trick (reflection test) questions on a smartphone or PC?

Turned out it didn't make a difference unless you let people self-select which device they used — and even that difference was better explained by gender and self-reported intuitive decision style.

https://doi.org/10.1080/07421222.2023.2196769

#decisionScience #cogSci #PsychMethods #UX #tech

Remember that "...WEIRDest people in the world" paper?

Now #xPhi has one: Of "171 experimental philosophy studies [from] 2017 [to] 2023 [including one of mine] most ...tested only Western populations but generalized beyond them without justification."

Incentives may be part of the issue: "studies with broader conclusions ...had higher citation impact."

https://doi.org/10.1017/psa.2023.109

#xPhi #PsychMethods #Culture #Demography #PhilSci

"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

#Civicbase brings upvoting and downvoting to preference measurement—but with a budget.

Participants can select or agree or disagree buttons (up to 7 times) to allocate a limited voting credits (that carry over to future studies?).

May reveal priorities that Likert scales and ranked-choices cannot.

https://doi.org/10.1002/aaai.12103

Presumably, this could be used for all sorts of preferences (beyond civics/politics).

#measurement #PsychMethods #openSource #decisionScience #poliSci #cogSci #gamification

How do we know what participants thought when we presented our stimuli?

#ProcessTracing can reveal what people saw (e.g., eye-tracking), consciously thought (e.g., concurrent think-aloud), etc.

Combining those two methods revealed:
(1) thinking aloud didn't impact gaze or word count
(2) retrospective think-aloud left out thoughts that were mentioned concurrently
(3) retrospective think-aloud introduced thoughts unmentioned concurrently

https://doi.org/10.1007/978-3-319-14956-1_5

#PsychMethods #CogSci #xPhi

How do we know what participants thought when we presented our stimuli?

#ProcessTracing can reveal what people saw (e.g., eye-tracking), consciously thought (e.g., concurrent think-aloud), etc.

Combining those two methods revealed:
(1) thinking aloud didn't impact gaze or word count
(2) retrospective think-aloud left out thoughts that were mentioned concurrently
(3) retrospective think-aloud introduced thoughts unmentioned concurrently

https://doi.org/10.1007/978-3-319-14956-1_5

#PsychMethods #CogSci #xPhi

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

Critical Metascience:

2022 has been a bumper year for what I’d call “critical metascience” - work that takes a step back and offers a critical perspective in the field.

My Top 10 papers of 2022 in this area are, in alphabetical order… 🥁 🧵👉

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#SociologyofScience
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon

1/12

Replicability and Theory:

“Our results suggest that many of the practices that have been proposed as a means to improve the replicability of psychological research—such as open data and methods…preregistration and Registered Reports…and basing conclusions on Bayesian inference…or p < .005 rather than p < .05…—do indeed improve confidence in replicability among our sample.”

Continued 🙂 🧵👉

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

@MarkRubin This is massively simplistic. Hypotheses include the criteria for delineating phenomena in need of explanation, satisfaction criteria for success, disciplinary standards and practices, and taxonomies of subjects under investigation. IMO.
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

What’s a hypothesis?

“A hypothesis is not simply a guess about the result of an experiment. It is a proposed explanation that can predict the outcome of an experiment. A hypothesis has two components: (1) an explanation and (2) a prediction. A prediction simply isn’t useful on its own.” (Haroz, 2014)

Blog post: http://steveharoz.com/blog/2014/mysterious-origins-of-hypotheses-in-visualization-and-chi/

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

A “quietist” response to the replication crisis:

“The quietist approach proposes that we should just accept that it is in the nature of science that we get things wrong, and that this is particularly true with sciences in early stages of development.”

Bird (2021). Understanding the replication crisis as a base rate fallacy.

🔒 https://doi.org/10.1093/bjps/axy051

🔓 https://kclpure.kcl.ac.uk/portal/files/95454096/Replication_base_rate_fallacy_REV2.pdf

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
@philosophy

Bad Stats / Poor Methods:

Qualitative study finds 39.8% of 548 psychology researchers believe that statistics and/or research methods are misused and/or misunderstood in the field.

Miranda et al. (May 2022). How do researchers in psychology perceive the field? A qualitative exploration of critiques and defenses. Collabra: Psychology.

https://doi.org/10.1525/collabra.35711

#Psychology
#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

No evidence of p-hacking in imaging research:

Analysis of 4,105 randomly sampled p-values finds no evidence of p-hacking in work published in over 100 imaging journals since 1972.

Rooprai et al. (2022): https://doi.org/10.1177/0846537122113941

#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

Looks like a great talk from Stephan Guttinger on Questionable Research Practices

“What should be abandoned is not the idea of questioning practice, but the idea that there is a class of questionable research practices.”

Slides: https://philstatwars.files.wordpress.com/2022/12/guttinger_draft_red.pdf

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#QRPs

“We are not only in a replication but an interpretation crisis, a crisis of theory building.”

Benjamin Krämer (@benjkraemer) (2022, November). Why are most published research findings under-theorized? In Questions of Communicative Change and Continuity.

🔓 https://www.nomos-elibrary.de/10.5771/9783748928232-23.pdf

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon
#Communication

“The appropriate conclusion based on significant social-psychology experimental findings could perhaps be characterized as ‘sometimes this happens.’….Although ‘sometimes this happens’ may be disappointing compared with establishing universal laws, perhaps the field should accept this with both humility and pride.”

Baumeister et al. (2022). https://doi.org/10.1177/17456916221121815

#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilSci
#SocialPsychology
@socialpsych

Replicating experiments:

New preprint considers the “minimum viable experiment to replicate.”

“In this paper, we introduce the idea of a minimum viable experiment that needs to be identified in practice for replication results to be clearly interpretable.”

Devezer & Buzbas. (2022). Preprint: http://philsci-archive.pitt.edu/21475/

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
@philosophy

On the questionable use of “Questionable Research Practices” (QRPs):

“Practices labelled as ‘QRPs’ can be both beneficial and problematic for research practice and targeting them without a sound understanding of their dynamic and context-dependent nature risks creating unnecessary casualties in the fight for a more reliable scientific practice.”

Stephan Guttinger. Talk to be presented on 1st Dec: https://phil-stat-wars.com/3-tentative-schedule/

#Statistics
#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis

New evidence that Bayes factors are misused in applied psychology:

“The way forward is not to ban Bayesian inference from our toolbox. Instead, more and better education on Bayesian inference is needed.”

Preprint: https://psyarxiv.com/du3fc/

#Psychology
#Stats
#Statistics
#PsychMethods
#Bayes

Outliers:

Should you identify them within groups or across groups, and what do you do with them when you find them?

Preprint (forthcoming in JEP: General): https://psyarxiv.com/47ezg/

Few quotes follow 🧵👉

#stats
#statistics
#psychmethods

🔹 "No field can grow without healthy critique" 🔹

(Berkeley Initiative for Transparency in the Social Sciences: https://twitter.com/UCBITSS/status/1557834242763079680)

Check out this list of over 80 articles that have a somewhat critical view on one or more issues associated with open science, science reform, and/or the replication crisis: https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-articles-critical-of-open-science

Includes a ResearchRabbit collection: https://www.researchrabbitapp.com/collection/public/V467G42MLX

#openscience
#metascience
#psychmethods
#replicationcrisis

Replication crisis:

“Don't get me wrong, I think there are real reasons to be concerned....But I'm not sure calling it a crisis is particularly helpful because I think that can just be a bit distracting.”

Interview with Marcus Munafo, chair of the UK Reproducibility Network

https://www.thenakedscientists.com/articles/interviews/are-reproducibility-rates-concern

#science
#openscience
#metascience
#psychmethods
#replicationcrisis

Careful! “The multiverse is a dangerous place…[and] it’s easy to generate data-analytic ‘black-holes’!”

Excellent talk today by Marco Del Giudice at Durham University's RIOTS Club.

See also Del Giudice and Gangestead (2021): https://doi.org/10.1177/2515245920954925

And https://fediscience.org/@MarkRubin/109342440756327400

#openscience
#metascience
#psychmethods
#statistics

@kaitclark thanks for doing this! I’d love to be added! #socialpsych #personalitypsych #openscience #psychmethods #metascience

@kaitclark @rdhawkins thank you! Please add: #socialpsychology #personality #sciencereform #openscience #psychmethods

So cool seeing everyone migrating over or at least giving this prehistoric animal app a try.

Someone should really start a #psychology, #psychscience, or #psychmethods server so we can all reconnect (and I can leech knowledge from all of you lol). Honestly, I would do it but I don’t know where to start and $$$

So cool seeing everyone migrating over or at least giving this prehistoric animal app a try.

Someone should really start a #psychology, #psychscience, or #psychmethods server so we can all reconnect (and I can leech knowledge from all of you lol). Honestly, I would do it but I don’t know where to start and $$$