#surveymethods — Public Fediverse posts

Can researchers detect #AI bots taking paid surveys?

#Prolific tested humans and #LLM agents with various #dataQuality checks.
- The company says they caught 100% of the non-humans.
- My take-away: #reCAPTCHA and #mouseTracking caught 95%

https://www.prolific.com/resources/authenticity-checks-how-we-tested-the-most-accurate-method-for-identifying-agentic-ai

#surveyMethods

Hey #SurveyMethods and #MedEd folks:

In a workshop for #MedSchool faculty about questionnaire design and survey research methods, what

- objectives should be prioritized?
- materials are out there?
- activities are worth it?
- assessments work?

Example: https://www.aamc.org/what-we-do/mission-areas/medical-education/meded-research-certificate-program/workshop-descriptions

We've found recruiting people for online #research via #onlineAdvertising yielded good results on overt and covert #dataQuality measures (perhaps because participation incentives aren't financial):

Attention checks passed ≅ 2.6 out of 3

ReCAPTCHA (v3) ≅ 0.94 out of 1.0

Sample size > 5000 (from six continents)

https://doi.org/10.1017/S0034412525000198

#surveyMethods #cogSci #psychology #xPhi #QualityControl #econ #marketing

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

RE: https://mastodon.acm.org/@neilernst/115607469843033537

😳 The #AI survey taker "rendered [attention quality] checks [ACQs] effectively obsolete. Across 6,000 total trials..., [it] committed only 10 errors, achieving an overall pass rate of 99.8% and scoring perfectly on 18 of the 20 ACQ types."

#surveyMethods #psychometrics #psychology #tech #psychology #philSci #SciComm #dataQuality

Can thinking aloud accurately capture how we decide?

Nisbet & Wilson's 1977 paper famously suggested it can't.

But #NLP and #AI methods may indicate that it can:
- https://escholarship.org/uc/item/4sb936m3
- https://openreview.net/forum?id=1Tny4KgGO2

#cogSci #psychology #philosophy #surveyMethods #thinkAloud #LLM

#ComparativeResearch #EducationMeasurement #SurveyMethods
Measuring education across countries is complex—but crucial for valid, comparable survey data. This study tests 16 coding strategies using ESS data and finds a strong contender for a new international standard.

Read the article:
Schneider S.L. & Urban J. (2025). A myriad of options: Validity and comparability of alternative international education variables. Survey Methods: Insights from the Field: 10.13094/SMIF-2025-00008

@GESIS
Thanks for sharing the talk!
Here is the Question-Link R-Package and an in depth tutorial on using it:
https://matroth.github.io/questionlink/index.html

And please note that we also offer consultations regarding harmonization techniques (and other survey method topics).
https://www.gesis.org/en/consulting/survey-methods-consulting

#surveymethods #harmonization #rstats

Given that surveys tend to overestimate belief in #conspiracyTheories (https://osf.io/preprints/psyarxiv/zsncr_v1) and support for #politicalViolence (https://doi.org/10.1073/pnas.2116870119), I wonder how much of the correlation between such variables remains after accounting for such measurement error.

#stats #psychometrics #surveyMethods

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

Matthias Roth (@rothm) and I published a new paper on survey data harmonization.

"One harmonization fits all? – Impact of missing population invariance on harmonization error when harmonizing social science survey questions with equating"

The paper again corroborates that observed score equating outperforms linear stretching on average. However, it also shows that it is better to derive a harmonization solution for single-item measures from a sample that is drawn from a similar population to the population you are interested in researching.

https://doi.org/10.1080/13645579.2025.2481112

#surveyresearch #surveymethods #harmonization

Thankful to get a copy of Reflection and Intuition in A Crisis-Ridden World: https://doi.org/10.4324/978100330036

Hoping for a digital copy to annotate!

Can't access the #book? Some free papers cover similar work:
- https://doi.org/10.1111/meta.12534
- https://doi.org/10.3390/jintelligence11040076
- https://doi.org/10.31234/osf.io/y8sdm

I've only been able to glance at a few sections, but this seems like a book I would want to write:
- broad range of topics that interest more than academics
- situated in the history of ideas
- attention to replicability of results
- anticipation of future directions

Bravo! 👏

#cogSci #psychology #politics #epistemology #ethics #extremism #conspiracyTheory #rationality #logic #surveyMethods #psychometrics

🗓️ Next week on Fri 6 Dec

Explore the challenges and opportunities for #socialsurvey #data collection in #Scotland.

Register online for this free in-person event #SocialResearch #SurveyMethods #ResearchMethods #DataCollection

https://www.eventbrite.co.uk/e/survey-futures-practice-forum-4-tickets-1046687960867

Excited for 2025 presentations in #NYC at the #APA (January 8 to 11): apaonline.org/mpage/2025eastern?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

The talk: https://researchgate.net/publication/370132037?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts
The poster: https://researchgate.net/publication/371248872?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

In a new paper, I simulate the consequences of combining survey data without proper harmonization techniques.

I demonstrate that there is a plausible risk of biased correlative analyses based on the integrated data, if we do not harmonize measurement units across different survey sources and instruments first.

#surveyresearch #surveymethods #harmonization

https://doi.org/10.5964/miss.11217

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Moral comparisons of utilitarian tradeoffs depended on the rating protocol?

Participants rated pairs of utilitarian tradeoffs. Relative differences for each pair depended on
- whether participants saw both tradeoffs at the same time or separately.
- whether the rating was comparative or quantitative.

Sometimes (although not most of the time), the average relative difference for one protocol reversed in the other protocol!

https://doi.org/10.1016/j.cognition.2024.105919

#SurveyMethods #xPhi #ethics #moralPsychology

Will people reason less reflectively when primed to think about threats to their health or resources?

Multiple pre-registered experiments (N > 3000) didn't detect any such reflection-suppressing effect of threat primes (compared to controls) — and this didn't seem to be a result of a failed manipulation.

https://doi.org/10.3758/s13428-024-02481-6

#decisionScience #surveyMethods #medicine #health #economics #psychology #edu #policy

If cognitive reflection tests measure a reasoning style that generalizes beyond the test itself, we should see test performance predicting many forms of good reasoning.

Yet another example of this: better reflection test performance predicts more (logically) consistent responses across opposing psychological test items (that is, reversed and non-reversed items on a scale): https://doi.org/10.1016/j.paid.2024.112811

#psychometrics #decisionScience #edu #assessment #psychology #surveyMethods

I recently found think-aloud research from 15 years ago, which may have revealed that expert entrepreneurs thought more reflectively than novices (MBA students).

I also like that this paper seems to relay some of the reasons that concurrent think-aloud protocols can outperform retrospective verbal report protocols.

http://doi.org/10.1016/j.jbusvent.2008.02.002

#business #decisionScience #economics #epistemology #dualProcessTheory #surveyMethods #ThinkAloudProtocol

People designing questionnaires for a German sample will find the just published new version of the "Demographische Standards" helpful. It offers guidance on how to collect varios socio-demographic variables (e.g., education, citizenship, employment) in a comprehensive manner.

https://doi.org/10.21241/ssoar.94099

#surveymethods #surveyresearch

The #ASA / #NSF / #BLS Senior Research Fellowship Program is searching for two fellows. One position is focused on #statistics, #economics, or #BehavioralScience research, and the other is focused on #DataScience. Applicants must work at a college/university or a 501(c)(3) #nonprofit.

Applications are due January 9, 2023. See the full details at https://www.bls.gov/osmr/asa_nsf_bls_fellowship_info.htm

#SurveyResearch #SocialScience #SurveyMethods #surveys #sampling #fellowship #QuantMethods

"We conclude that potential utility of NPSs [nonprobability samples from commercial survey organizations] will remain out of reach unless scholars confront the tension between the operation of online survey organizations and the goals of transparent research." https://academic.oup.com/poq/article-abstract/87/3/816/7275175 #surveymethodology #surveymethods #nonprob @surveystatisticsandmethodology @surveystatsandmethodology

Interesting review paper on applied survey methods in Annual Reviews of Economics https://www.annualreviews.org/doi/pdf/10.1146/annurev-economics-091622-010157 @surveystatisticsandmethodology @surveystatsandmethodology #surveymethodology #surveymethods

SHARE Nutzerworkshop & Survey Lifecycle Operators Workshop (SLOW) vom 27. bis 29. September http://blog.surveydata.org/share-nutzerworkshop-survey-lifecycle-operators-workshop-slow-vom-27-bis-29-september/ @surveystatisticsandmethodology @surveystatsandmethodology #surveymethods

Very excited about the two great sessions on data donation with many amazing presentations at #esra23. You can find the abstracts here:
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=54#main
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=178#main
#datadonation #surveymethods #computationalsocialscience #digitaltracedata

Interesting theory-based paper on participation on prob-based online panels https://academic.oup.com/ijpor/article/35/2/edad012/7143700 @surveystatisticsandmethodology @surveystatsandmethodology #survey #surveymethods #surveymethodology

Nice, very hands-on paper on how to run surveys in market-research panels "Using Market-Research Panels for Behavioral Science: An Overview and Tutorial" https://journals.sagepub.com/doi/full/10.1177/25152459221140388 @surveystatisticsandmethodology #surveymethods #survey #surveymethodology

Interested in a bit of #WeekendListening around surveys, polling, and opinions?

If you have access to the #BBC's material, how about listen first to this episode from "The Forum",
From Straw Poll to Opinion Poll
https://www.bbc.co.uk/sounds/play/w3cswprr

And then, for a bit of comic relief, to this from "Stand-up Specials"
Geoff Norcott: #Geofferendum
https://www.bbc.co.uk/sounds/play/m001cx2s

#SurveyMethods

Julia Rohrer posted about the pron-nonprob-debate.
https://www.the100.ci/2023/03/07/non-representative-samples-what-could-possibly-go-wrong/
This symbolic depiction of Missingness Completely at Random makes me faint.
#surveymethods #surveymethodology

My paper "Harmonizing single-question instruments for latent constructs with equating using political interest as an example" was just published in Survey Research Methods: https://doi.org/10.18148/srm/2022.v16i3.7916
🎉
#surveymethods #harmonization

Very excited about the two great sessions on data donation with many amazing presentations at #esra23. You can find the abstracts here:
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=54#main
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=178#main
#datadonation #surveymethods #computationalsocialscience #digitaltracedata

Very excited about the two great sessions on data donation with many amazing presentations at #esra23. You can find the abstracts here:
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=54#main
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=178#main
#datadonation #surveymethods #computationalsocialscience #digitaltracedata

Very excited about the two great sessions on data donation with many amazing presentations at #esra23. You can find the abstracts here:
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=54#main
https://www.europeansurveyresearch.org/conf2023/prog.php?sess=178#main
#datadonation #surveymethods #computationalsocialscience #digitaltracedata