#surveymethods — Public Fediverse posts

Can researchers detect #AI bots taking paid surveys?

#Prolific tested humans and #LLM agents with various #dataQuality checks.
- The company says they caught 100% of the non-humans.
- My take-away: #reCAPTCHA and #mouseTracking caught 95%

https://www.prolific.com/resources/authenticity-checks-how-we-tested-the-most-accurate-method-for-identifying-agentic-ai

#surveyMethods

Hey #SurveyMethods and #MedEd folks:

In a workshop for #MedSchool faculty about questionnaire design and survey research methods, what

- objectives should be prioritized?
- materials are out there?
- activities are worth it?
- assessments work?

Example: https://www.aamc.org/what-we-do/mission-areas/medical-education/meded-research-certificate-program/workshop-descriptions

We've found recruiting people for online #research via #onlineAdvertising yielded good results on overt and covert #dataQuality measures (perhaps because participation incentives aren't financial):

Attention checks passed ≅ 2.6 out of 3

ReCAPTCHA (v3) ≅ 0.94 out of 1.0

Sample size > 5000 (from six continents)

https://doi.org/10.1017/S0034412525000198

#surveyMethods #cogSci #psychology #xPhi #QualityControl #econ #marketing

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

Excited for 2025 presentations in #NYC at the #APA (January 8 to 11): apaonline.org/mpage/2025eastern?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

The talk: https://researchgate.net/publication/370132037?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts
The poster: https://researchgate.net/publication/371248872?utm_source=mastodon&utm_medium=social&utm_campaign=fedica-Text-posts

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

The #ASA / #NSF / #BLS Senior Research Fellowship Program is searching for two fellows. One position is focused on #statistics, #economics, or #BehavioralScience research, and the other is focused on #DataScience. Applicants must work at a college/university or a 501(c)(3) #nonprofit.

Applications are due January 9, 2023. See the full details at https://www.bls.gov/osmr/asa_nsf_bls_fellowship_info.htm

#SurveyResearch #SocialScience #SurveyMethods #surveys #sampling #fellowship #QuantMethods