#mturk — Public Fediverse posts

Is #Amazon going to sunset #mTurk?

As of July 30 2026, #AWS will no longer accept new customers (requestors).

"Existing customers can continue to use the service as normal."

https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-public.html

#cogSci #psychology #xPhi #AI #tech #news #crowdWork #gigWork #surveyResearch

Is #Amazon going to sunset #mTurk?

As of July 30 2026, #AWS will no longer accept new customers (requestors).

"Existing customers can continue to use the service as normal."

https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-public.html

#cogSci #psychology #xPhi #AI #tech #news #crowdWork #gigWork #surveyResearch

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

I forgot to share the #mTurk data quality result that got scooped:

“In late 2020…. Participants from the United States were recruited from Amazon Mechanical Turk, -#CloudResearch, #Prolific, and a #university. One participant source yielded up to 18 times as many low-quality respondents as the other three.”

https://doi.org/10.1093/analys/anaf015

#psychology #philosophy #surveyMethods #quantMethods #dataScience #qualityControl

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

🎉 In press: Reflection-#Philosophy Order Effects & Correlations Across Samples

Thought experiments caused reflective decisions?
Reflection tests didn't change philosophical decisions!
18X more junk data from #mTurk (vs other sources)!

#Preprint + audio: https://byrdnick.com/archives/28438/upon-reflection-ep-13-reflection-philosophy-order-effects-and-correlations-across-samples

🎉 In press: Reflection-#Philosophy Order Effects & Correlations Across Samples

Thought experiments caused reflective decisions?
Reflection tests didn't change philosophical decisions!
18X more junk data from #mTurk (vs other sources)!

#Preprint + audio: https://byrdnick.com/archives/28438/upon-reflection-ep-13-reflection-philosophy-order-effects-and-correlations-across-samples

"Without crowdsourcing, manually reviewing and labeling thousands of images like this would take days or weeks. With #MTurk you can get it done in a few hours."
https://blog.mturk.com/tutorial-how-to-label-thousands-of-images-using-the-crowd-bea164ccbefc

Regardez cette page, ça ressemble à la promotion d'un service informatique comme un autre. Mais contrairement aux autres, c'est du nouvel esclavagisme numérique.

C'est pas nouveau dans l'absolu mais je trouve ça fascinant de cynisme comment une personne lambda peux déclencher un ordre d'exploitation néo coloniale avec un simple clic, tout ça pour tagger ses photos de vacances ou pour des libristes qui voudraient développer des modèles de #llm open source *ha ha je ris jaune*

"Without crowdsourcing, manually reviewing and labeling thousands of images like this would take days or weeks. With #MTurk you can get it done in a few hours."
https://blog.mturk.com/tutorial-how-to-label-thousands-of-images-using-the-crowd-bea164ccbefc

Regardez cette page, ça ressemble à la promotion d'un service informatique comme un autre. Mais contrairement aux autres, c'est du nouvel esclavagisme numérique.

C'est pas nouveau dans l'absolu mais je trouve ça fascinant de cynisme comment une personne lambda peux déclencher un ordre d'exploitation néo coloniale avec un simple clic, tout ça pour tagger ses photos de vacances ou pour des libristes qui voudraient développer des modèles de #llm open source *ha ha je ris jaune*

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

#decisionScience #philosophy #xPhi #epistemology #bioethics #cognitiveScience #mTurk #Prolific #UniversityParticipants #surveyMethods #DualProcessTheory

2️⃣ Ethical Dilemmas in Conducting Human Subjects Research on Amazon Mechanical Turk (MTurk)

https://ssrn.com/abstract=4419210

#MTurk #MechanicalTurk #Dehumanization #techworker #techworkerrights #amazonmechanicalturk #clickworker #AIEthics #aws #amazon

2️⃣ Ethical Dilemmas in Conducting Human Subjects Research on Amazon Mechanical Turk (MTurk)

https://ssrn.com/abstract=4419210

#MTurk #MechanicalTurk #Dehumanization #techworker #techworkerrights #amazonmechanicalturk #clickworker #AIEthics #aws #amazon

Are moral decisions impacted by economic environment?

Perhaps! “…in low-income nations, tournament-based compensation increased deontological commitments [but] in higher-income nations, the effect on deontological commitments reversed …consistent with the historical development of the doux commerce thesis.”

A fun #economics paper that reminds me of the early days of #xPhi: https://doi.org/10.1093/jleo/ewae016

#philosophy #psychology #policy #history #ethics #development #decisionScience #mTurk

Are moral decisions impacted by economic environment?

Perhaps! “…in low-income nations, tournament-based compensation increased deontological commitments [but] in higher-income nations, the effect on deontological commitments reversed …consistent with the historical development of the doux commerce thesis.”

A fun #economics paper that reminds me of the early days of #xPhi: https://doi.org/10.1093/jleo/ewae016

#philosophy #psychology #policy #history #ethics #development #decisionScience #mTurk

Could it be that letting respondents choose their own rating scale might reduce the occurrence of #ResponseStyles?

https://link.springer.com/article/10.3758/s13428-024-02429-w

#Psychometrics #MTurk

"The performance of the workers who animate the artifice is obscured by the spectacle of the machine" -- Ayhan Aytes

#Fauxtomation #MTurk

"The performance of the workers who animate the artifice is obscured by the spectacle of the machine" -- Ayhan Aytes

#Fauxtomation #MTurk

A question for the #HCI folks (please repost for more reach):

Since the #LLM cat is definitely out of the bag now, do you feel that #Prolific, #MTurk, etc. are still a reliable source of survey participants and consequently, survey data?

A question for the #HCI folks (please repost for more reach):

Since the #LLM cat is definitely out of the bag now, do you feel that #Prolific, #MTurk, etc. are still a reliable source of survey participants and consequently, survey data?

“Digital Sweatshops’ MTurk is a “crowdsourcing marketplace” where tech companies can access “a global, on-demand, 24x7 workforce” at “significantly lower costs” by taking advantage of the platform’s “pay-per-task model.”

https://briarpatchmagazine.com/articles/view/the-workers-ai-hides

#ai #MTurk #exploitation #Capitalism always keen to “reduce the cost of labour”
#HITs Human Intelligence Tasks

Found out that my #Mturk Worker and Requester accounts are suspended (after 5+ years). My Phd student's application got immediate rejected. Other Mturk workers report similar (link). What's going on Amazon ?

https://www.reddit.com/r/mturk/comments/173v3qt/mturk_account_suspended_with_no_reason/

Found out that my #Mturk Worker and Requester accounts are suspended (after 5+ years). My Phd student's application got immediate rejected. Other Mturk workers report similar (link). What's going on Amazon ?

https://www.reddit.com/r/mturk/comments/173v3qt/mturk_account_suspended_with_no_reason/

What's the difference between microtask platforms like #mTurk or Prolific and market-research panels?

CloudResearch recently published an #openAccess paper that explains. (The paper also includes a step-by-step tutorial for using their "Prime Panels" in conjunction Qualtrics). 

https://doi.org/10.1177/25152459221140388

#humanSubjectsResearch #researchMethods #surveyMethodology #Psychology #CogSci #xPhi

What's the difference between microtask platforms like #mTurk or Prolific and market-research panels?

CloudResearch recently published an #openAccess paper that explains. (The paper also includes a step-by-step tutorial for using their "Prime Panels" in conjunction Qualtrics). 

https://doi.org/10.1177/25152459221140388

#humanSubjectsResearch #researchMethods #surveyMethodology #Psychology #CogSci #xPhi

A side question, given the high accuracy of classification of human generated vs synthetic text in this paper: if mechanical turk workers are mostly from certain countries, could the authors be training their model to recognise the style of English writing typical of those countries as "human"? Not that it would invalidate their results, but it could impact followup studies. #LLM #MechanicalTurk #ChatGPT #MTurk

A side question, given the high accuracy of classification of human generated vs synthetic text in this paper: if mechanical turk workers are mostly from certain countries, could the authors be training their model to recognise the style of English writing typical of those countries as "human"? Not that it would invalidate their results, but it could impact followup studies. #LLM #MechanicalTurk #ChatGPT #MTurk

@skry @mosheroperandi

The description of mturk:

'Amazon Mechanical Turk is a crowdsourcing website with which businesses can hire remotely located "crowdworkers" to perform discrete on-demand tasks that computers are currently unable to do as economically.'

LOL😂

I used to be a #mTurk crowdworker. Back when I first had to quit work and became totally homebound several years ago due to disability, every 10-cent job mattered

At the time, goal was to earn cable & internet bill via mTurk jobs

@skry @mosheroperandi

The description of mturk:

'Amazon Mechanical Turk is a crowdsourcing website with which businesses can hire remotely located "crowdworkers" to perform discrete on-demand tasks that computers are currently unable to do as economically.'

LOL😂

I used to be a #mTurk crowdworker. Back when I first had to quit work and became totally homebound several years ago due to disability, every 10-cent job mattered

At the time, goal was to earn cable & internet bill via mTurk jobs

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

with Meysam Alizadeh & Maël Kubli

We find that zero-shot #ChatGPT:
> has better accuracy than #MTurk
> has better intercoder agreement than MTurk and trained coders
> is 20x cheaper than MTurk

https://arxiv.org/abs/2303.15056

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

with Meysam Alizadeh & Maël Kubli

We find that zero-shot #ChatGPT:
> has better accuracy than #MTurk
> has better intercoder agreement than MTurk and trained coders
> is 20x cheaper than MTurk

https://arxiv.org/abs/2303.15056

RT @turkopticon

It's time to recognize the challenges we face, take into account the autonomy & well-being of those working on #MTurk & create solutions that ensure the fairest treatment of Turkers worldwide.

Next Wednesday @ #Mozfest
:undefined:https://mozillafestival.org/en/tickets
:undefined:https://schedule.mozillafestival.org/session/U7L8W8

RT @turkopticon

It's time to recognize the challenges we face, take into account the autonomy & well-being of those working on #MTurk & create solutions that ensure the fairest treatment of Turkers worldwide.

Next Wednesday @ #Mozfest
:undefined:https://mozillafestival.org/en/tickets
:undefined:https://schedule.mozillafestival.org/session/U7L8W8

Next was an engaging talk by @KassemSarrah on #work and alienation in the platform economy at the @oiioxford. This is a multi-faceted analysis of the fulfillment center and #mturk #workers at #Amazon, and while there's an explicit agenda to the analysis I found the distinction between place-based and virtual organizing compelling https://www.youtube.com/watch?v=M75-cRiY6rU (3/4)

Next was an engaging talk by @KassemSarrah on #work and alienation in the platform economy at the @oiioxford. This is a multi-faceted analysis of the fulfillment center and #mturk #workers at #Amazon, and while there's an explicit agenda to the analysis I found the distinction between place-based and virtual organizing compelling https://www.youtube.com/watch?v=M75-cRiY6rU (3/4)

RT @turkopticon

Are you curious about the challenges of virtual global organizing in the #GigEconomy? Our session at #Mozfest on "Uniting the Amazon #MTurk Workers Behind AI" is in less than 3 weeks!

Don't miss it!
https://mozillafestival.org/en/tickets
https://schedule.mozillafestival.org/session/U7L8W8

RT @turkopticon

Are you curious about the challenges of virtual global organizing in the #GigEconomy? Our session at #Mozfest on "Uniting the Amazon #MTurk Workers Behind AI" is in less than 3 weeks!

Don't miss it!
https://mozillafestival.org/en/tickets
https://schedule.mozillafestival.org/session/U7L8W8

What conclusion do you draw, if any, when all (and only) the people who wrote in "non-binary" to the question, "What is your gender?" failed a survey's attention check?

Why would someone who puts in the effort to write a (vs. select a pre-written) response have been inattentive earlier in the survey?

I doubt that people reporting non-binary gender are generally less attentive on surveys.

So was it most likely a fluke?

#mTurk #CloudResearch #researchMethods #demographics #dataQuality

What conclusion do you draw, if any, when all (and only) the people who wrote in "non-binary" to the question, "What is your gender?" failed a survey's attention check?

Why would someone who puts in the effort to write a (vs. select a pre-written) response have been inattentive earlier in the survey?

I doubt that people reporting non-binary gender are generally less attentive on surveys.

So was it most likely a fluke?

#mTurk #CloudResearch #researchMethods #demographics #dataQuality

mTurk HITs completed

Can you get them automatically added to a dataset when running a survey?

Sort of like you can automatically add Worker ID, Assignment ID, and HIT ID to the dataset with certain URL parameters.

#psychology #survey #research #mTurk #dataQuality

mTurk HITs completed

Can you get them automatically added to a dataset when running a survey?

Sort of like you can automatically add Worker ID, Assignment ID, and HIT ID to the dataset with certain URL parameters.

#psychology #survey #research #mTurk #dataQuality

@atriverside This new preprint doesn’t “refute” the earlier paper, neither did that one “trash” #MTurk. This new commentary points out that the claim by the earlier paper that only a handful of participants were human is misleading, which is true: many were discarded not for obvious cues that they were automated scripts harvesting easy money, but rather, because even when human, they failed the selection criteria. The point that many psychology papers based on #MTurk are likely poor remains.

@atriverside This new preprint doesn’t “refute” the earlier paper, neither did that one “trash” #MTurk. This new commentary points out that the claim by the earlier paper that only a handful of participants were human is misleading, which is true: many were discarded not for obvious cues that they were automated scripts harvesting easy money, but rather, because even when human, they failed the selection criteria. The point that many psychology papers based on #MTurk are likely poor remains.

I deeply appreciate that Cloud Research keeps putting out white papers and pubs related to their #mTurk #research platform.

https://link.springer.com/article/10.3758/s13428-022-01999-x

I deeply appreciate that Cloud Research keeps putting out white papers and pubs related to their #mTurk #research platform.

https://link.springer.com/article/10.3758/s13428-022-01999-x

Vanessa Williamson has another take on the ethical quagmire of #MTurk here: https://www.cambridge.org/core/services/aop-cambridge-core/content/view/B1BDFB1111B416DD0B71540CD6E7D94F/S104909651500116Xa.pdf/div-class-title-on-the-ethics-of-crowdsourced-research-div.pdf

Read the latest in a long line of repeated indications that #MTurk is an ethical and methodological disaster: https://csl.mpg.de/427800/webb_tangney__too_good_to_be_true_2022.pdf

TLDR: author pays research subjects minimum wage; subjects treat her survey with the same respect they’ve been shown.

To the author’s credit, she says as much: “Owing to the compensation structure, ‘workers’ have little incentive to invest the extra time and thought required by open-ended qualitative items.”

@thefirstred

TLDR: author pays subjects at paltry minimum wage; subjects behave as the author valued them.

Loud protestations by some on this thread aside, #MTurk is an ethical and methodological clusterf*ck, and that’s been known for some time.

@thefirstred

TLDR: author pays subjects at paltry minimum wage; subjects behave as the author valued them.

Loud protestations by some on this thread aside, #MTurk is an ethical and methodological clusterf*ck, and that’s been known for some time.