#phacking — Public Fediverse posts

“Promiscuous dichotomisation is yet another way of torturing the data to produce false confessions of which we must be eternally wary.” #statistics #P-hacking #stats

RE: https://bsky.app/profile/did:plc:3zyo4iakhqs47bttjalqgbk7/post/3m5jwkr7swc2j

#phacking #LLM performance:

"The researchers found that big players like Meta, Google, OpenAI, and Amazon are given special privileges to privately test multiple versions of their models and only publish the best results. This hidden practice allows them to inflate their rankings by cherry picking data, making their models appear stronger than they actually are."

https://arxiv.org/abs/2504.20879?utm_source=beehiiv&utm_medium=newsletter&utm_campaign=mediamobilize&_bhlid=045dede5ad6eedcb96ec953f2fbac11a166a2243

via Sabine Hossenfelder's newsletter

🧵
> ... unethical behaviour during the report of results is.. P hacking... frequent in research.. [of a] clinical nature... two main reasons.. First, scientists are often evaluated by the number and quality of publications, and sometimes this pressure to get sig­nificant results makes some scientists cherry-pick their results. Second (and more frequent), some inexperienced analysts are unaware of the importance of #MultipleTesting and think this is
OK. But it is not! #PHacking
@[email protected]

Fun class class survey of other undergrads, current N=55. I'm doing #irresponsible #DataAnalysis bc not actual #research.

Still #WTF?

ghost_recv_log = How many times have you been ghosted? (log-transformed)

mosi = Misperception of others' sexual interest

bjw = Belief in a just world

swls = Satisfaction with life

csei = College self-efficacy

High self-esteem assholery? IDK.

OH WAIT. Gender!

Shit. We didn't ask.

#NotResearch #Datafishing #phacking but it's #OK I'm a #professional #oops

Table 1 of the "False-positive psychology" paper (Simmons, Nelson & Simonsohn 2011, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2916240) estimate the false-positive rate of some questionable #ResearchPractices, both alone and in combination (see attached figure).

I remember reading somewhere that the authors later stated that the numbers were inaccurate and should have been somewhat higher. Does anyone have a reference for this? #FalsePositivePsychology #QRP #phacking #HARKing #ReproducibiliTea

[en] Cheating in Science: Harvard "Honesty Scholar" May Have Been Caught in Dishonesty

"... dishonesty can lead to creativity" - an interesting and somewhat amusing read.

The New York Times: "Questions about a widely cited paper are the latest to be raised about methods used in #behavioral research."

https://datacolada.org/111

#ResearchHighlights #honesty #dishonesty #phacking #harking #dredging #gino #harvard #fraud #cheating #academic #datacolada

“It’s a bit like seeing a rabbit shape in the clouds and then testing whether all clouds look like rabbits… using the same cloud. I hope you appreciate that you’re going to need some new clouds to test your theory.

Any datapoint you use to inspire a theory or question can’t be used to test that same theory.”

#phacking #statistics #rstats

https://towardsdatascience.com/the-most-powerful-idea-in-data-science-78b9cd451e72

@sociology @communicationscholars @rstats

"Do not chase better metrics. Chase better models."

Excellent lighting talk on how spatial sampling affects the quality of data driven models by Luísa Lucchese

https://youtu.be/q9dy2dQuTak

Chasing better #metrics is like #phacking

#GeoAI #gischat #datascience #machinelearning

If you’re interested in #statistics, #Metascience, #openScience, #pHacking, etc., this is quite a good read.

https://joebakcoleman.com/blog/2023/pcurve/

In 2012, Noam Chomsky mentioned a Lisp-language guy, #PatWinston. Winston said that, in the AI^1 field, people were directed away from "original questions." That sent me to #Chomsky's #PowersAndProspects(1996). _Prospects_ reminded me of Alex Carey's _Taking the Risk Out of Democracy_ and I now see #Mayo's #HawthorneStudies as 1927-1932 #PHacking. #GeorgeEltonMayo! in textbooks.
^1 AI, short for SALAMI: Systematic Approaches to Learning Algorithms and Machine Inferences #AISalami

> Psychologists who specialize in exercise music have quantified.. [that] listening to songs with high tempos motivates us to run faster, and the swifter we move, the quicker we prefer our music. Likewise, when drivers hear loud, fast music, they unconsciously step a bit harder on the gas pedal. Walking at our own pace creates an unadulterated feedback loop between the rhythm of our bodies and our mental state that we cannot experience as easily.. during any other.. locomotion.
#PHacking?

> OUTRIGHT FAKERY IS CLEARLY more common in.. sciences than we’d like to believe. But it may not be the biggest threat to their credibility... #MichaelKinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” #Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”
#PHacking #OnBullshit?