#phacking — Public Fediverse posts

An anecdotal account of how easy it is to circumvent LLMs' overt resistance to p-hacking data sets:

https://statmodeling.stat.columbia.edu/2026/10/03/using-the-computer-to-p-hack-id-rather-use-it-to-fit-multilevel-models/

#Science #AI #LLMs #NoAI #phacking

6/
She notes [1]:
"Back in 1975, psychologist Tony Greenwald noted that science is prejudiced against #NullHypotheses: We even refer to sound work supporting a null hypothesis as “failed experiments.” That prejudice results in #PublicationBias: researchers are less likely to write up studies showing no effect; editors less likely to accept them, and so no one can learn from them"

[1] cites the #pHacking bias: "the practice of trying many analyses and only reporting the 'significant' results"

5/
An issue of too vast a scope, @deevybee notes [1]: "It’s not that [scientists] are unconcerned about doing #science well; it's just that many of them don’t recognise that there are serious problems with current practices"
and asks "How can that be?"

"Many researchers persist in doing research in a way virtually guaranteed not to deliver meaningful results" incurring insidious biases: "#PublicationBias, low statistical power, #pHacking and #HARKing (hypothesising after results are known)" [1]

“Promiscuous dichotomisation is yet another way of torturing the data to produce false confessions of which we must be eternally wary.” #statistics #P-hacking #stats

RE: https://bsky.app/profile/did:plc:3zyo4iakhqs47bttjalqgbk7/post/3m5jwkr7swc2j

#phacking #LLM performance:

"The researchers found that big players like Meta, Google, OpenAI, and Amazon are given special privileges to privately test multiple versions of their models and only publish the best results. This hidden practice allows them to inflate their rankings by cherry picking data, making their models appear stronger than they actually are."

https://arxiv.org/abs/2504.20879?utm_source=beehiiv&utm_medium=newsletter&utm_campaign=mediamobilize&_bhlid=045dede5ad6eedcb96ec953f2fbac11a166a2243

via Sabine Hossenfelder's newsletter

🧵
> ... unethical behaviour during the report of results is.. P hacking... frequent in research.. [of a] clinical nature... two main reasons.. First, scientists are often evaluated by the number and quality of publications, and sometimes this pressure to get sig­nificant results makes some scientists cherry-pick their results. Second (and more frequent), some inexperienced analysts are unaware of the importance of #MultipleTesting and think this is
OK. But it is not! #PHacking
@[email protected]

Fun class class survey of other undergrads, current N=55. I'm doing #irresponsible #DataAnalysis bc not actual #research.

Still #WTF?

ghost_recv_log = How many times have you been ghosted? (log-transformed)

mosi = Misperception of others' sexual interest

bjw = Belief in a just world

swls = Satisfaction with life

csei = College self-efficacy

High self-esteem assholery? IDK.

OH WAIT. Gender!

Shit. We didn't ask.

#NotResearch #Datafishing #phacking but it's #OK I'm a #professional #oops

Table 1 of the "False-positive psychology" paper (Simmons, Nelson & Simonsohn 2011, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2916240) estimate the false-positive rate of some questionable #ResearchPractices, both alone and in combination (see attached figure).

I remember reading somewhere that the authors later stated that the numbers were inaccurate and should have been somewhat higher. Does anyone have a reference for this? #FalsePositivePsychology #QRP #phacking #HARKing #ReproducibiliTea

[en] Cheating in Science: Harvard "Honesty Scholar" May Have Been Caught in Dishonesty

"... dishonesty can lead to creativity" - an interesting and somewhat amusing read.

The New York Times: "Questions about a widely cited paper are the latest to be raised about methods used in #behavioral research."

https://datacolada.org/111

#ResearchHighlights #honesty #dishonesty #phacking #harking #dredging #gino #harvard #fraud #cheating #academic #datacolada

“It’s a bit like seeing a rabbit shape in the clouds and then testing whether all clouds look like rabbits… using the same cloud. I hope you appreciate that you’re going to need some new clouds to test your theory.

Any datapoint you use to inspire a theory or question can’t be used to test that same theory.”

#phacking #statistics #rstats

https://towardsdatascience.com/the-most-powerful-idea-in-data-science-78b9cd451e72

@sociology @communicationscholars @rstats

"Do not chase better metrics. Chase better models."

Excellent lighting talk on how spatial sampling affects the quality of data driven models by Luísa Lucchese

https://youtu.be/q9dy2dQuTak

Chasing better #metrics is like #phacking

#GeoAI #gischat #datascience #machinelearning

If you’re interested in #statistics, #Metascience, #openScience, #pHacking, etc., this is quite a good read.

https://joebakcoleman.com/blog/2023/pcurve/

> Psychologists who specialize in exercise music have quantified.. [that] listening to songs with high tempos motivates us to run faster, and the swifter we move, the quicker we prefer our music. Likewise, when drivers hear loud, fast music, they unconsciously step a bit harder on the gas pedal. Walking at our own pace creates an unadulterated feedback loop between the rhythm of our bodies and our mental state that we cannot experience as easily.. during any other.. locomotion.
#PHacking?

Significant

https://xkcd.com/882

#jellybeans #xkcd #science #probability #phacking