P-Hacking: How to Make Anything “True” with Enough Data Tricks
Just keep testing until something works. What could possibly go wrong?
Fellow Data Tinkerers!
Thank you to those who provided feedback this week and to the 2 people who shared the publication with others. I really appreciate it ❤️
I wanted to share an example of what you could unlock if you share Data Tinkerer with just 3 other people.
There are 100+ more cheat sheets covering everything from Python, R, SQL, Spark to Power BI, Tableau, Git and many more. So if you know other people who like staying up to date on all things data, please share Data Tinkerer with them!
Now, without further ado, let’s get into a data analysis issue you want to avoid!
You’ve launched a new A/B test: a green “Buy Now” button vs. the classic blue.
First check: p = 0.08
Second check: p = 0.06
Third check after lunch: p = 0.049
It’s significant, baby!
You screenshot it. You Slack it. You present it. But deep down, you know: you didn’t find truth, you just refreshed the data until the universe gave up.
Welcome to the world of p-hacking, where stats are optional and significance is flexible