There's a simple bias that seems to endure. Anyone may accidentally fall into its trap. Once they do, they make it impossible for others to avoid doing the same thing. It's called the Survivorship Bias, and it can be used to convince people of nearly anything.
Let's say that I walked up to you in the street and told you I was psychic and I could prove it. The more fleet-footed of you would manage to keep walking, but I think I'd back a few of you into a corner and make you watch my demonstration. It's a video of me walking into a room with a group of witnesses. One of the witnesses - picked at random - flips a coin. After he flips, but before he shows the coin to anyone, I guess whether it's heads or tails. He reveals it to the witnesses. I'm right. The demonstration goes on, and each time I'm right.
You're naturally skeptical, so you start questioning the set-up. I have sworn statements from all the witnesses that they were all pulled off the street. I even have the video of the selection process. I have a video of what I did all that day, showing that I didn't set up anything that might clue me in on the coin flip results. After a lot of investigating, you determine that there's no trick. I'm just correctly guessing the coin each time. You're right. There isn't a trick.
There's just hundreds days of set-up. On every one of those days, I have people go out and find witnesses, and people film me going about my business. On every one of those days, I walk into the room and start guessing. And on every one of those days, except for the day that I showed you, I guess wrong. I've showed you the one remarkable case, the one day that I guessed right.
To be fair, that's more survivorship deception. But it does show how coincidence can, when taken out of context, look like something more. It's been used by psychic researchers in the past. Test enough people and some of them will give answers that are coincidentally accurate. Then pretend that those people are psychics.
It's also been used in economics. I had an economics teacher who joked that the best way to make millions as an investment adviser was to send out, to thousands of people, two variations of a letter. One would predict one trend in the stock market and the other would predict the opposite trend. Let the market takes its course. Strike the people that got the inaccurate letter from your mailing list. Then send out another set of letters. Repeat the pattern until you've got only a few clients, and each one of them is absolutely convinced that you're infallible.
How often does it happen in science? Without a doubt some people pull (consciously or not) the same trick. Drug companies, when they submit drugs for testing, have to register all tests of a drug. Some companies got in the habit of sponsoring multiple tests, and only showing the results of those that were up to snuff. Not that Big Pharma is the only industry that pulls this.
There are plenty of people who believe that natural remedies work. They have the studies to prove it, they say. And they're right. It's just that they're either ignoring, or have never seen, the many other studies that prove their natural remedy has no effect. If you do a hundred studies on whether wheat grass juice stops migraines, one of them might have striking results. That doesn't actually make wheat grass juice effective.
Which is why a paper, Why Most Published Research Findings are False, made waves. There are a lot of ways to unintentially apply survivorship bias. Widen the criteria for certain effects, leave them open to interpretation, use too small a selection pool, or too wide a selection pool with too many criteria. Eventually, one test is going to show something that is a coincidence and not reliable results.
Even publishing itself can skew results. Few people are going to feel the need to go public with a title like, "Tried it. Didn't Work." The only way for that title to get attention is if it followed a paper called, "Tried it. Did Work." And just like that, you get multiple results, that people can interpret in multiple ways.
Via PLOS Medicine