Most of the people I talk to are well-educated engineers in Silicon Valley, yet even they don’t understand what is true and false in our society.
Fallacy 1: “I want to see peer-reviewed results.”
Peer review just means that some people, usually experts, have read the paper, not that they have duplicated the results. In fact, they virtually never attempt to reproduce the results in the paper.
Going deeper, there’s several problems with peer review:
- the only person with the grant money to write the paper is the author, so there’s usually no way to even finance reproducing the results
- paper authors seldom include the raw data with their paper for 2 reasons: they can get more papers out of the same data set, and they can get credit for each paper without furthering competitors. Of course without the underlying data, who knows if their are mistakes or malfeasance?
- When experiments are duplicated, they seldom match. Recently some drug companies have attempted to reproduce important papers to create known foundations for their own programs. They virtually always fail to see the same results. (Mendeleev himself likely published fake results, as statistics shows his ratios are too good.)
- peer reviewers are subject to the same politics and biases as any other human endeavor
- In the dismal case of “medicine science”, generally all of the New England Medical Journal published results are considered to be wrong after 10 years
- In the case of physics, how do you verify even a single paper on string theory?
Fallacy 2: “It’s good/safe if it is FDA-approved.”
FDA approval just means that a new drug is better than a placebo in trials, not that it is better than existing drugs. Also, the drug may work well in one ethnic or gender group and not in another group. The fact that drugs have unintended effects when used off-label proves that the manufacturer doesn’t even know what all the effects of the drug are.
Fallacy 3: “I want data-driven results.”
Data-driven analysis is all the rage in Silicon Valley today, but few people I talk to even know what that means, or how futile it is in inventing new products.
Prerequisites would be:
- enough relevant data
- enough statistical and domain knowledge to model the data, design tests and interpret the results
- enough resources to do the data analysis without reducing ability to do product development or customer support. Google and Yahoo! can afford legitimate A/B testing. Most startups simply can’t.
“The best way to predict the future is to invent it.” Apple’s recent consumer products success is because of delivering and marketing products that nobody was asking for yet, not analytics.
Fallacy 4: “The economy is this or that. Unemployment is this or that.”
Economists are captive to their employers, whether government or private. Either they parrot what their employer wants, and what their “economic school” dictates, or they’re soon unemployed.
Regarding statistics on unemployment, they’re generally quietly restated a couple years after a recession. In the case of Silicon Valley, during and immediately after a recession the newspapers publish unemployment rates of 10%, then restated them as 25% to 30% two years later.