Meaningless research

Medicine and doctors are often symbolized by reference to Greek mythology:

Nowadays, though, much of so-called medical research would be better represented by p-values rampant over a field of nonsensical “associations”.

I had recently drawn attention (Statistics literacy) to a fine article about the lack of understanding of statistics that pervades the medical scene (Do doctors understand test results?). It explains how the risk of false-positive tests should be — but is not — understood by doctors and communicated to patients; and how relative risks are cited instead of absolute risks — a confusion that Big Pharma does much to promulgate because it helps to sell pills.
Another set of misunderstandings underlies much of the “research” that is picked up by the media as significant news: data mining coupled with the virtually universal mistake of taking correlations as indicating causation. A recent case in point concerns Alzheimer’s Disease (AD):

”Sleep disorders may raise risk of Alzheimer’s, new research shows
Sleep disturbances such as apnea may increase the risk of Alzheimer’s disease, while moderate exercise in middle age and mentally stimulating games, such as crossword puzzles, may prevent the onset of the dementia-causing disease, according to new research to be presented Monday”.

Note that “may increase” and “may prevent” both imply causation, that sleep disorders may actually cause AD while physical and mental exercise may bring about (cause) protection. But the evidence is only that there is some sort of correlation; and it’s vitally important to keep in mind that correlation doesn’t mean an association every time, it means only that two things are found together apparently more often than one might expect purely as a result of chance.

Note too that “may” is an often-used weasel word to insinuate something while guarding against being held accountable for actually asserting it; e.g., big campaign contributions may influence politicians; conflicts of interest may influence researchers; and so on.

Note as well “to be presented”, which illustrates publicity-seeking by researchers and the complicity of the media in that, as they ignore the fact that at this moment the matter is nothing more than hearsay: science is supposed to published and evaluated before being taken at all seriously.

As to “purely as a result of chance”, everyone should understand, but few do, that the almost universally used method of calculating these probabilities as “p-values” is itself quite misleading; see “Statistics can lie, but Jack Good never did — a personal essay in tribute to I J Good, 1916-2009”.
The take-away lesson is that what researchers claim as “statistically significant” is often not at all significant, indeed it may be entirely meaningless; yet it will still be picked up by the media and ballyhooed as the latest breakthrough.

Here’s how researchers in medically related fields (and elsewhere too, of course) can generate publications effortlessly and prolifically while disseminating misleading notions:
Select a topic — AD, say.
Collect a data set of people who have that condition, the data including every conceivable characteristic: age (in several categories such as young, early middle age, middle age, late middle age, young-old, fairly old, quite old); exercise habits (light, moderate, heavy); alcohol consumption (light, moderate, heavy); diet (many variables — fat, meat, dairy, vegan, gluten-free, etc.); other medical conditions and history (many indeed); race and ethnicity; any others you can think of (urban or rural, say; employment history and type of employment — veteran; blue or white collar; innumerable possibilities); tests by MRI, complete blood analysis, etc.
Feed all the data into a computer, and set it to find correlations.

Purely by chance, at least 1 of every 20 possible pairings will produce a “p ≤ 0.5 statistically significant” result. Since p ≤ 0.5, this is publishable, especially since the written report fails to emphasize that this resulted from a random sweep through 20 times as many pairings.

Naturally such results are quite likely to make little sense, since they are random by-chance associations. For example:
“[M]oderate physical exercise in middle age could decrease the risk that their cognitive deficits progress to dementia. . . .
Oddly, however, the association did not hold for people who engaged in light or vigorous exercise in middle age or for any level of physical activity later in life.
On a similarly counterintuitive note, another study suggested that high blood pressure among people at least 90 years old — “the oldest old” — may protect against cognitive impairment. . . . although hypertension is believed to increase the risk of Alzheimer’s and dementia for middle-aged people, the risk may shift with time” [emphases added].

The reason these results seem so incongruous and counterintuitive is, of course, that they were never genuine results at all, just “associations” that occurred when looking for correlations among a whole host of possibilities.

The notion that moderate exercise but not light or heavy exercise might actually be a significant cause of something like Alzheimer’s is not entirely beyond the realm of possibility, I suppose. Still, it seems sufficiently farfetched that I would hesitate — or be ashamed — even to mention the possibility until it had been reproduced in quite a few studies.
On the other hand I’m perfectly willing to see an association between high blood pressure and good cognition in the elderly, since good cognition depends on plentiful oxygen which depends on a good blood flow; and since arteries become less flexible with age, more pressure is needed to achieve that.
On a further hand, though, the notion that high blood pressure increases the risk of dementia in middle age strikes me as sufficiently absurd as to be dismissed pending the strongest most direct possible evidence; it “is believed” by whom?

Common sense cries out to be applied whenever a p ≤ 0.5 association is touted as meaning something. Try it out on the suggestion that “A daily high dose of Vitamin E may slow early Alzheimer’s disease”. Think about the caveats in that piece, and the trivial magnitude of the reported possible effect.

That respected mass media feature such garbage may well be quite harmful. I would expect some number of people will start taking vitamin E supplements immediately, whether or not there are any indications that they are not getting enough of it already. Not to speak of all the >90-year-olds desperately trying to raise their blood pressure and puzzled about how to decide how much exercise at their age is “moderate” but not “light” or “heavy”; and all the 70-to-90-year-olds wondering at what age high blood pressure stops causing Alzheimer’s and starts protecting against it.