I’ve been reading Ben Goldacre’s “Bad Science”. “Shocking” is one word I’d use. The people behind “Brain Gym”, “detoxing” footbaths, Patrick Holford, Homeopaths, most of the pharmaceutical industry, Gillian McKeith and others ought to be thoroughly ashamed of themselves.
“Puzzled” is another word I’d use. Why are so many trials, even those published in “prestigious” journals, misleading? Ben’s answer partly lies around:
1. Crap protocols
2. Crap write-up
3. Publication bias
I would like to make a suggestion as to how these three can be easily, and cheaply, combated. My suggestion is the, modestly titled, “Lloyd meta-protocol”.
The “Lloyd meta-protocol” is to be followed by a journal and consists of three things:
1. The publish/not publish decision is not taken after the results are known. Where it is to be published is not taken after the results are known. The decision is taken by peer-review of the protocol for the test. These protocols are published in the journal with space for letters criticising protocols. An amended protocol may be published making changes to meet criticisms of the protocol. No actual testing happens until after the protocol is “finalised”. (A protocol is “finalised” when a decision not to further amend it is taken.) Following this protocol actually stops badly designed trials being carried out.
2. The finalisation of the test protocol involves the advance acceptance of the paper by the journal and the agreement on the part of the researchers that there will be a paper. Any trials where the paper fails to materialise after a certain time-limit a reported on by a short one-paragraph assuming that the results were entirely negative. This part of the protocol protects against “publication bias”.
3. That agreement that the paper will be published might lead one to expect that any old crap knocked up by the researchers, with all the ways of being dishonest in the write up Ben told us about, will be accepted. The “Lloyd meta-protocol” avoids that by removing the task of writing up the paper from the research group and giving it to the journal. The test is written up by the journal. Sure, the research group will sit with the writer. The research group will suggest wording (perhaps, even, giving a draft of a paper to the journal) but the person who actually chooses which words are used and in which order will be a journal appointee.
Those “playing with a straight bat” should welcome these procedures. The criticism of the test protocols, at present, happens when all that expensive field research has been done. If those reading the paper are not happy about the protocols then that field research has been a waste of time and money. Under the Lloyd meta-protocol rejection happens early and rejection happens cheaply. They will be guaranteed the journal in which the results will appear. No need to sweat on this one, no need to work on some research only to find you can’t publish it in “Prestigious Journal” but only “Obscure Piece of Toilet Paper”. But there’s more. If “Prestigious Journal” follows the “Lloyd meta-protocol” then readers of the journal will know that all trials reported:
o Have peer-reviewed protocols
o Are not subject to publication bias
o Are written up independently of the researchers
In short, “Prestigious Journal” will become “Super-prestigious journal” and which researchers wouldn’t want to publish in that?
Those currently “playing it a bit dodgy” should be tempted by these procedures. As above, publication (if they followed the protocol) would be in “Super-prestigious Journal” following the Lloyd meta-protocol. How’s that going to sound? Pretty bloody good. If you’ve got a good product then you could make it look really good by playing a few dodgy tricks. But now you’d loose out on that “stamp of approval”. Does the prestige that can be got by fiddling protocols, dodgy write ups and publication bias really make up for the loss of the “stamp of approval”? Maybe, maybe not. But at least now they’ll have to think about it.
Medical professionals should adore these procedures. No more trailing through paper after paper just to find you should ignore the results. As well as publication in a Lloyd meta-protocol compliant journal being a “stamp of approval”, publication in a non-compliant journal is a warning sign. These people before they did the test were determined to do something dodgy. Avoid them:
- Try X, Y, Z it reduces the A, B, C count in populations 1,2,3
- Was that in “Super-prestigious Journal”?
- No, it was in “Obscure Piece of Toilet Paper”
- Go away then
- Try X, Y, Z it reduces the A, B, C count in populations 1,2,3
- Was that in “Super-prestigious Journal”?
- Yes, it was.
- Interesting, tell me more
The rest of us should welcome these procedures. Despite having read Ben’s book I do not have the ability to read through a medical research paper and, sorry Ben, I don’t really want to. I don’t think I can rely on an abstract (Ben’s taught me that much), but I’d be quite happy to rely on it if there was a big blue “Lloyd meta-protocol compliant” tick next to it.
The journals should welcome these procedures. It will increase their prestige (from prestigious to “super-prestigious) and, importantly, it’s easy for one journal to do it on its own, right now. We do not need a central registry of trials; the journal announces that it is its own registry. We don’t need some great world-conference to “renegotiate” a “settlement” on “conduct and publication of research”. “Obscure Piece of Toilet Paper” can remain as it is, you can’t hide bad results there anymore, just because “Super-prestigious Journal” (on its own) already has an agreement to publish it.
That’s probably the biggest argument in favour of the Lloyd meta-protocol: it doesn’t cost anything and you could start on Monday.
A categorial end in sight
2 days ago
