Add group to favorites
Back to post list
Send new message to group Search:
Post Subject:
Interesting study of bias in medical research
Interesting study of bias in medical research
I have just watched a video of a fascinating presentation by
Prof. John Ioannidis at NIH entitled: "Great Teachers:
Translation, Replication and Credibility of Research Findings".
It is available on the web at:
* videocast.nih.gov/PastEventDetail.asp?13691
I think this is available to all, but I'm not certain because I'm
watching it from inside the NIH firewall.
Ioannidis has done a series of studies of medical publishing
extending over quite a few years. His method is to do an
extensive search of the literature on a specific topic, identify
all studies on that topic, isolate the findings of each study,
and perform statistical analysis to find out how often and with
what degree of correlation initial results are confirmed by
follow on studies.
I'm not a statistician but I hope I'm presenting a reasonable
precis of what he said.
What he said was that the vast majority of "statistically
significant" results reported in the scientific literature (89%
if I remember correctly) turn out not to be statistically
significant at all after later studies are done to attempt to
replicate the results. He believed that the most common source
of "significance" in results turns out to be bias of one of
several kinds. Typical sources of bias are:
Sample size bias:
Many studies are reporting effects that are small enough
that the sample sizes are insufficient to justify the
reported effects.
Selection bias:
The authors analyze and report findings selectively,
leaving out data that they regarded as "insignificant",
but which in fact often is significant.
Publication bias:
Journals look for articles that show highly statistically
significant results, therefore biasing publishing in the
direction of those articles that contain lots of outliers
in their findings.
Fraud:
He said that there were reported cases of this and seemed
to think that more of the bias that authors introduce is
not as inadvertent as we would hope that it is.
One thing that I found particularly interesting in his
presentation was that he derived numerical values for bias. For
example, in one specific field, he compared early reports with
a huge, well conducted study of thousands of patients, and
found that there was a built in bias of 1.3 in the results of the
early reports. If I understand that correctly, he is saying
that, in that field, unless a particular effect is at least 30%
greater than the controls, it is indistinguishable from the
built-in bias in that field.
When a very bright Chinese student came to work in his lab, he
did a parallel study on Chinese publishing in the same field and
found the bias there was 3.0 (compared to 1.3 in European and
American journals.) In that field, the Americans and Europeans
were doing a bad job, but the Chinese were atrocious.
What all this means to me is that I'm going to add a lot more
grains of salt to all of the latest research findings that we see
reported daily in the scientific press.
Alan
I have just watched a video of a fascinating presentation by
Prof. John Ioannidis at NIH entitled: "Great Teachers:
Translation, Replication and Credibility of Research Findings".
It is available on the web at:
* videocast.nih.gov/PastEventDetail.asp?13691
I think this is available to all, but I'm not certain because I'm
watching it from inside the NIH firewall.
Ioannidis has done a series of studies of medical publishing
extending over quite a few years. His method is to do an
extensive search of the literature on a specific topic, identify
all studies on that topic, isolate the findings of each study,
and perform statistical analysis to find out how often and with
what degree of correlation initial results are confirmed by
follow on studies.
I'm not a statistician but I hope I'm presenting a reasonable
precis of what he said.
What he said was that the vast majority of "statistically
significant" results reported in the scientific literature (89%
if I remember correctly) turn out not to be statistically
significant at all after later studies are done to attempt to
replicate the results. He believed that the most common source
of "significance" in results turns out to be bias of one of
several kinds. Typical sources of bias are:
Sample size bias:
Many studies are reporting effects that are small enough
that the sample sizes are insufficient to justify the
reported effects.
Selection bias:
The authors analyze and report findings selectively,
leaving out data that they regarded as "insignificant",
but which in fact often is significant.
Publication bias:
Journals look for articles that show highly statistically
significant results, therefore biasing publishing in the
direction of those articles that contain lots of outliers
in their findings.
Fraud:
He said that there were reported cases of this and seemed
to think that more of the bias that authors introduce is
not as inadvertent as we would hope that it is.
One thing that I found particularly interesting in his
presentation was that he derived numerical values for bias. For
example, in one specific field, he compared early reports with
a huge, well conducted study of thousands of patients, and
found that there was a built in bias of 1.3 in the results of the
early reports. If I understand that correctly, he is saying
that, in that field, unless a particular effect is at least 30%
greater than the controls, it is indistinguishable from the
built-in bias in that field.
When a very bright Chinese student came to work in his lab, he
did a parallel study on Chinese publishing in the same field and
found the bias there was 3.0 (compared to 1.3 in European and
American journals.) In that field, the Americans and Europeans
were doing a bad job, but the Chinese were atrocious.
What all this means to me is that I'm going to add a lot more
grains of salt to all of the latest research findings that we see
reported daily in the scientific press.
Alan
Re: Interesting study of bias in medical research
Alan Meyer a scris:
> I have just watched a video of a fascinating presentation by
> Prof. John Ioannidis at NIH entitled: "Great Teachers:
> Translation, Replication and Credibility of Research Findings".
> It is available on the web at:
>
> * videocast.nih.gov/PastEventDetail.asp?13691
>
> I think this is available to all, but I'm not certain because I'm
> watching it from inside the NIH firewall.
>
> Ioannidis has done a series of studies of medical publishing
> extending over quite a few years. His method is to do an
> extensive search of the literature on a specific topic, identify
> all studies on that topic, isolate the findings of each study,
> and perform statistical analysis to find out how often and with
> what degree of correlation initial results are confirmed by
> follow on studies.
>
> I'm not a statistician but I hope I'm presenting a reasonable
> precis of what he said.
>
> What he said was that the vast majority of "statistically
> significant" results reported in the scientific literature (89%
> if I remember correctly) turn out not to be statistically
> significant at all after later studies are done to attempt to
> replicate the results. He believed that the most common source
> of "significance" in results turns out to be bias of one of
> several kinds. Typical sources of bias are:
>
> Sample size bias:
>
> Many studies are reporting effects that are small enough
> that the sample sizes are insufficient to justify the
> reported effects.
>
> Selection bias:
>
> The authors analyze and report findings selectively,
> leaving out data that they regarded as "insignificant",
> but which in fact often is significant.
>
> Publication bias:
>
> Journals look for articles that show highly statistically
> significant results, therefore biasing publishing in the
> direction of those articles that contain lots of outliers
> in their findings.
>
> Fraud:
>
> He said that there were reported cases of this and seemed
> to think that more of the bias that authors introduce is
> not as inadvertent as we would hope that it is.
>
> One thing that I found particularly interesting in his
> presentation was that he derived numerical values for bias. For
> example, in one specific field, he compared early reports with
> a huge, well conducted study of thousands of patients, and
> found that there was a built in bias of 1.3 in the results of the
> early reports. If I understand that correctly, he is saying
> that, in that field, unless a particular effect is at least 30%
> greater than the controls, it is indistinguishable from the
> built-in bias in that field.
>
> When a very bright Chinese student came to work in his lab, he
> did a parallel study on Chinese publishing in the same field and
> found the bias there was 3.0 (compared to 1.3 in European and
> American journals.) In that field, the Americans and Europeans
> were doing a bad job, but the Chinese were atrocious.
>
> What all this means to me is that I'm going to add a lot more
> grains of salt to all of the latest research findings that we see
> reported daily in the scientific press.
>
> Alan
Alan Meyer a scris:
> I have just watched a video of a fascinating presentation by
> Prof. John Ioannidis at NIH entitled: "Great Teachers:
> Translation, Replication and Credibility of Research Findings".
> It is available on the web at:
>
> * videocast.nih.gov/PastEventDetail.asp?13691
>
> I think this is available to all, but I'm not certain because I'm
> watching it from inside the NIH firewall.
>
> Ioannidis has done a series of studies of medical publishing
> extending over quite a few years. His method is to do an
> extensive search of the literature on a specific topic, identify
> all studies on that topic, isolate the findings of each study,
> and perform statistical analysis to find out how often and with
> what degree of correlation initial results are confirmed by
> follow on studies.
>
> I'm not a statistician but I hope I'm presenting a reasonable
> precis of what he said.
>
> What he said was that the vast majority of "statistically
> significant" results reported in the scientific literature (89%
> if I remember correctly) turn out not to be statistically
> significant at all after later studies are done to attempt to
> replicate the results. He believed that the most common source
> of "significance" in results turns out to be bias of one of
> several kinds. Typical sources of bias are:
>
> Sample size bias:
>
> Many studies are reporting effects that are small enough
> that the sample sizes are insufficient to justify the
> reported effects.
>
> Selection bias:
>
> The authors analyze and report findings selectively,
> leaving out data that they regarded as "insignificant",
> but which in fact often is significant.
>
> Publication bias:
>
> Journals look for articles that show highly statistically
> significant results, therefore biasing publishing in the
> direction of those articles that contain lots of outliers
> in their findings.
>
> Fraud:
>
> He said that there were reported cases of this and seemed
> to think that more of the bias that authors introduce is
> not as inadvertent as we would hope that it is.
>
> One thing that I found particularly interesting in his
> presentation was that he derived numerical values for bias. For
> example, in one specific field, he compared early reports with
> a huge, well conducted study of thousands of patients, and
> found that there was a built in bias of 1.3 in the results of the
> early reports. If I understand that correctly, he is saying
> that, in that field, unless a particular effect is at least 30%
> greater than the controls, it is indistinguishable from the
> built-in bias in that field.
>
> When a very bright Chinese student came to work in his lab, he
> did a parallel study on Chinese publishing in the same field and
> found the bias there was 3.0 (compared to 1.3 in European and
> American journals.) In that field, the Americans and Europeans
> were doing a bad job, but the Chinese were atrocious.
>
> What all this means to me is that I'm going to add a lot more
> grains of salt to all of the latest research findings that we see
> reported daily in the scientific press.
>
> Alan
Re: Interesting study of bias in medical research
Alan Meyer a scris:
> I have just watched a video of a fascinating presentation by
> Prof. John Ioannidis at NIH entitled: "Great Teachers:
> Translation, Replication and Credibility of Research Findings".
> It is available on the web at:
>
> * videocast.nih.gov/PastEventDetail.asp?13691
>
> I think this is available to all, but I'm not certain because I'm
> watching it from inside the NIH firewall.
>
> Ioannidis has done a series of studies of medical publishing
> extending over quite a few years. His method is to do an
> extensive search of the literature on a specific topic, identify
> all studies on that topic, isolate the findings of each study,
> and perform statistical analysis to find out how often and with
> what degree of correlation initial results are confirmed by
> follow on studies.
>
> I'm not a statistician but I hope I'm presenting a reasonable
> precis of what he said.
>
> What he said was that the vast majority of "statistically
> significant" results reported in the scientific literature (89%
> if I remember correctly) turn out not to be statistically
> significant at all after later studies are done to attempt to
> replicate the results. He believed that the most common source
> of "significance" in results turns out to be bias of one of
> several kinds. Typical sources of bias are:
>
> Sample size bias:
>
> Many studies are reporting effects that are small enough
> that the sample sizes are insufficient to justify the
> reported effects.
>
> Selection bias:
>
> The authors analyze and report findings selectively,
> leaving out data that they regarded as "insignificant",
> but which in fact often is significant.
>
> Publication bias:
>
> Journals look for articles that show highly statistically
> significant results, therefore biasing publishing in the
> direction of those articles that contain lots of outliers
> in their findings.
>
> Fraud:
>
> He said that there were reported cases of this and seemed
> to think that more of the bias that authors introduce is
> not as inadvertent as we would hope that it is.
>
> One thing that I found particularly interesting in his
> presentation was that he derived numerical values for bias. For
> example, in one specific field, he compared early reports with
> a huge, well conducted study of thousands of patients, and
> found that there was a built in bias of 1.3 in the results of the
> early reports. If I understand that correctly, he is saying
> that, in that field, unless a particular effect is at least 30%
> greater than the controls, it is indistinguishable from the
> built-in bias in that field.
>
> When a very bright Chinese student came to work in his lab, he
> did a parallel study on Chinese publishing in the same field and
> found the bias there was 3.0 (compared to 1.3 in European and
> American journals.) In that field, the Americans and Europeans
> were doing a bad job, but the Chinese were atrocious.
>
> What all this means to me is that I'm going to add a lot more
> grains of salt to all of the latest research findings that we see
> reported daily in the scientific press.
>
> Alan
i don't understand i want more details about this subject because i;m
verry interested! Thank's a lot! Have a good day!!!
Alan Meyer a scris:
> I have just watched a video of a fascinating presentation by
> Prof. John Ioannidis at NIH entitled: "Great Teachers:
> Translation, Replication and Credibility of Research Findings".
> It is available on the web at:
>
> * videocast.nih.gov/PastEventDetail.asp?13691
>
> I think this is available to all, but I'm not certain because I'm
> watching it from inside the NIH firewall.
>
> Ioannidis has done a series of studies of medical publishing
> extending over quite a few years. His method is to do an
> extensive search of the literature on a specific topic, identify
> all studies on that topic, isolate the findings of each study,
> and perform statistical analysis to find out how often and with
> what degree of correlation initial results are confirmed by
> follow on studies.
>
> I'm not a statistician but I hope I'm presenting a reasonable
> precis of what he said.
>
> What he said was that the vast majority of "statistically
> significant" results reported in the scientific literature (89%
> if I remember correctly) turn out not to be statistically
> significant at all after later studies are done to attempt to
> replicate the results. He believed that the most common source
> of "significance" in results turns out to be bias of one of
> several kinds. Typical sources of bias are:
>
> Sample size bias:
>
> Many studies are reporting effects that are small enough
> that the sample sizes are insufficient to justify the
> reported effects.
>
> Selection bias:
>
> The authors analyze and report findings selectively,
> leaving out data that they regarded as "insignificant",
> but which in fact often is significant.
>
> Publication bias:
>
> Journals look for articles that show highly statistically
> significant results, therefore biasing publishing in the
> direction of those articles that contain lots of outliers
> in their findings.
>
> Fraud:
>
> He said that there were reported cases of this and seemed
> to think that more of the bias that authors introduce is
> not as inadvertent as we would hope that it is.
>
> One thing that I found particularly interesting in his
> presentation was that he derived numerical values for bias. For
> example, in one specific field, he compared early reports with
> a huge, well conducted study of thousands of patients, and
> found that there was a built in bias of 1.3 in the results of the
> early reports. If I understand that correctly, he is saying
> that, in that field, unless a particular effect is at least 30%
> greater than the controls, it is indistinguishable from the
> built-in bias in that field.
>
> When a very bright Chinese student came to work in his lab, he
> did a parallel study on Chinese publishing in the same field and
> found the bias there was 3.0 (compared to 1.3 in European and
> American journals.) In that field, the Americans and Europeans
> were doing a bad job, but the Chinese were atrocious.
>
> What all this means to me is that I'm going to add a lot more
> grains of salt to all of the latest research findings that we see
> reported daily in the scientific press.
>
> Alan
i don't understand i want more details about this subject because i;m
verry interested! Thank's a lot! Have a good day!!!
