Systematic reviews

WP_Post Object
(
    [ID] => 1301
    [post_author] => 10
    [post_date] => 2017-07-05 13:19:06
    [post_date_gmt] => 2017-07-05 13:19:06
    [post_content] => A systematic review is a summary of the total research evidence available on a particular subject, designed to provide more reliable information than single studies.

Specific methods are used to search academic journals and identify which studies are suitable for inclusion in the review (e.g. those of suitably high quality). From these identified trials a smaller number may be found suitable to go into a meta-analysis. A meta-analysis is a statistical technique used to quantitatively analyse the combined results of multiple studies to generate a more meaningful overall result.

Systematic reviews are intended to draw conclusions from data collected from multiple similar studies, yet the set of trials used in the more famous systematic reviews of homeopathy vary widely in both the type of homeopathic treatment given and the medical conditions being investigated. Particularly, the term ‘homeopathy’ has been interpreted several different ways in the context of homeopathy research1. This means that studies investigating very different approaches are all described as trials of ‘homeopathy’ and are frequently (and inappropriately) analysed together in systematic reviews.

For example, trials pooled for systematic reviews have studied very different aspects of homeopathy making it inappropriate to directly compare them. These different aspects of homeopathy include:
  • A standard single homeopathic medicine given to all participants
  • A standard combination of several homeopathic medicines given to all participants
  • Individualised prescribing without a homeopathic consultation
  • Treatment by a homeopath including individualised prescription and consultation
  • Isopathic trials e.g. homeopathic pollen for treatment of hay fever
For future homeopathy research to be meaningful, specific definitions within the umbrella term of ‘homeopathy’ are needed to promote clarity in the reporting, design and interpretation of homeopathy research, i.e. stating whether a trial is investigating treatment by a homeopath, the action of a homeopathic medicine or the principles of homeopathy2. A small number of systematic reviews have confirmed that homeopathy can be effective for a small number of specific conditions, but much more research of this kind is needed to investigate a wider range of conditions before we can be sure how effective homeopathy may be under idealised research conditions. As such, the British Homeopathic Association (BHA) has a work programme of systematic reviews in place to provide up-to-date analysis of the clinical evidence base for homeopathy3. A number of review papers have been published by the BHA team, led by Dr Robert Mathie. Importantly, one of these reviews has fully characterised the clinical trial evidence that we have for ‘homeopathy’ (including all trials up to 2011) so that we may know which published trials are peer-reviewed, placebo controlled, individualised or clinical homeopathy etc.(4). In addition, Mathie et al published a state-of-the-art, high quality systematic review with an accompanying meta-analysis of randomised placebo-controlled trials of individualised homeopathic treatment across a spectrum of clinical conditions (5). The results of this meta-analysis showed that homeopathic medicines (remedies), when prescribed during individualised treatment, consistent with the usual care provided by homeopaths in a real-world clinical context, are almost twice as likely to have a beneficial effect than placebo. In other words, the most reliable trials of individualised homeopathic treatment show that remedies can have specific and detectable beneficial effects beyond placebo. For a more detailed exploration and explanation of this paper, see the Homeopathic Research Institute (HRI) Autumn 2015 quarterly research article here. While the recent (2014) meta-analysis of Mathie et al (5) looked at the effects of individualised homeopathic treatment, five major systematic reviews have been carried out previously (dating from 1991 to 2005) looking at placebo-controlled RCTs of ‘homeopathy’ in general, making no distinction between the types of homeopathy being given to patients – four reviews were tentatively positive and one drew a negative conclusion.
Conclusion: Homeopathy is probably different to placebo
Reference Kleijnen et al. 1991.[6]
Size 105 trials identified; results based on 105 trials
Author comment “The evidence presented in this review would probably be sufficient for establishing homeopathy as a regular treatment for certain indications. There is no reason to believe that the influence of publication bias, data massage, bad methodology, and so on is much less in conventional medicine, and the financial interests for regular pharmaceutical companies are many times greater.”
Reference Linde et al. 1997 [7]
Size 186 trials identified; results based on 89 trials.
Author comment “The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition.”
Reference Linde et al. 1999 [8]
Size Re-analysis of same data as the 1997 study, but low quality trials excluded.
Author comment “We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.”
Reference Cucherat et al. 2000 [9]
Size 118 trials identified; results based on 16 trials.
Author comment “There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials.”
Conclusion: Homeopathy is no different to placebo
Reference Shang et al. 2005[10]
Size 110 homeopathy trials identified; results based on 8 trials. 110 conventional medicine trials identified; results based on 6 trials.
Author comment “Biases are present in placebo-controlled trials of both homeopathy and conventional medicine. When account was taken for these biases in the analysis, there was weak evidence for a specific effect of homeopathic remedies, but strong evidence for specific effects of conventional interventions. This finding is compatible with the notion that the clinical effects of homeopathy are placebo effects.”
  Common findings throughout these studies included problems with variable trial quality, variable trial design, publication bias (where there is a tendency for studies with a positive result to be published more readily than those which are negative or inconclusive) and the small number of studies available in total. Despite these limitations, four of the five reviews found that even high quality trials produced positive results in favour of homeopathy. The authors called for more research with a focus on the role of homeopathy in the treatment of specific disease conditions and an emphasis on larger, high quality studies. This has been echoed in a recent review of the systematic reviews of homeopathy, again, published by Mathie et al (11). In addition to academic systematic reviews of homeopathy, there have been a number of organisational and governmental reviews, all of which have been met with intense debate.
  • A 2006 Health Technology Assessment (HTA) report on homeopathy was commissioned by the Federal Social Insurance Office (FSIO) in Switzerland as part of the ‘Complementary Medicine Evaluation Programme’ (PEK) which was set up following a decision by the Swiss government in 1998 to ‘provisionally’ include complementary medical disciplines in the list of services covered by the national statutory health insurance scheme. The English translation of the Swiss Health Technology Assessment (HTA) report on homeopathy was published in December 2011(12). The authors conclude the report by saying: “In summary, it can be said that there is sufficient evidence for the preclinical effectiveness and the clinical efficacy of homeopathy and for its safety and economy compared with conventional treatment.” An overview of the report can be accessed via the Homeopathy Research Institute.
 
  • In November 2009 the Science and Technology Select Committee of the UK Parliament also carried out an “evidence check” into homeopathy. Read the original committee report. While this Select Committee report (referred to as the “EC2”, or Evidence Check 2) has since become notorious and is often quoted as a definitively negative review of the evidence base in homeopathy, it remains outside the accepted scientific literature. The EC2 was not performed as a formal systematic review following robust research processes, unlike the Swiss report which arguably followed a recognized HTA methodology. Additionally, the UK government itself responded to the recommendations outlined in the EC2 report by dismissing them, and a number of MPs rejected the report as unsound. The homeopathic profession also responded to the Select Committee’s EC2 report, thoroughly critiquing it. More detailed information on the issues around this report can be found Overall, the EC2 is rejected by the homeopathic community as being biased and scientifically unsound, yet it continues to be used as a reference point by non-homeopathic organisations for the evidence behind provision of homeopathy in the UK.
 
  • A similar situation has recently arisen in Australia where the National Health and Medical Research Council (NHMRC) published a report on homeopathy in March 2015, concluding that there are no health conditions for which there is sufficient reliable evidence that homeopathy is effective. This negative conclusion is in direct contrast to other academically performed and published systematic reviews of homeopathy in clinical conditions for which homeopathy was found to be beneficial or the results were tentatively positive towards homeopathy(11). Furthermore, the NHMRC report is mired in methodological problems, the most notable of which is that they performed a systematic review of systematic reviews rather than identifying original trials themselves and extracting outcome data afresh. More detailed commentary on the NHMRC report can be found on the HRIand BHA Needless to say, the Australian Homeopathic Association (in collaboration with Complementary Medicines Australis - CMA) has mounted a vigorous response to this negative report. Consequently, in September 2015, the NHMRC agreed to undertake a formal internal review of its processes to address alleged procedural irregularities, bias and methodological flaws.
  References:
  1. Relton C et al. ‘Homeopathy’: Untangling the debate. Homeopathy, 2008; 97: 152-5
  2. Hahn, R.G., 2013. Homeopathy: Meta-Analyses of Pooled Clinical Data. Forshende Komplementarmedizin; 20(5):376-381
  3. http://www.britishhomeopathic.org/our-systematic-review-programme/
  4. Mathie, RT et al. 2013. http://www.ncbi.nlm.nih.gov/pubmed/23290875
  5. Mathie, RT et al. 2014. http://www.ncbi.nlm.nih.gov/pubmed/25480654
  6. Kleijnen J, et al. Clinical trials of homeopathy. Br Med J, 1991; 302: 316–23
  7. Linde K, et al. Are the clinical effects of homoeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet, 1997; 350: 834–43
  8. Linde K, et al. Impact of study quality on outcome in placebo controlled trials of homeopathy. J Clin Epidemiol, 1999; 52: 631–6
  9. Cucherat M, et al. Evidence of clinical efficacy of homeopathy – A meta-analysis of clinical trials. Eur J Clin Pharmacol, 2000; 56: 27–33
  10. Shang A, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet, 2005;366:726–32
  11. Mathie R, et al. Controlled clinical studies of homeopathy. Homeopathy, 2015; http://dx.doi.org/10.1016/j.homp.2015.05.003
  12. Bornhoft G, Matthiessen P (eds), 2011. Homeopathy in Healthcare: Effectiveness, Appropriateness, Safety, Costs. Berlin, Springer-Verlag.
    [post_title] => Systematic reviews [post_excerpt] => [post_status] => publish [comment_status] => closed [ping_status] => closed [post_password] => [post_name] => systematic-reviews [to_ping] => [pinged] => [post_modified] => 2017-07-05 13:19:06 [post_modified_gmt] => 2017-07-05 13:19:06 [post_content_filtered] => [post_parent] => 0 [guid] => http://homeopaths.wpengine.com/?post_type=resources&p=1301 [menu_order] => 0 [post_type] => resources [post_mime_type] => [comment_count] => 0 [filter] => raw )

A systematic review is a summary of the total research evidence available on a particular subject, designed to provide more reliable information than single studies.

Specific methods are used to search academic journals and identify which studies are suitable for inclusion in the review (e.g. those of suitably high quality). From these identified trials a smaller number may be found suitable to go into a meta-analysis. A meta-analysis is a statistical technique used to quantitatively analyse the combined results of multiple studies to generate a more meaningful overall result.

Systematic reviews are intended to draw conclusions from data collected from multiple similar studies, yet the set of trials used in the more famous systematic reviews of homeopathy vary widely in both the type of homeopathic treatment given and the medical conditions being investigated. Particularly, the term ‘homeopathy’ has been interpreted several different ways in the context of homeopathy research1. This means that studies investigating very different approaches are all described as trials of ‘homeopathy’ and are frequently (and inappropriately) analysed together in systematic reviews.

For example, trials pooled for systematic reviews have studied very different aspects of homeopathy making it inappropriate to directly compare them. These different aspects of homeopathy include:

  • A standard single homeopathic medicine given to all participants
  • A standard combination of several homeopathic medicines given to all participants
  • Individualised prescribing without a homeopathic consultation
  • Treatment by a homeopath including individualised prescription and consultation
  • Isopathic trials e.g. homeopathic pollen for treatment of hay fever

For future homeopathy research to be meaningful, specific definitions within the umbrella term of ‘homeopathy’ are needed to promote clarity in the reporting, design and interpretation of homeopathy research, i.e. stating whether a trial is investigating treatment by a homeopath, the action of a homeopathic medicine or the principles of homeopathy2.

A small number of systematic reviews have confirmed that homeopathy can be effective for a small number of specific conditions, but much more research of this kind is needed to investigate a wider range of conditions before we can be sure how effective homeopathy may be under idealised research conditions. As such, the British Homeopathic Association (BHA) has a work programme of systematic reviews in place to provide up-to-date analysis of the clinical evidence base for homeopathy3.

A number of review papers have been published by the BHA team, led by Dr Robert Mathie. Importantly, one of these reviews has fully characterised the clinical trial evidence that we have for ‘homeopathy’ (including all trials up to 2011) so that we may know which published trials are peer-reviewed, placebo controlled, individualised or clinical homeopathy etc.(4).

In addition, Mathie et al published a state-of-the-art, high quality systematic review with an accompanying meta-analysis of randomised placebo-controlled trials of individualised homeopathic treatment across a spectrum of clinical conditions (5). The results of this meta-analysis showed that homeopathic medicines (remedies), when prescribed during individualised treatment, consistent with the usual care provided by homeopaths in a real-world clinical context, are almost twice as likely to have a beneficial effect than placebo. In other words, the most reliable trials of individualised homeopathic treatment show that remedies can have specific and detectable beneficial effects beyond placebo. For a more detailed exploration and explanation of this paper, see the Homeopathic Research Institute (HRI) Autumn 2015 quarterly research article here.

While the recent (2014) meta-analysis of Mathie et al (5) looked at the effects of individualised homeopathic treatment, five major systematic reviews have been carried out previously (dating from 1991 to 2005) looking at placebo-controlled RCTs of ‘homeopathy’ in general, making no distinction between the types of homeopathy being given to patients – four reviews were tentatively positive and one drew a negative conclusion.

Conclusion: Homeopathy is probably different to placebo
Reference Kleijnen et al. 1991.[6]
Size 105 trials identified; results based on 105 trials
Author comment “The evidence presented in this review would probably be sufficient for establishing homeopathy as a regular treatment for certain indications. There is no reason to believe that the influence of publication bias, data massage, bad methodology, and so on is much less in conventional medicine, and the financial interests for regular pharmaceutical companies are many times greater.”
Reference Linde et al. 1997 [7]
Size 186 trials identified; results based on 89 trials.
Author comment “The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition.”
Reference Linde et al. 1999 [8]
Size Re-analysis of same data as the 1997 study, but low quality trials excluded.
Author comment “We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.”
Reference Cucherat et al. 2000 [9]
Size 118 trials identified; results based on 16 trials.
Author comment “There is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials.”
Conclusion: Homeopathy is no different to placebo
Reference Shang et al. 2005[10]
Size 110 homeopathy trials identified; results based on 8 trials. 110 conventional medicine trials identified; results based on 6 trials.
Author comment “Biases are present in placebo-controlled trials of both homeopathy and conventional medicine. When account was taken for these biases in the analysis, there was weak evidence for a specific effect of homeopathic remedies, but strong evidence for specific effects of conventional interventions. This finding is compatible with the notion that the clinical effects of homeopathy are placebo effects.”

 

Common findings throughout these studies included problems with variable trial quality, variable trial design, publication bias (where there is a tendency for studies with a positive result to be published more readily than those which are negative or inconclusive) and the small number of studies available in total. Despite these limitations, four of the five reviews found that even high quality trials produced positive results in favour of homeopathy. The authors called for more research with a focus on the role of homeopathy in the treatment of specific disease conditions and an emphasis on larger, high quality studies. This has been echoed in a recent review of the systematic reviews of homeopathy, again, published by Mathie et al (11).

In addition to academic systematic reviews of homeopathy, there have been a number of organisational and governmental reviews, all of which have been met with intense debate.

  • A 2006 Health Technology Assessment (HTA) report on homeopathy was commissioned by the Federal Social Insurance Office (FSIO) in Switzerland as part of the ‘Complementary Medicine Evaluation Programme’ (PEK) which was set up following a decision by the Swiss government in 1998 to ‘provisionally’ include complementary medical disciplines in the list of services covered by the national statutory health insurance scheme. The English translation of the Swiss Health Technology Assessment (HTA) report on homeopathy was published in December 2011(12). The authors conclude the report by saying: “In summary, it can be said that there is sufficient evidence for the preclinical effectiveness and the clinical efficacy of homeopathy and for its safety and economy compared with conventional treatment.” An overview of the report can be accessed via the Homeopathy Research Institute.

 

  • In November 2009 the Science and Technology Select Committee of the UK Parliament also carried out an “evidence check” into homeopathy. Read the original committee report. While this Select Committee report (referred to as the “EC2”, or Evidence Check 2) has since become notorious and is often quoted as a definitively negative review of the evidence base in homeopathy, it remains outside the accepted scientific literature. The EC2 was not performed as a formal systematic review following robust research processes, unlike the Swiss report which arguably followed a recognized HTA methodology. Additionally, the UK government itself responded to the recommendations outlined in the EC2 report by dismissing them, and a number of MPs rejected the report as unsound. The homeopathic profession also responded to the Select Committee’s EC2 report, thoroughly critiquing it. More detailed information on the issues around this report can be found Overall, the EC2 is rejected by the homeopathic community as being biased and scientifically unsound, yet it continues to be used as a reference point by non-homeopathic organisations for the evidence behind provision of homeopathy in the UK.

 

  • A similar situation has recently arisen in Australia where the National Health and Medical Research Council (NHMRC) published a report on homeopathy in March 2015, concluding that there are no health conditions for which there is sufficient reliable evidence that homeopathy is effective. This negative conclusion is in direct contrast to other academically performed and published systematic reviews of homeopathy in clinical conditions for which homeopathy was found to be beneficial or the results were tentatively positive towards homeopathy(11). Furthermore, the NHMRC report is mired in methodological problems, the most notable of which is that they performed a systematic review of systematic reviews rather than identifying original trials themselves and extracting outcome data afresh. More detailed commentary on the NHMRC report can be found on the HRIand BHA Needless to say, the Australian Homeopathic Association (in collaboration with Complementary Medicines Australis – CMA) has mounted a vigorous response to this negative report. Consequently, in September 2015, the NHMRC agreed to undertake a formal internal review of its processes to address alleged procedural irregularities, bias and methodological flaws.

 

References:

  1. Relton C et al. ‘Homeopathy’: Untangling the debate. Homeopathy, 2008; 97: 152-5
  2. Hahn, R.G., 2013. Homeopathy: Meta-Analyses of Pooled Clinical Data. Forshende Komplementarmedizin; 20(5):376-381
  3. http://www.britishhomeopathic.org/our-systematic-review-programme/
  4. Mathie, RT et al. 2013. http://www.ncbi.nlm.nih.gov/pubmed/23290875
  5. Mathie, RT et al. 2014. http://www.ncbi.nlm.nih.gov/pubmed/25480654
  6. Kleijnen J, et al. Clinical trials of homeopathy. Br Med J, 1991; 302: 316–23
  7. Linde K, et al. Are the clinical effects of homoeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet, 1997; 350: 834–43
  8. Linde K, et al. Impact of study quality on outcome in placebo controlled trials of homeopathy. J Clin Epidemiol, 1999; 52: 631–6
  9. Cucherat M, et al. Evidence of clinical efficacy of homeopathy – A meta-analysis of clinical trials. Eur J Clin Pharmacol, 2000; 56: 27–33
  10. Shang A, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet, 2005;366:726–32
  11. Mathie R, et al. Controlled clinical studies of homeopathy. Homeopathy, 2015; http://dx.doi.org/10.1016/j.homp.2015.05.003
  12. Bornhoft G, Matthiessen P (eds), 2011. Homeopathy in Healthcare: Effectiveness, Appropriateness, Safety, Costs. Berlin, Springer-Verlag.

 

 

Share this page