What Can You Trust?
Search the Internet for “cancer therapy” and you will find tens of millions of webpages. How can you sort through to find the few pages that will be most valuable for you? How can you possibly know which websites or information to trust? How do you analyze and find clarity when “experts” give opposite recommendations? How can you tell the “snake oil” from truly valuable therapies?
First, please understand that BCCT does not claim to have all the answers. As we pull together our information and summaries, we struggle with these same issues: Is this source reliable? Is this research valid? Is this expert qualified to speak about this therapy?
However, even as we are always learning ourselves, we share with you some guidelines for determining whether a website or article rates higher or lower on a trust scale.
Type and Strength of Evidence
Evidence that a therapy “works” runs a whole range from completely unreliable to trustworthy. We present an overview of some of the issues in determining whether evidence is reliable and appropriate, first from a researcher's viewpoint, and then from a clinician's frame.
The Researcher's View: Hierarchy of Evidence
We present 12 levels in a research hierarchy of strength of evidence, starting with unreliable evidence and working down through increasingly credible sources of evidence (all examples are fictional). The evidence toward the bottom of this list (larger numbers) is generally regarded as stronger and more reliable than evidence toward the top of the list.
1. Nonspecific testimonials
2. Specific testimonials
3. Clinical or expert testimonial
4. Clinical observations, also called case studies
5. Retrospective observational studies
6. Prospective observational studies
7. Experimental studies with animals
8. Small prospective, experimental clinical studies, sometimes called “pilot studies”
The practice of “blinding” trials of treatments—so that patients are not aware of which treatment they receive—is held up as a hallmark of rigorous research. Blinding the physicians or practitioners to the treatment (a “double-blind study”) is considered even better.1
9. Large prospective, experimental clinical studies
10. Review of groups of studies
11. Meta-analysis of several studies
12. Clinical practice guidelines
Observational studies that come closer to real-life situations may provide more valuable information for clinical use than studies that are considered the “gold standard.”
The Clinician's View: Therapies in the Real World
In assessing the effectiveness of cancer treatments, the outcome being measured can make a big difference in interpretation. For example, studies reporting only progression-free survival (PFS) may find effectiveness, yet many patients do not perceive much benefit.
The Gold Standard: Why Randomized Controlled Trials Don't Always Tell the Real-World Story
Many oncologists rely heavily on the outcomes of "gold standard" randomized clinical trials (RCTs) in recommending therapies to patients. Some researchers tell a more complex story. RCTs try to control all variables, but in the real world, outcomes may be different. That's why observational studies—what clinicians see in practice—actually matter.
The Evidence House: Valuing What the Physician Sees in Practice
BCCT advisor Wayne Jonas, MD, explains that the strongest study designs don’t always provide the best evidence:9
As most clinicians know, the reasons that patients recover from illness are complex and synergistic, and many cannot simply be isolated in controlled environments. The best evidence under these circumstances may be observational data from clinical practice that can estimate the likelihood of a patient's recovery in a realistic context.
Jonas has proposed an “evidence house” with different “rooms” and different “wings” for different audiences and purposes:
- Rooms in one wing of the house contain types of scientific information such as laboratory research. These rooms seek to find causes of disease, how therapies work, and proof of effectiveness—types of information that can be difficult to determine or that may take many decades.
- Another wing has rooms with information about therapies’ relevance and usefulness in clinical practice rather than absolute proof of effectiveness.
"The Lower the Risk of Harm, the Lower the Burden of Proof"
Experimental therapies both in mainstream cancer medicine and in complementary medicine can be considered science-informed rather than fully science-based. Because the scientific process can be slow to accumulate enough evidence to be conclusive, and because cancer patients often don’t have decades to wait for rigorous research results, science-informed therapies are often the only option beyond standard therapies. These might include therapies supported by case studies and observational studies, or for which a strong theoretical rationale exists but empirical studies are unavailable, incomplete or inconclusive.
Therapies that are not especially dangerous and that have credible evidence that they may be helpful do not need as much proof of benefit as therapies that involve more risk or expense.
BCCT advisor Dr. Donald Abrams makes another critical point with respect to science-informed therapies: The lower the risk of harm, the lower the burden of proof. The burden of proof is lower if a therapy meets these criteria:
- It is unlikely to do harm.
- The patient considers it affordable.
- The patient is drawn to it or believes it may have value.
Therapies that are not especially dangerous and that have credible evidence that they may be helpful do not need as much proof of benefit as therapies that involve more risk or expense. BCCT views the use of science-informed, low-risk, affordable therapies as a reasonable option for patients. Stronger evidence of benefit is needed for therapies that are risky, expensive or otherwise burdensome.
Websites or people who are trying to sell something have an incentive to make their product look the best that it can. Even when sales people have honorable intentions, benefits can be unconsciously promoted and potential harm downplayed. However, just because a website sells products doesn’t mean their information isn’t valid. BCCT recommends that you check the information on a site against other highly credible sources, which we describe below.
Websites or people who are trying to sell something have an incentive to make their product look the best that it can.
A few “red flags” cause us to question the value and validity of some sources:
Sites with these characteristics, and especially more than one of these, should be treated with a great deal of caution. BCCT recommends validating any information from these sites through more transparent or authoritative sources.
The qualifications of authors should be described. Formal education and training, experience and independent investigation are all valid qualifications. These should be made available to the reader. Any conflicts of interest, whether financial or organizational, should be listed for authors.
BCCT’s Approach to Information
When we evaluate claims regarding therapies and treatments, BCCT strives to consider both experts’ financial interests and where they draw evidence from. We indicate in our footnotes where our information comes from and provide a link if possible so that you can check the source yourself. We look for the most credible sources available.
Clues to be skeptical and to verify any information through more reliable sources:
- Is this person or website encouraging you to buy something?
- Does this person or site make only vague statements about effectiveness without any evidence?
- Do you have to pay for a product or therapy before you can receive specific information about it?
Sources We Trust
Our Resources collection includes many books, websites, videos and other resources that we have found to be trustworthy. However, we are open to critiques from our users of these resources. We are deeply grateful to our users for alerting us about resources that should be reconsidered or removed.
Written by Nancy Hepp, MS and reviewed by Laura Pole, RN, MSN, OCNS, and Michael Lerner, PhD; most recent update on October 23, 2018.
- Society for Integrative Oncology: Don't Believe Everything You Read on the Internet
- National Cancer Institute: Using Trusted Resources
- Williams JT. Credible complementary and alternative medicine websites. Journal of the Advanced Practitioner in Oncology. 2013 Mar-Apr; 4(2): 123–124.
- American College of Healthcare Sciences: How to Find Credible Complementary Alternative Medicine Cancer Resources
- Cancer Research UK: Information on the web
- American Cancer Society: Cancer Information on the Internet
- Jonas WB. The evidence house: how to build an inclusive base for complementary medicine. Western Journal of Medicine. 2001 Aug;175(2):79-80.
- Dr. Nagourney's Blog: Cancer patients want to win, but is the playing field level?
- Burns PB1, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plastic and Reconstrive Surgery. 2011 Jul;128(1):305-10.
- Editors: Iris F. F. Benzie and Sissi Wachtel-Galor: Herbal Medicine, 2nd Edition: Biomolecular and Clinical Aspects
- Donald I. Abrams, MD, and Andrew T. Weil, MD: Integrative Oncology, 2nd Edition
- American Society of Clinical Oncology: Cancer.Net