What Can You Trust?

Key Points

  • The burden of proof of a therapy or approach can vary according to the relative risks and benefits.
  • Although some study designs are considered stronger than others, interpreting study results involves assessing the trade-offs between highly controlled situations and relevance to real life, then using the evidence that makes the most sense in the situation.
  • Therapies that are not especially dangerous and that have credible evidence that they may be helpful do not need as much proof of benefit as therapies that involve more risk or expense.
  • When assessing the reliability of a book, website or other resource, consider the authors' qualifications and financial ties,
  • BCCT recommends checking sources and validating information obtained from sources with any financial stake in therapies.

Search the Internet for “cancer therapy” and you will find tens of millions of webpages. How can you sort through to find the few pages that will be most valuable for you? How can you possibly know which websites or information to trust? How do you analyze and find clarity when “experts” give opposite recommendations? How can you tell the “snake oil” from truly valuable therapies?

First, please understand that BCCT does not claim to have all the answers. As we pull together our information and summaries, we struggle with these same issues: Is this source reliable? Is this research valid? Is this expert qualified to speak about this therapy?

However, even as we are always learning ourselves, we share with you some guidelines for determining whether a website or article rates higher or lower on a trust scale.

Nancy Hepp

Type and Strength of Evidence

Evidence Trade-offs

Evidence that a therapy “works” runs a whole range from completely unreliable to trustworthy. We present an overview of some of the issues in determining whether evidence is reliable and appropriate, first from a researcher's viewpoint, and then from a clinician's frame.

The Researcher's View: Hierarchy of Evidence

We present 12 levels in a research hierarchy of strength of evidence, starting with unreliable evidence and working down through increasingly credible sources of evidence (all examples are fictional). The evidence toward the bottom of this list (larger numbers) is generally regarded as stronger and more reliable than evidence toward the top of the list.

1. Nonspecific testimonials

Read more

2. Specific testimonials

Read more

3. Clinical or expert testimonial

Read more

4. Clinical observations, also called case studies

Read more

5. Retrospective observational studies

Read more

6. Prospective observational studies

Read more

7. Experimental studies with animals

Read more

8. Small prospective, experimental clinical studies, sometimes called “pilot studies”

Blinded Trials

The practice of “blinding” trials of treatments—so that patients are not aware of which treatment they receive—is held up as a hallmark of rigorous research. Blinding the physicians or practitioners to the treatment (a “double-blind study”) is considered even better.1

Read more

Read more

9. Large prospective, experimental clinical studies

Read more

10. Review of groups of studies

Read more

11. Meta-analysis of several studies

Read more

12. Clinical practice guidelines

Read more

Observational studies that come closer to real-life situations may provide more valuable information for clinical use than studies that are considered the “gold standard.”

The Clinician's View: Therapies in the Real World

Progression-free Survival

In assessing the effectiveness of cancer treatments, the outcome being measured can make a big difference in interpretation. For example, studies reporting only progression-free survival (PFS) may find effectiveness, yet many patients do not perceive much benefit.

Read more

The Gold Standard: Why Randomized Controlled Trials Don't Always Tell the Real-World Story

Many oncologists rely heavily on the outcomes of "gold standard" randomized clinical trials (RCTs) in recommending therapies to patients. Some researchers tell a more complex story. RCTs try to control all variables, but in the real world, outcomes may be different. That's why observational studies—what clinicians see in practice—actually matter.

Read more

The Evidence House: Valuing What the Physician Sees in Practice

BCCT advisor Wayne Jonas, MD, explains that the strongest study designs don’t always provide the best evidence:9

As most clinicians know, the reasons that patients recover from illness are complex and synergistic, and many cannot simply be isolated in controlled environments. The best evidence under these circumstances may be observational data from clinical practice that can estimate the likelihood of a patient's recovery in a realistic context.

Read more

Evidence House graphic
Evidence House illustration; click to open the full journal article

Jonas has proposed an “evidence house” with different “rooms” and different “wings” for different audiences and purposes:

  • Rooms in one wing of the house contain types of scientific information such as laboratory research. These rooms seek to find causes of disease, how therapies work, and proof of effectiveness—types of information that can be difficult to determine or that may take many decades.
  • Another wing has rooms with information about therapies’ relevance and usefulness in clinical practice rather than absolute proof of effectiveness.

Read more

"The Lower the Risk of Harm, the Lower the Burden of Proof"

Experimental therapies both in mainstream cancer medicine and in complementary medicine can be considered science-informed rather than fully science-based. Because the scientific process can be slow to accumulate enough evidence to be conclusive, and because cancer patients often don’t have decades to wait for rigorous research results, science-informed therapies are often the only option beyond standard therapies. These might include therapies supported by case studies and observational studies, or for which a strong theoretical rationale exists but empirical studies are unavailable, incomplete or inconclusive.

Therapies that are not especially dangerous and that have credible evidence that they may be helpful do not need as much proof of benefit as therapies that involve more risk or expense.

BCCT advisor Dr. Donald Abrams makes another critical point with respect to science-informed therapies: The lower the risk of harm, the lower the burden of proof. The burden of proof is lower if a therapy meets these criteria:

  1. It is unlikely to do harm.
  2. The patient considers it affordable.
  3. The patient is drawn to it or believes it may have value.

Therapies that are not especially dangerous and that have credible evidence that they may be helpful do not need as much proof of benefit as therapies that involve more risk or expense. BCCT views the use of science-informed, low-risk, affordable therapies as a reasonable option for patients. Stronger evidence of benefit is needed for therapies that are risky, expensive or otherwise burdensome.

Financial Ties

Websites or people who are trying to sell something have an incentive to make their product look the best that it can. Even when sales people have honorable intentions, benefits can be unconsciously promoted and potential harm downplayed. However, just because a website sells products doesn’t mean their information isn’t valid. BCCT recommends that you check the information on a site against other highly credible sources, which we describe below.

Websites or people who are trying to sell something have an incentive to make their product look the best that it can.

Read more

Author Qualifications

Red Flags

A few “red flags” cause us to question the value and validity of some sources:

  • Authors:
    • Authors and sponsors are not identified.
    • Conflicts of interest (such as an author’s or “expert’s” ties to manufacturers or organizations) are hidden.
  • Claims:
    • A site or person claims that one therapy (likely a therapy for purchase on the site) can cure cancer.
    • A site or person purports to have connections to God or spiritual forces that users must pay to access.
    • A site or person is excessively critical of approaches different from theirs or pushes their therapy to the exclusion of others.
  • Evidence:
    • Testimonials from users or sellers are the only source of evidence.
    • No references or details are provided for unnamed studies purporting to prove a therapy is effective.
    • Studies cited are not published in reputable, peer-reviewed journals. Some “pay to publish” journals will publish anything from authors, regardless of its scientific merit.
    • A person or site dismisses those who criticize their product or service without any evidence.
    • Information is not dated.

Sites with these characteristics, and especially more than one of these, should be treated with a great deal of caution. BCCT recommends validating any information from these sites through more transparent or authoritative sources.

The qualifications of authors should be described. Formal education and training, experience and independent investigation are all valid qualifications. These should be made available to the reader. Any conflicts of interest, whether financial or organizational, should be listed for authors.

BCCT’s Approach to Information

When we evaluate claims regarding therapies and treatments, BCCT strives to consider both experts’ financial interests and where they draw evidence from. We indicate in our footnotes where our information comes from and provide a link if possible so that you can check the source yourself. We look for the most credible sources available.

Clues to be skeptical and to verify any information through more reliable sources:

  • Is this person or website encouraging you to buy something?
  • Does this person or site make only vague statements about effectiveness without any evidence?
  • Do you have to pay for a product or therapy before you can receive specific information about it?

Sources We Trust

Our Resources collection includes many books, websites, videos and other resources that we have found to be trustworthy. However, we are open to critiques from our users of these resources. We are deeply grateful to our users for alerting us about resources that should be reconsidered or removed.

Written by Nancy Hepp, MS and reviewed by Laura Pole, RN, MSN, OCNS, and Michael Lerner, PhD; most recent update on October 23, 2018.

View All References

More Information

Enter your comments or questions below.

Comments (0)




Allowed tags: <b><i><br>Add a new comment: