Understanding Medical Research – Part 1

Background

When someone is first diagnosed with a serious illness like scleroderma, many people turn to the Internet to supplement the information they received from their doctor. The amount of information that patients receive from their doctors varies widely – from just a basic verbal “There is no real treatment but we can deal with many of the symptoms that you will probably develop over time” from most primary care providers to a long discussion with handouts if you are lucky enough to be diagnosed at a scleroderma center.

In many cases, following their diagnosis, patients seek out additional information by searching the Internet. They quickly find a number of sources of up-to-date research-based information on the latest treatments. However, they will also find other sources of information that are much more questionable, especially if they start searching for information using terms like “cure”.

The goal of this three-part series on Understanding Medical Research is to help you to sort through and understand how to “weigh” the many different types of information that you will encounter when searching the Internet. This will also be useful when you read information posted in one of the many online support forums for scleroderma patients that are found on Facebook and other sites. By the time you finish this series, you should have a good understanding of what not to believe, what to believe, and how much to believe, as you weigh the vast quantities information that is quickly and easily available in this modern world of social media and Google searches. You will understand what can be inferred from single patient case studies and research, the limitations of observational studies, and the various types of clinical research studies.

Here is what will be covered in this series:

  1. Testimonials, Anecdotal Reports, Case Studies, and Single-Patient Research: one patient at a time
  2. Observational Studies: lots of people, lots of problems
  3. Clinical Research Studies: from open label pilot studies to the “gold standard” (double-blind randomized controlled trial)

I expect that there will be a number of questions and comments following each of these posts. I will try to address these as quickly and fully as I can. Once the entire series is posted, I will consolidate the information in these posts (updated to address the issues that are raised by posted comments and questions) into a single Technical Article that will be added to the Additional Articles section of this website.

Part 1: Testimonials, Anecdotal Reports, Case Studies, and Single-Patient Research

In this section of this document, we will look at the value of information that can be learned from research and case studies that involve one patient at a time. While you might expect that there is little that can be learned from the experience of a single patients, there are actually some ways that early stage research can be done with individual patient case studies and experiments. But before we look at the varying types of medical case studies and research that can be done with individual patients, it is helpful to first look at other sources of information that patients often encounter that may or may not be meaningful or relevant from a research perspective.

Testimonials

If you do a Google search for “scleroderma treatment”, you will see a number of well-known and respected websites, e.g., the Mayo Clinic, Johns Hopkins, the Scleroderma Research Foundation, that will explain in varying degrees of detail what current research indicates are the most effective treatment options for patients with various forms of scleroderma.   These websites will explain that there is no current effective overall treatment for scleroderma but there are treatments that may help to slow down overall disease progression and other treatments that can help specific symptoms that commonly arise in scleroderma patients. One common characteristic of these websites is that their website address will end in “.org”, indicating that these are non-profit organizations.

If, however, you change the search to “scleroderma cure”, you will instead see a mixture of websites. Some of these are the same websites as above. However, you will also start to see a number of websites that indicate that contrary to what is stated on medical websites, scleroderma can be effectively treated and often cured. There are several common characteristics of these websites:

  • There is always a claim of a very high success rate for the recommended treatment approach. In two actual examples from scleroderma related treatment sites, one claimed a “19 out of 20 success rate” and the other indicated a “90% cure rate”.
  • The website sells the books/machines/supplements that you need to get the benefit of this treatment approach.
  • Typically they will state that these are treatments that the medical establishment doesn’t want you to know about because they would lose money if patients found out about these easier and cheaper ways to treat their disease.
  • The website address ends in “.com”, indicating that they are a commercial site, not a non-profit organization.
  • The website has a number of testimonials from a number of patients that claim that they have benefitted from this treatment.

My general guideline in evaluating medical treatments is to always start with this – if someone is recommending a treatment/device/drug that they are also selling, there is a built-in conflict of interest and you should always be skeptical. There is always concern that patient testimonials may be made up or exaggerated since there is often compensation paid to patients that allow their stories to be used on the website.

Anecdotal Reports

What do I mean by an “Anecdotal Report” and how is it different from a “Testimonial”? Like a testimonial, an anecdotal report is a personal comment by a patient about their personal experience with a particular treatment approach, whether it is a drug, supplement, diet, device, or whatever. In this modern era, many patients with serious medical conditions turn to online support communities where they can receive emotional support and in some cases educational information from trained medical professionals or knowledgeable patients. In addition, patients will frequently ask about other patients experience with a particular drug or treatment. When patients post about their experiences with a drug or a medical treatment, this is an example of an anecdotal report.

So how is a comment from a patient on a patient support forum different from a patient testimonial on a commercial website?   The difference is that there is no (obvious) conflict of interest or secondary gain on the part of the patient who is writing about his or her personal experience with the drug or other treatment approach. For this reason, an anecdotal report from a patient in a setting like a support forum is more likely to at least reflect a genuine patient experience than a testimonial where the patient may be getting compensated in some way in exchange for the website being allowed to include the testimonial.

But this still doesn’t address the bigger question: just because a patient reports feeling better after taking a drug or undergoing a treatment, does this mean that the drug or treatment actually was the reason the patient felt better? It turns out that determining whether or not a drug or treatment is effective is actually much more difficult than you might expect, for reasons that are discussed in the next section.

Case Studies

As we move on from the least credible type of information (testimonials) to the ultimate destination of the “gold standard” double-blind randomized controlled trial, the next stop is the medical case study or case report. Basically, a medical case study is a description of a patient experience in a “natural setting,” often done for instructional purposes. While a case study can involve an intervention (treatment, drug, device), in many cases it is instead a description of a patient’s background, medical history, current symptoms, etc. In the world of medicine, case studies are presented in “grand rounds” as teaching examples, or written up by physicians or other trained medical personnel and published in research journals.

Case studies include a description of the patient’s background and symptoms, often including objective data such as laboratory test results or observable clinical symptoms. This adds a great deal of credibility to the information that is being documented in the case report. Physicians that stay current in their field by reading research journals often turn to case studies for ideas when treating patients with challenging problems that are not effectively addressed by standard treatment approaches.

However, is important to understand that a typical case study is not (yet) an example of formal clinical research, even if there is a treatment involved. Most case studies are observational and written after the fact even if it is documenting a treatment that the clinician tried that appears to be effective. Observational case studies are often the basis for future formal research. For example, if a clinician tries an intervention that appears to be beneficial, this may cause a researcher to form a hypothesis that s/he can test formally in the future. But the key point about a case study as compared to a more casual anecdotal report is that because it is written up formally and normally needs to meet the standards of a research journal in order to be publishable, it gives a lot more useful and validated information that can be used by other clinicians or as the basis for more formal research studies.

Single-Subject Clinical Research

The Post Hoc Ergo Propter Hoc Problem

While there is a lot of potential difference in credibility, all three examples that have been discussed so far – testimonials, anecdotal reports, and case studies – have one thing in common. With the exception of case studies that are done just to illustrate the natural progression of a disease, the basic information presented in all of three of these examples generally takes the form:

  • A patient has a particular medical problem (cancer, diabetes, scleroderma) and currently has symptoms x, y, and z
  • An intervention of some type (treatment/drug/procedure) was done to try to help improve some of the patient’s symptoms
  • After the intervention, the patient’s symptoms changed, usually for the better. (Some medical case studies do report interventions that appear to make the patient’s symptoms worse but that is not very common.)

What is implied by reports like these is that the patient’s symptoms improved because of the intervention. But in reality that may or may not be true. This is a very important point so we will look at the reasons for this in a bit of detail.

Humans are extremely good at reaching conclusions, which is probably a good thing. If you were an early human 50,000 years ago in a grassy meadow and saw the tall grass moving nearby because there was a saber tooth tiger, you might quickly decide that whenever you saw the tall grass moving in a meadow it was probably a good idea to flee the meadow to safer ground. The early humans who reached the conclusion that moving tall grass might be very dangerous probably lived longer than those that decided it was just a friendly gazelle. So the ability to make a cause and effect connection between moving grass and danger was clearly a good survival skill.

The medical analog to this is to assume that if a patient has symptoms A and we try treatment B and the patient gets better, then we naturally assume that treatment B caused the patient to get better. And that may in fact be the case, but sometimes (mixing metaphors) it was just a gazelle in the grass after all.

In formal logic, this is known as the post hoc, ergo propter hoc (after this, therefore because of this) fallacy. In other words, the assumption is that because B follows A, A causes B.

Here are some examples:

  • Example 1: A rooster crows, then the sun comes up. Therefore the rooster’s crowing causes the sun to come up.
  • Example 2: The patient has a bad cold and goes to her doctor. The doctor gave her a prescription for an antibiotic. Two days later she feels better. Therefore, she reaches the logical (but incorrect) conclusion that antibiotics cure colds.
  • Example 3 (200 years ago): The patient complains of dizziness, excessive sweating, shortness of breath, and painful swelling of his big toe. The healer decides to do bloodletting to let the bad humors out and the patient feels better a few days later. Therefore the healer assumes that bleeding the patient helped him to feel better. (Oops – bad example. It turns out that the patient had a rare genetic disease called Polycythemia Vera that causes the body to produce excess red blood cells, resulting in the symptoms listed above. One of the modern treatments is actually bloodletting!)

The point of these examples is simple: just because a patient feels better after a treatment of some kind, it does not necessarily mean that the treatment is the reason for the improvement. It may in fact actually be the reason, but it is very important that this potential cause-effect relationship is proven scientifically and not just assumed to be true just because it makes intuitive sense.

In the last section of this article, we will discuss how researchers design clinical studies that can rigorously test drugs and other potential treatments. These studies can be expensive, time consuming, and difficult to do, especially with rare diseases where finding enough suitable patients can be very difficult. However, it turns out that even with a single patient, there sometimes is a way that researchers can determine that an intervention of some kind (drug, treatment, device) is very likely to be effective for this particular patient. This point is very important – even if you do appropriate single-subject research to establish a likely cause and effect relationship, you cannot generalize from this single patient to all patients with similar conditions and symptoms. It might turn out that this particular patient has a very rare genetic mutation and for anyone else who does not have this genetic mutation, the treatment might be dangerous. But when a single-subject study is done properly, it does serve as a reasonable basis for trying this intervention with a larger group of patients as a next step. It also may prompt individual clinicians to try this intervention if they are not able to offer an alternative effective treatment, as is often the case with many diseases.

Single-Subject Research Design

Entire books have been written on the subject of research designs, including single-subject designs, so we will spend very little time on this topic. However, there is a single-subject research design that occasionally is seen in the field of medicine that is considered the “gold standard” research design for single-subject studies. There are practical reasons why it is rarely seen in medicine (in contrast to psychology or social science research), which we will touch on in a moment, but it is worth knowing about for those rare occasions when it is used.

Let’s start by looking at some terminology that is used when discussing research design in single-patient research. All of the testimonials, anecdotal reports, and case studies we have been talking about fall into a research design category called the “A-B” design. A is the baseline that represents the patient’s medical condition and symptoms before the treatment/medication is tried. B represents the patient’s medical condition later in time after the patient has been receiving the treatment/medication. For the reasons noted above, this research design is suggestive that the treatment was the reason for the change in the patient’s medical condition, but not conclusive.

In order to strengthen the evidence that there is, in fact, a cause and effect relationship between the treatment and the change in the patient’s medical condition, we must move to a more complicated design called an “A-B-A” design. In this single-subject research design, the treatment is removed or stopped after the initial effect is seen. If the patient’s symptoms then gradually return or start to return to the baseline condition, then this greatly increases the likelihood that the intervention did in fact cause the change in symptoms in the first place.

Obviously there are a lot of practical and ethical problems in stopping a treatment that appears to be helping the patient, which is one of the reasons this is not very commonly done in a medical setting. Also, in some cases, you can’t reverse a treatment once it is done, for example, if you were to want to test a treatment that makes changes to a patient’s DNA. But there are situations that naturally produce this research design. For example, many immunosuppressant medications that are given to patients to help control symptoms in autoimmune diseases like scleroderma or lupus are so toxic that patients cannot take these drugs for more than a year. In this situation, if symptoms improve at the end of the one-year treatment period but later return to baseline, this is a natural example of this research design and demonstrates probable cause and effect, i.e., the treatment probably did cause the change in symptoms.

But the “gold standard” of single-subject research design takes things one step further. It is called the “A-B-A-B Reversal Design”. This is a fancy name, but all it really means is that after you withdraw the treatment and see the patient’s symptoms begin to return, you again re-apply the treatment. If you again see the symptoms improve, you have now established with a great deal of certainty that the treatment is in fact the reason that the patient’s symptoms improved –not just chance. Continuing our example above where the patient’s symptoms returned after stopping the immunosuppressant treatment, if you then put the patient back on immunosuppressants and see symptom improvement again, you have done an A-B-A-B Reversal Design experiment. This establishes with a very high degree of probability that the treatment is the reason for the symptom improvement for this particular patient.

Summary – What Can You Learn From One Patient?

The best way to evaluate the effectiveness of a treatment is by doing formal clinical research, as will be discussed later in this article. However, there is definitely information that can be learned from reports from or about individual patients, as long as you understand the limitations of this type of information:

  • Patient testimonials found on websites that are selling products or services are not a good source of information for a variety of reasons and should generally not be considered useful or reliable.
  • Anecdotal reports from individual patients, as are commonly found on patient support groups on Facebook or other sites, can be a good place for patients to learn about what other patients have actually experienced with a drug or treatment they are considering.   Anecdotal reports can also be useful to researchers and clinicians as sources of ideas for future research, but not as a reliable source of information that can or should be used clinically in most cases.
  • Formal case studies can definitely be useful sources of information for researchers and clinicians as long as they understand the limitations. In particular, there are two issues with individual patient case studies that need to be carefully considered:
    • You cannot establish a formal cause and effect relationship between a treatment and a change in the patient’s symptoms if all that is done is to try the treatment and observe that the patient’s symptoms changed. In order to establish (with a high degree of certainty) that the treatment causes the symptom change, a more complex study design is needed that may not be feasible for clinical or ethical reasons.
    • Even when a single-patient study is done using a more complex research design, the results cannot be generalized beyond the individual patient, since the patient may have unique individual characteristics that may not be present in other patients. However, a well-designed single-patient study does provide good evidence that the intervention may have a beneficial effect, adding support for clinicians trying the intervention with other patients or researchers moving to a full-blown clinical research study with multiple patients.

Coming up in Part 2: Long-Term Observational Studies

Share

One Response to Understanding Medical Research – Part 1