Course 8: The Challenges and Failures of Nutrition Studies

Introduction

Nutrition studies tend to disproportionately attract media attention and catch the public’s eye when compared to other scientific studies. Nutrition studies often claim that a specific food or diet will cause or prevent a certain disease. Though they receive a lot of publicity, their conclusions and results must not be taken for granted and must be scrutinized. Nutrition studies often employ methodologies that lead to inaccurate responses and involve confounding variables which complicate results. “Understanding the limitations of dietary assessment techniques and the quantification of the errors involved has been handicapped for decades by a lack of independent methods for validation. Only for the 24hr recall technique or for individual meals (where direct observations are possible) have any true studies of validity been undertaken. In early studies in which a subject’s intake was covertly observed and then assessed by recall, intake was usually underestimated… Nevertheless until recently, dietary intakes were reported as if valid, and the interpretation of links between intake and health were based, often erroneously, on the assumption of validity.”(1) Since nutrition studies often employ methodologies that have inherent biases, it is important to take these limitations into account when analyzing such studies.

Inaccurate Methodology

One of the main problems with nutrition studies is that they often ask study participants to recall what they have eaten over a certain period of time. It is unlikely that a person would be able to precisely recall how much of a certain food s/he has consumed during the past week, month, or year.  Nevertheless, studies attempt to relate past self-recalled nutrition data to medical outcomes and health conditions. For example, a study on diet and lens opacity concluded that “in men, after controlling for age, smoking, and heavy drinking, intakes of numerous nutrients in the highest versus lowest quintile were associated with 40-50 % reduced odds of more severe nuclear sclerosis.”(2) Though this might seem like a reasonable conclusion, upon examining the methodology of this study, the results appear much less reliable. The study asked participants about their usual diets over a one-year period of time, 10 years before the interview.  They first asked participants “a series of questions about lifestyle during that time… to help the participant orient to that past time period.” (3) The researchers then used a food frequency questionnaire with a 99-item list and asked participants to list the frequency of consumption of food on the list as well as food not on the list. “For each food, the respondent was asked how frequently it was eaten and whether the usual portion was small, medium, or large.” (4) This data was subsequently used to calculate each person’s vitamin intake to see if there was a relationship between diet and development of nuclear opacities. The authors concluded that “the intake of vitamin supplements (in men and women) and certain foods (particularly in men) may explain associations of several nutrients with risk for nuclear sclerosis."(5) This methodology likely leads to inaccurate results and the authors’ conclusion may therefore be invalid.  If you were asked how many fruit and vegetables you ate each day ten years ago as well as the portion size, your recall and self-reporting would likely be inaccurate. Though the study attempts to orient people to recall the food that they had eaten 10 years ago, it is still improbable that people would be able to recall any specifics regarding their food intake. Unless someone holds themselves to a strict diet in which they require themselves to eat X numbers of fruits and vegetables per day, it is exceedingly rare for someone to accurately be able to recall their daily fruit and vegetable consumption ten years ago, or even one week ago.  In addition, it is not just the average person who has a hard time recalling what s/he has eaten. A study on food intake measurement in scientists found that “very few subjects were able to accurately remember the types and amounts of food they had consumed in the previous 24 hours.” (6) If scientists are unable to remember what they have eaten in the past 24 hours, it is unreasonable to assume that a person would be able to accurately remember what s/he consumed 10 years ago. This is just one example of the very ineffective methodology that tends to be involved with nutrition studies.  The research validity article further explains the common lack of validity in published research studies.

Self-Report Bias

Nutrition studies tend to also be affected by self-report bias. Self-report bias is when research participants respond in a way that makes them look as good as possible. Thus, participants tend to under-report behaviors deemed inappropriate by researchers or other observers, and over-report behaviors viewed as appropriate.(7) A comprehensive review of studies in which energy intake (EI) was reported and measured found that there is a widespread bias for people to underestimate their EI.  In addition, those who underestimated their energy requirements had a general tendency to “report more ‘good’ foods such as meat, fish, vegetables, salad and fruit and less ‘bad’ foods such as cakes, cookies, sugar, candies and fats.”(8)

It is critical to take self-report bias into account when evaluating nutrition studies which focus on weight loss. More than 300 million American are obese and often seek medical evaluation for failure to lose weight despite following calorie-restricted diets. A study on obese subjects who reportedly restricted their intake to less than 1200 kcal per day, but still didn’t lose weight, found that the energy intake reported by the participants was 1028±148 kcal per day, whereas their actual energy intake was 2081±522 kcal per day. This corresponds to a mean difference for individual subjects of 47±16%. Therefore, these subjects severely underreported their energy intake. Self-recall bias also affected the participants’ perception of their exercise habits. On average, the subjects reported the amount of energy they expended in physical activities as being 1022±185kcal per day, whereas the actual energy expended was 771±264 kcal per day. (The control group which consisted of obese people who had had success losing weight on calorie restricted diets also underreported their caloric intake and over reported their physical activity but to a lesser extent.)(9) Thus, the authors concluded that “subjects presenting with an inability to lose weight despite a history of caloric restriction may represent a subgroup of obese people who severely misreport their food intake and level of physical activity.”(10) A study on fiber intake and weight loss also came to a similar conclusion and found that “energy intake is typically underreported by 10% to 50% on average, with overweight and obese individuals underreporting to a greater extent than normal-weight individuals.”(11) Therefore, when analyzing the results of nutrition studies, which rely on self-reported data, it is important to realize that the data will likely not accurately reflect the participants’ actual food intake or energy expended from exercise.

Ensuring the Accuracy and Validity of Nutrition Studies

In order to ensure the accuracy and validity of nutrition studies, each study’s methodology must be analyzed. As mentioned previously, studies which ask participants to recall what they have eaten are often affected by self-recall bias. Similarly, studies which ask participants to remember what they ate years, months or even weeks ago are also inaccurate. Thus, in order to avoid these biases, it is best for nutrition studies which use participant recall of consumption to have subjects remember no further back than the previous 24 hours. One of the main methods used to ensure accurate and valid results for nutrition studies is the USDA Automated Multiple-Pass Method (AMPM). The AMPM is a computerized method for collecting interview-administered 24-hour dietary recalls. It employs 5 steps to enhance complete and accurate food recall. It is used with a companion food model booklet which aids the study participant to more accurately estimate portion sizes.(12)

The 5 steps include:

1. Quick List: collect a list of food and beverages consumed the previous day

2. Forgotten Foods: probe for foods forgotten during the Quick List.

3. Time and Occasion: collect time and eating occasion for each food.

4. Detail Cycle: for each food, collect detailed description, amount, and additions. Review 24-hour day.

5. Final Probe: final probe for anything else consumed.

The USDA five-step multiple pass method has been found to be more accurate than the food frequency questionnaire previously mentioned. A study on middle-aged men in Washington D.C. found that “there were no significant differences between actual and recalled intakes of energy, protein, carbohydrate, or fat, respectively. Accuracy of recall was not related to body mass index in that the obese men recalled food intake as accurately as the nonobese men.” (13) Similarly, a study on nutrient intake in pre-menopausal women found that “AMPM dietary recall accurately measured mean total energy and absolute nutrient intakes at the group level, whereas these dietary measures were significantly underestimated by the food-frequency questionnaires.” (14)

Confounding Variables

Nutrition studies are often influenced by confounding variables which may cause a researcher to find a correlation between two variables when there is not actually a correlation. For example, confounding variables have affected studies relating antioxidant intake to risk for age-related cataract and maculopathy. A review of the literature on nutritional antioxidants and age-related cataract and maculopathy found over 70 studies that have attempted to relate antioxidant intake to risk for age-related cataract and maculopathy.  The review found that “while data from the observational studies generally support a protective role for antioxidants in foods or supplements, results from intervention trials are less encouraging with respect to limiting risk for ARC/ARM prevalence or progress through antioxidant supplementations… Without more information, it is difficult to parse these results.”(15) It is also difficult to draw conclusions from these results since there are various risk factors for age-related cataract. Oxidative stress resulting from sunlight exposure and smoking are just a few of such risk factors. Thus, rates of age-related cataract are likely not only affected by the quantity of antioxidants consumed.

Similarly, other studies have attempted to relate healthy diets to the prevalence of nuclear cataract. A study on women found that “adherence to the 1990 dietary guidelines for Americans, as reflected by Health Eating Index (HEI) scores for diets over a 10-year period, was associated with lower risk for early nuclear lens opacities” and that “having a high 1995 Health Eating Index score was the strongest modifiable predictor of low prevalence of nuclear cataract.”(16) Though women who had the highest Health Eating Index scores had lower risk of developing nuclear cataract, they also had “characteristics that are often associated with Americans who adopt healthy lifestyles: being more likely to take supplements, have lower body fat (as reflected by BMI), and have higher levels of physical activity and being less likely to smoke.” (17) Thus, it is likely that other environmental factors may have influenced or caused the observed association that women with a high Health Eating Index score have a lower risk for developing nuclear cataract.

The Media and Flip-Flopping

The media will often declare that a certain food or supplement is great for you, just to refute it a short time afterwards. “Unfortunately, it’s often hard to get a straight answer. One day the ‘experts’ say one thing. The next, they seem to say another. Then it all appears to switch back again. Such flip-flops can be maddening, especially when you’re making your best effort to live a healthy lifestyle.” (18) For example, when researchers stated three decades ago that saturated fat was problematic for heart health, people replaced butter in their diets with margarine. Recent research, however, has found that since margarine contains trans fat, it is less heart healthy than butter, which contains saturated fat.(19) This example, among many others, reflects the fact that nutrition science is a multi-faceted field which is constantly advancing and changing.

However, it is critical to realize that the media often only reports studies which refute previous findings on a given topic in order to attract attention. (20) “When a headline touts a shocking new ‘answer’ to a nutrition question, read the story with a critical eye. It may indeed be a carefully researched report, but often it is a sensational story intended to catch the attention of newspaper and magazine buyers, not to offer useful nutrition information.”(21) Since this phenomenon is unfortunately so pervasive, it is important to critically analyze nutrition studies being reported by the media. In order to do so, one must take into consideration and evaluate if the media is reporting on only one study or many. If only one study is mentioned, the study was likely only reported because its results ran contrary to all previous studies conducted.  Thus, “it bears repeating that the findings of a single study never prove or disprove anything. Study results may constitute strong supporting evidence for one view or another, but they rarely merit the sort of finality implied by journalistic phrases such as 'now we know' or 'the answer has been found.'” (22) It is also important to analyze the study’s methodology and ensure that it is unbiased, and look at the sample size. A larger study with a diverse participant population is likely to be more accurate than a study that looks at a small homogenous group. (23)

Conclusion

Though nutrition studies have the potential to add important perspectives to public health, they are fraught with many pitfalls. Nutrition studies often use inaccurate methodology and suffer from self-recall bias. In addition, nutrition studies tend to be sensationalized in the media. It is therefore important to scrutinize a study’s methodology and determine if similar studies have been done in order to ensure their validity and accuracy.  

Footnotes

(1) Livingstone, B., and Black, A. “Markers of the Validity of Reported Energy Intake.” J. Nutr. 133. 3 (2003).

(2) Mares-Perlman, J., Brady, W., Klein, B., et. al. “Diet and Nuclear Lens Opacities.” American Journal of Epidemiology. 141.4 (1995).

(3) Ibid.

(4) Ibid.

(5) Ibid.

(6) Todd, K., Hudes, M., and Calloway, D. “Food intake measurement: problems and approaches.” Am J Clin Nutr. 37.1 (1983).

(7) Donaldson, S., and Grant-Vallone, E. “Understanding Self-Report Bias in Organizational Behavior Research.” Journal of Business and Psychology. 17.2 (2002).

(8) Livingstone, B., and Black, A. “Markers of the Validity of Reported Energy Intake.” J. Nutr. 133. 3 (2003).

(9) Lichtman, S., Pisarka, K., Berman, E., and et. al. “Discrepancy between Self-Reported and Actual Caloric Intake and Exercise in Obese Subjects.” N Engl J Med. 327. (1992).

(10) Ibid.

(11) Howarth, N., Huang, T., and Roberts, S. “Dietary Fiber and Fat are Associated with Excess Weight in Young and Middle-Aged US Adults.”J Am Diet Assoc. 105. 9 (2005).

(12) US Department of Agriculture, Agricultural Research Service. “Features of AMPM.” (October 2019) https://www.ars.usda.gov/northeast-area/beltsville-md-bhnrc/beltsville-human-nutrition-research-center/food-surveys-research-group/docs/ampm-features/. Accessed 5 May 2020.

(13) Conway, J., Ingwersen, L., and Moshfegh, A. “Accuracy of Dietary Recall Using the USDA Five-Step Multiple-Pass Method in Men: An Observational Validation Study.” J Am Diet Assoc. 104. (2004).

(14) Ibid.

(15) Chiu, C., and Taylor, A. “Nutritional antioxidants and age-related cataract and maculopathy.” Experimental Eye Research. 84. (2007).

(16) Mares, J., Voland, R., Adler, R., et. al. “Health Diets and the Subsequent Prevalence of Nuclear Cataract in Women.” Arch Ophthalmol. 128.6 (2010): 738-749.

(17) Ibid.

(18) Harvard T.H. Chan School of Public Health. “Nutrition Research and Mass Media: An Introduction.” https://www.hsph.harvard.edu/nutritionsource/media-full-story/.

(19) Ibid.

(20) Ibid.

(21) Webb, F., Whitney, E., Sizer, F., and Whitney, E. “Nutrition: concepts and controversies.”

(22) Ibid.

(23) Harvard T.H. Chan School of Public Health. “Nutrition Research and Mass Media: An Introduction.” https://www.hsph.harvard.edu/nutritionsource/media-full-story/.