30 Bananas a Day!

A Cancer Epidemiologist refutes Denise Mingers China Study Claims due to incorrect data analysis

From Denise's blog...

"An exhaustive analysis of the raw data from the China Project by
educator and freelance writer Denise Minger [41]  shows that Campbell
failed to take into account other disease-causing variables (increased
Hepatitis B and schistosomiasis infection and rates, industrial work
hazards, etc.) that tend to cluster in higher-cholesterol counties in
the China Study. Campbell also omitted data showing a higher
correlation between wheat flour intake and many diseases (notably
coronary heart disease, cervical cancer, hypertension  and stroke) than
with animal protein intake."


The following is from an Epidemiologist that refutes Denise Mingers China Study Claims due to incorrect data analysis...

OH MY. By request of beautiful Freelee, I've taken a look at Denise's analysis. I'm an epidemiologist, and on top of that my research focuses on cancer (not that this makes me completely infallible, but at least I feel equipped to provide an informed critique of her statistical capability). Dr. Campbell was certainly gracious in his response to criticism, but I cannot be so kind. Denise is incredibly naive in her crude analysis of the raw data. She uses correlations and ecologic comparisons to draw conclusions about relationships between diet and outcome (cancer, cardiovascular disease, etc.). WRONG WRONG WRONG!!

A correlation does not an association make.

And, as epidemiologists, our studies are intended to determine associations between exposures and disease. (Yes, there are special methods to determine actual causes of disease, but for most of us, associations will do.) See point 1 below for more on this.

Denise, while meticulous, went through a series of exercises only to:

1) Provide a series of correlations, which honestly, is just the FIRST STEP of any good statistical analysis. Let me explain in a nutshell - a correlation is a linear (assumes a "straight-line" relationship - but not all things are related in this manner), unadjusted (does not account for multiple factors that could potentially confound the relationship between an exposure, like diet, and outcome, like cancer), and non-directional (it does not say if one caused the other or the other way around). An association, on the other hand, is generally adjusted for potential confounding factors and - if a study is properly conducted - gives us an idea of temporality or direction. While we certainly look at correlations between all factors (i.e. between the exposure, potential confounding factors, and the outcome), typically more complex modeling of the data ensues so that multiple factors can be accounted for when investigating the relationship between an exposure and the outcome.

2) Much of her conclusions are drawn from purely ecologic data - that is, data that is in aggregate - such as evaluating total cholesterol and colorectal cancer (as Denise does). Sure, it can be informative, but it doesn't tell us anything about some of the other factors that might be related to cholesterol and colorectal cancer. And while she does perform a stratified analysis (stratifying on schistomiasis), which is a form of "adjustment for confounding"), it still does not take into account other possible confounders and still only tells us about general patterns, but nothing of individual-level associations. Furthermore, she doesn't present results for regions with schistomiasis. What if there was also little correlation between cholesterol and colorectal cancer in these regions? There might be other factors unaccounted for.

Ecologic studies are considered to be at the bottom of the "epidemiologic study totem pole." And we can NOT draw individual-level conclusions from them, i.e. we cannot say that an individual who consumes less fat will, on average, be protected from breast cancer (even if that's true, we cannot draw this conclusion from an ecologic study - there's even a term for it: "ecologic fallacy").

OK, my disclaimer: I'm an epidemiologist, and yes, scientists are NOT objective (I'll say it: I'm an ardent veggie with a happy veggie family). Hell, science is not objective. I mean, you could be given a blob of numbers that mean nothing. It takes some context, interpretation, and data processing to make anything meaningful out of those numbers. Yes, scientists can be biased, and so can the studies they conduct, and the analysis of those studies. But good scientists do the best they can, are open about their methods, and fair when discussing their results. I applaud Dr. Campbell for making his raw data available - very few scientists do this. I will be totally honest and say I have not read "The China Study" (I guess I feel it'd just be preaching to the choir, but I think I will read it now...). But I know enough to know that Denise's analysis was crude at best and completely wrong at worst. No card-carrying epidemiologist would EVER be able to publish her results and draw the conclusions that she does.

I've posted the following comment on Denise's blog (which, was there for a few minutes, and now when I go back to the site, it is mysteriously not there anymore...):
Your analysis is completely OVER-SIMPLIFIED. Every good epidemiologist/statistician will tell you that a correlation does NOT equal an association. By running a series of correlations, you’ve merely pointed out linear, non-directional, and unadjusted relationships between two factors. I suggest you pick up a basic biostatistics book, download a free copy of “R” (an open-source statistical software program), and learn how to analyze data properly. I’m a PhD cancer epidemiologist, and would be happy to help you do this properly. While I’m impressed by your crude, and – at best – preliminary analyses, it is quite irresponsible of you to draw conclusions based on these results alone. At the very least, you need to model the data using regression analyses so that you can account for multiple factors at one time.

** Updated to include an example from Denise's analysis rather than my original example of fat consumption and breast cancer.

I just realized that there's still some trail left about "fat consumption and breast cancer". I should clarify. Denise looked at cholesterol level in each Chinese county and the corresponding incidence rates of colorectal cancer in that region (this is what makes it "ecologic" - each dot represents a county). But the statement still stands - we can't make individual-level conclusions about cholesterol, colorectal cancer, and schistomiasis.

I also just want to add that when she refers to "statistical significance", all that's being tested is the "null hypothesis" that there is no correlation (i.e. correlation = 0). it is not testing whether an exposure is or is not arisk factor for the outcome, even though Denise uses this term loosely.

Views: 5275

Replies to This Discussion

The author wants to stay out of the whole thing as much as possible so I won't be sharing their full name here however they have openly posted in other threads here...
It's important to identify the source.
This is so great. It's exactly what I was thinking on raw data. I don't understand why she (Denise) used ALL the raw data. I don't know any statistician who does that.

That is statistics 101. It's what we did for our first projects - we were all handed the same raw data, with a variety of questions and then asked to interpret it light of those questions. Using every single piece of raw data for every question would have yielded a big fat zero on the project. Simply because the results would have been irrelevant. Not every piece of data is relevant to the hypothesis, after all.

I don't have access to the book that has the raw data, but if I did, I could compute it a dozen different ways to make it say a dozen different things. None of them would be right. I don't have the years of knowledge Dr. Campbell does in knowing what to look for, how to interpret this particular data, etc. Of all scientists, Dr. Campbell seems like he'd be least biased - considering his background. I mean, what does he have to gain from skewing his data? Are the fruit farmers paying his checks? Doubt it.
Thanks for sharing Anna, this is very good to learn from someone versed in the correct interpretation of statistics.

"I mean, what does he have to gain from skewing his data? Are the fruit farmers paying his checks? Doubt it.

-:-D exactly!

A number of people have pointed out that the criticisms of Denise's analysis apply to Campbell's as well, and since they seem to be at least somewhat familiar with statistics, I'll expand on my initial critique.

First and foremost Denise did not take into account potential confounders. I think everyone understands at this point that confounders can bias the observed correlation towards or away from the null (i.e., correlation=0). While she took schistosomiasis into account by restricting her analysis to counties without schistosomiasis, it doesn’t tell us whether schistosomiasis really is a confounder – it simply removed the “effect” of schistosomiasis. Furthermore, her p-values only reflect the test of whether the correlation was significantly different from zero. Not if there was a statistically significant change in the exposure-outcome correlation after taking schistomiasis into account.

Let me repeat that. The p-values Denise provides reflect whether correlation=0. They do not tell us whether or not schistosomiasis is a potential confounder. To help us determine this, we need to know if the correlation of +33 for all counties was statistically significantly different from the correlation of +13 for just the counties without schistosomiasis. This is where 95% confidence intervals would be helpful, but Denise doesn't provide these. Nor does she tell us what the correlation is only among counties with schistosomiasis. There are several ways to tease out whether we should include a factor in our final analysis, but here are two commonly used methods, using the schistosomiasis/cholesterol/colorectal cancer example:

Method 1:
1. Calculate correlation for entire sample
--> Denise calculated this to be +33.

2. Now stratify on the variable you think is a potential confounder, i.e., schistosomiasis, and calculate the correlation within each stratum.
--> Denise stratified on county but we'll let this slide b/c this was probably her only choice. For counties with no schistosomiasis, the correlation was +13. What about the correlation for counties with schistosomiasis? Denise does not provide this.

3. Compare the within-strata correlations (+13 and ??) to the correlation for the the entire sample (+33), and test whether they are statistically significantly different from each other (not whether they are significantly different from 0). One should first perform a global test, and if the result is significant, proceed with pair-wise tests.
--> Denise did not do this.

4. If the correlations are significantly different from each other, then there is evidence that there may be confounding. If they are not significantly different from each other, there is evidence for no confounding.
--> Denise did not do this.

5. Bonus step: if the pair-wise tests between the stratum-specific correlations are significant, this is evidence that schistosomiasis is an *effect modifier*, not a confounder.
--> Denise did not do this.

Method 2:
1. Run a full model that includes cholesterol and schistosomiasis as exposures (ideally, the model would include more than just this, but we'll keep it simple) and colorectal cancer as the outcome. Obtain the adjusted correlation, and make a note of the residual deviance or log likelihood for the model.

2. Run a reduced model that does not include the variable you think is a potential confounder, i.e., just include cholesterol as an exposure. Make a note of the residual deviance or log likelihood for this reduced model.

3. Now take the difference of the deviances or the -2 times the difference in the log likelihoods. This is your chi-square test statistic with k degrees of freedom (in our example, the degrees of freedom=1). Calculate the corresponding p-value. A significant/small p-value strongly suggests that the we should stick with the full model (i.e., the one with cholesterol and schistosomiasis). A large/non-significant p-value suggests that the full model doesn't add much more information and therefore we would opt for the more parsimonious model. In other words, the reduced model (i.e., the one with cholesterol only) is probably sufficient.

I'm assuming Denise did none of this since there was no mention of it. To her credit, Denise does mention why she took a look at schistosomiasis, but didn't follow through with a complete analysis. Therefore, there isn't much ground for her to stand on.

When people criticize Campbell for not including schistomiasis, it is very possible that upon further inspection, it was not a potential confounder as Denise concluded based on her results. A factor is a confounder if and only if it:
1. Is associated with the exposure (cholesterol)
2. Is a risk factor or protective factor for the outcome (colorectal cancer), and
3. Is not on the causal pathway between the exposure and outcome.

Perhaps criterion 1 was not met and therefore not included in Campbell's final analysis. Only Campbell and colleagues know for sure what the detailed analyses were; a final presentation will always include only the most salient points.

As for many of Campbell's conclusions being drawn from purely ecologic data, I think this ignores the fact that while the China-Cornell-Oxford Project was a large component of the book "The China Study," the book's thesis is based on *hundreds* (in fact, nearly 1000) of additional references that corroborate the Project's findings.
brilliant veganmama, I will be sharing this around :)
Thanks to B for finding these comments on Denise's blog. I'd like to highlight a few things in an effort to provide a clearer picture for folks (my comments in bold)...


Hi Mr. Freddy,

I’ve parted ways from 30BAD and won’t be posting there again, but you’re free to pass this along if folks there are confused.

I’ve seen Campbell’s responses to previous critics and have been perplexed by the “misinterpreting uncorrected raw data” accusation.
[A reference to results that haven't taken into account other explanatory factors] My best guess is that he’s referring to the “Death from all causes” or “Death from all cancers” variable, which several critics cite in their reviews in order to vindicate animal foods. Both of these variables can be misleading taken out of context: In the raw data, correlations between animal food consumption are inverse for death from all causes (meaning the meat eaters tend to live longer) and also inverse for death from all cancers (meaning the meat eaters tend to have lower rates of cancer, in totality). These are easy things to cite for anyone looking to discredit “The China Study.” [And yet, she chose do present simple correlations without adjustment for confounders]

But what the uncorrected data here overlooks are the many, many confounding variables at play. Do the meat eaters also live in areas with better health care and living conditions (leading to fewer instances of non-diet-related disease) [I highly doubt that in the context of cancer, back in the 80's, better health care would contribute to lower incidence rates of the disease. I mean, do we really think that in rural China at the time, screening was widespread? Mammography wasn't even widespread in Japan until 2000! However, yes, living conditions could play a role in the incidence of cancer, cardiovascular disease, etc. but the really big point is this: Campbell was looking at all-cause mortality (death). Not incidence (new development) of disease. The two are related, sure, but they are not directly related.]? Do the meat eaters experience less “death from external causes,” another variable that contributes to all-cause mortality? Any number of entangled variables could sway the “Death from all causes” variable, rendering it fairly useless uncorrected.

Similarly, the “Death from all cancers” variable can be misleading without looking at individual rates of specific cancers [While true, assessing death from all causes is pretty common practice when trying to determine an overall effect, so I don't think a researcher should be vilified for it!]. Some are obviously related to lifestyle habits (like smoking and lung cancer), exposure to external hazards (like toxins in the workplace), infections (like hepatitis B or schistosomiasis)–so on and so forth. If, for instance, plant-eaters tended to be heavier smokers than the meat-eaters and exhibited much higher rates of lung cancer, then the “Death from all cancers” variable would lean in favor of meat consumption for reasons unrelated to diet [Again, she seems to be using incidence and mortality interchangeably. But let's pretend she's not - she's basically saying that other factors would explain all-cause mortality, not diet? So there's no evidence that meat is beneficial either? Well, heck. Then why not stop eating meat since it's cruel and unnecessary and stop smoking?].

In these cases, I’d certainly agree with Campbell that using the uncorrected data is unwise and potentially misleading [and yet again, that's what she presented on her blog]. That said, it appears Campbell himself relies on the raw data, since the correlations he cites are only valid before correcting for confounding variables [How can she know this? She didn't adjust for confounding variables. She also didn't properly assess whether a potential confounder was in fact a confounder.]

The analysis on this page avoids those traps by looking at individual cancers instead of cancer in the aggregate, dividing populations into high-risk and low-risk groups, and adjusting for variables known to influence disease rates. [What she's actually done is study disease more precisely by restricting her outcome to specific cancers, which yes, is a good thing. As for dividing the population, yes, she restricted to counties w/o schistosomiasis in one analysis, but again, didn't give us results for counties with schistosomiasis for comparison. In epi, we almost always want a comparison group. And, contrary to what she's said, she didn't adjust for confounding variables - she admits that she doesn't later on for the sake of "keeping her readers on track."]

I hope that clarifies some things. [No, it doesn't really, but that's ok. Sorry, had to be a bit snarky there.]



Hey John, it’s probably sufficient to post this on one entry instead of three of them.

I agree wholeheartedly with what Campbell says about the limitations of the China Project data (and for the record, I read the warning chapter in the China Study monograph before diving into the data). If you read my critique, you’ll see that I don’t slap down the raw correlations for this very reason: They’re misleading and can easily imply trends that aren’t actually there. This is why I focus on untangling variables and adjusting for confounding factors, thus rendering the data no longer ‘raw’.
[She's contradicting herself again... at least, with future comments with regard to confounding...]

Campbell’s claims, on the other hand, only appear to be valid before those adjustments are made. In every instance I analyzed, his claims matched with the raw correlations but not with the corrected ones [but she didn't calculate corrected ones!].

If you feel you or someone you know would be better qualified to handle the statistics, by all means, track down a copy of “Diet, Life-style and Mortality in China” and analyze it for yourself. I’ve tried to be very transparent with my process here so that others may replicate my methods or identify any logical errors, should there be any. If you have suggestions for how I can improve upon this analysis, I’d be glad to hear them. Apart from that, RE: your quote from Campbell — you’re preachin’ to the choir. :)

great response!.When I read Denise's article against the China Study I was thinking that her correlations were not correct but i was not able to say it as perfectly as you have! Thanks and I will always have this source when this article is brought up to me again!
Yes, I think it is clear that Denise knows some statistics and is very smart with a inquiring mind...but to be honest, the analyses she has performed are not up to snuff. She does not appear to have command of the field of nutritional epidemiology even at the level of the introductory graduate textbook "Nutritional Epidemiology" by Walter Willett, otherwise she would see the flaws and mistakes she made in her own analysis.

Btw, I thought she was doing 80-10-10, but now on her blog, she says she eats animal products...is this true??
apparently it is so. she eats animals for "health reasons."
I really liked Denise and had previously found her to be very smart, thoughtful, and respectful, but something here is just not adding up for me.

One, I recall her just 2-4 months ago saying that she had been eating LFRV for 5-7 years and feeling great on it, so I'm surprised by the sudden change if everything was going well.

Two, even if she was motivated to include animal products because of the so-called new "flaws" she found in the China Study, the correlations she found with meat consumption and disease were typically null, not negative, so even her own analysis does not support eating meat as healthier than not...so this seems to me to be self-justification, rather than a scientifically-justified change.

Three, her tone in her analysis was irreverent, in a field in which she is not an authority and has no expertise...yet she refers to her "BS-o-meter", as if she is the one who knows more about what is going on than an expert in field. What?!? Even if I found a scientific authority to be wrong, I would never be so disrespectful or presumptuous! This to me suggests either than she has an axe to grind and is trying to prove a certain point and is not open-minded, or that her ego has gotten in the way simply because she knows how to use a simple statistics software program.

Just my thoughts on trying to make sense of this....
"Three, her tone in her analysis was irreverent, in a field in which she is not an authority and has no expertise...yet she refers to her "BS-o-meter", as if she is the one who knows more about what is going on than an expert in field. What?!? "

- That is so true Courtney, I found this to be very strange & condescending in light of her inexperience. Sounds like you may be onto something here...

"This to me suggests either than she has an axe to grind and is trying to prove a certain point and is not open-minded, or that her ego has gotten in the way simply because she knows how to use a simple statistics software program."


30BaD Search

Latest Activity

ednshell left a comment for June Summer
8 hours ago
Profile IconJune Summer, Rebecca and Misty LoPresti joined 30 Bananas a Day!
9 hours ago
Courtney Beth replied to S3MPIT3RNAL vids's discussion Somebody please answer
12 hours ago
pradtf replied to pradtf's discussion great site with nutritional research information
12 hours ago
ednshell replied to S3MPIT3RNAL vids's discussion Somebody please answer
14 hours ago
Dominic replied to S3MPIT3RNAL vids's discussion Somebody please answer
14 hours ago
Peter's photo was featured
15 hours ago
Alex Curtis-Slep posted a video

Vegan Tokyo Bowls!

My dad and I made vegan Tokyo Bowls! I got the recipe from the Forks Over Knives family book. We used some veggies we had on hand, couscous, quinoa and made ...
15 hours ago
Ben Trovato posted videos
15 hours ago
ednshell and Nav Gosal are now friends
15 hours ago
Jamie replied to Megan Daultrey's discussion Vegan friends on snapchat
Alex Curtis-Slep posted a video

June 10th Fruitluck at Eureka Springs Park!

We're having a Fruitluck on Saturday June 10th at Eureka Springs Park in Tampa, Florida from 11am-2pm! Bring fruit to share, hang out with amazing people and...
Stephanie Williams posted videos
ednshell replied to Hummingbird's discussion Parkinson's Disease
Jan replied to Hummingbird's discussion Parkinson's Disease
Hummingbird replied to Hummingbird's discussion Parkinson's Disease

© 2017   Created by TheBananaGirl.   Powered by

Badges  |  Report an Issue  |  Terms of Service