Video game research: Why correlation does not equal causation

Okay, so I ran across an article. Usually, the media likes to hype up studies and misrepresent them. Sometimes, people doing research on video games just don’t understand video games. Usually, the goal of video game research is to tell you how evil this media is and how it’s going to either cause us to become psycho killers or vegetables who aren’t able to function outside of our parents’ basements.

Some days, I can just accept that people do bad science, and that the media sucks at being able to report science. However, other days it just bugs me. Here’s a link to the MSNBC article.

The original article doesn’t appear to actually be out yet, so I can’t read it. However, all that are presented in terms of data are descriptive information (and the “average” age of players is essentially meaningless data on it’s own). The other thing seems to basically be a comparison of players vs. non-players on two variables: weight and depression.

However, since these are naturally occurring variables, you can’t determine if one caused the other. You could just as easily argue that being overweight and depressed leads you to playing computer games. Also, what alternative behaviors would this group of people be doing? Perhaps the alternative to playing video games would have been watching TV. Maybe employment could confound their data (unemployed people would potentially have a lot of time to sit at home & be depressed). However, since they didn’t actually report a correlation analysis (in the group of video game players, does # of hours playing the video game actually have a relationship with weight & depression symptom severity?). Also, were they playing solitary console or computer games, or were they playing online MMO games where they actually interact with other live people on a regular basis?

It’s really easy to misrepresent your population if you go into a study looking for blood, instead of looking for positive and protective aspects of game play. The MSNBC article also manage to stereotype female game players as calling them all pretty much depressed, using the game to “self medicate.” Okay, at least in this case the game wouldn’t be causing the depression. Um… have they ever actually talked to people who play video games and asked what they like about playing the games?  Or, maybe look into what positive aspects video games could have? Could online communities be used to improve females’ feelings of social support and provide protective factors if they had the depression before they sought out video game play as a distraction? Also, what percentage of the female gamers actually had clinically diagnosable depression? If it’s still a small number of people, then it isn’t really cause for concern and they could even maybe focus on the positive aspects of engaging in something cognitively challenging to better their spatial and problem solving skills.

Also, their sample all came from the Seattle, WA area. You know what is in the Seattle, WA area? One of Microsoft’s big campus headquarters. So, if they sampled people only from there, it may not be representative of gamers who live in other areas.  I’m just going to leave it up to your imagination what effect you think living right by Microsoft’s big US headquarters would have on your physical & mental well being. I think sampling in the Seattle area would lead to possibly having a survey sample that wasn’t representative of the whole US gaming population.

Also, showing the mean of players between 19 and 90 is not meaningful data on it’s own. The range of players (were there people in their 60s to 90s playing?) would actually be more interesting data. Or, maybe a graph showing how many people played at various ages (maybe the article does that?).  This is why the mean misrepresents the important parts of this data. Lets say (for example) you take 10 participants and have these ages: 19, 19, 20, 23, 50, 40, 30, 90, 60, and 35. The “average” of these 10 players is about 38. However, the number “38″ does not actually describe anything useful about that group of numbers, and actually misrepresents what that data is telling you, and the outlier sitting at 90 is actually skewing that data to be quite a lot older than most of the players actually are.

I found another article that compared their average of video game players to the actual US average, where they said that the US average age of adults was 35 (Williams, Yee, & Caplan, 2008). So, if the average age of adults is 35… then saying the average age of gaming adults is 35 is not all that interesting.

I know that the media usually does a bad job of reporting news from research articles, but I’m pretty sure that the article it’s self would still be well… depressing. When you are a scientist (even if it’s social sciences), you need to be careful not to use statistics in a way that just reinforces stereotypes and adds nothing actually new to the research field.

For example, Williams, et al. (2008) conducted a different study where they looked at EverQuest 2 players, and found that they had a mean average salary that was hither than the general US population. The EQ2 players actually had lower BMI scores (weight divided by height) than the average American adult. So, this is the opposite finding than what the MSNBC article is reporting for weight statistics. They seem to have found similar depression rates, however the EQ2 sample only had 20% of their 7,000 sample was female (which means the actual number of depressed females had to have been somewhere around 500 people). However, EQ2 isn’t necessarily even the most popular game anymore – so, EQ2 players may also not represent the larger gaming community. For the MSNBC article, their total sample size was only about 500 people – which means the actual number of depressed female gamers is going to be a really small number of actual people who were both female and depressed. So, it’s hard to understand what it means for female gamers in samples where females are really outnumbered by males.

At the very least, these statistics don’t prove that video games caused people to be one way or another. It could just be that people with whatever physical or personality characteristic just like playing the games, which is interesting, but it’s not worth freaking out about.

This entry was posted in Uncategorized. Bookmark the permalink. Both comments and trackbacks are currently closed.

7 Comments

  1. Thetalina
    Posted September 6, 2009 at 12:35 pm | Permalink

    I’m a WoW player who also happens to do survey research for a living. I’d have to say that compared to most media reporting of studies, this article is actually pretty good, although the headline is terrible. The article itself doesn’t make any claims about causation, and makes the point about possible self medication (which would be depression causing gaming rather than the other way around).

    The headline suggests that the average player is fat and depressed. The article of course suggests no such thing. Just that there is higher incidence of being overweight or being depressed. But it doesn’t say what percent of gamers are either of those things. If 10% of non gamers are overweight and 15% of gamers are overweight, that’s certainly consistent with the article saying that gamers are more likely to be overweight. But it’s a far cry from saying that the average gamer is overweight. Under those circumstances, the average gamer would be pretty much like everyone else.

    This is a really common problem in every area, not just gaming. Headlines are generally written by an editor, and the writer has no say over it. The headline writer is looking for something catchy that will attract pageviews. This is just another instance where a reasonably good article (although lacking for not including any actual numbers from the study) is spoiled by a terribly misleading headline.

  2. Kaelik
    Posted September 6, 2009 at 5:12 pm | Permalink

    It’s not just a misunderstanding of the difference between correlation and causation. People, especially in the media and government, are usually complete failures when it comes to science and basic reasoning skills. Sometimes it’s just because they are honestly stupid…but most of the time it’s because they came to the conclusion LONG before they examined the data and formulated their argument.

    It’s hard to argue with them, since their goal is to bash gaming (or whatever the target at hand is) rather than really understand the issue. The approach I like is an example of how silly their logic is…”Shopping at Wal-Mart leads to being poor”. After all, any survey of Wal-Mart shoppers will undoubtedly reveal an average income below that of people who are instead shopping at the Louis Vuitton store. Based on the logic of the “scientific study” in the MSNBC article, I can conclude that shopping at Wal-Mart MAKES you poorer than shopping at Louis Vuitton.

    If you want to lose weight and become less depressed, stop playing WoW. If you want to become rich, shop at expensive stores…I’m a genius!

    Also, it’s worth noting that since I’ve started playing WoW, I’ve lost about 25 lbs, and continue to lose more. I don’t attribute that to WoW either, except maybe in the sense that it’s hard to heal Trial of the Crusader while shoving Cheetos into my mouth (although maybe THAT’S what people are doing when they stand in the fire on the demon boss).

  3. Other Cassie
    Posted September 6, 2009 at 6:27 pm | Permalink

    I think Seattle is not a great place to look at depression for many reasons, including that if I recall correctly the weather has a measurable effect on depression there. (I may love grey days, but I think I’m in the minority.)

  4. Lissanna
    Posted September 6, 2009 at 7:45 pm | Permalink

    You guys all have really good points. :)

  5. Posted September 7, 2009 at 7:30 am | Permalink

    THANK YOU! I knew I couldn’t be the only one who looks at studies linking anything to anything and and thinks that it may be that the second one that causes the first. I actually once took a basic statistics class, and though the whole class was interesting, the stuff I learned in the first month was by far the most applicable to real life. If more people know about the basic concepts of statistics, like outliers and correlation not implying causation and hidden variables, we wouldn’t have media outlets using statistics for scare tactics rather than useful information.

  6. Ogre Jehosephatt
    Posted September 7, 2009 at 8:45 am | Permalink

    BMIs are something that always annoy me because they’re so misleading. Using BMI measurements, many professional athletes would be considered obese. Personally, I’m a bit overweight and rather out of shape, but the doc says I’m healthy anyways. AND I’m still more physically capable (faster, stronger, nimbler, tougher) than most people I run into. BMIs strike me a so superficial that they are meaningless.

    And, heh, I wonder if anyone can show a correlation between intelligence and depression, and then show a correlation between intelligence and gamers.

  7. Lissanna
    Posted September 7, 2009 at 12:17 pm | Permalink

    Yes, I agree that BMI is not always the best measure to use to determine how healthy someone is.