Breaking the echo-chamber: usability and data driven design

One thing that game companies have a lot of is data. They record tons of ‘big’ data using their various analytics. However, this only tells the company what people have done (where did they go? What did they click on? How often do they log in? How much money are we making?). This big analytics data says nothing about how people feel (Why did they do something? Did they enjoy what they did? What changes would make people happier?). The use of usability testing to generate data can help break up echo-chambers that are likely to form within companies or within feedback forums.

To understand the motivation of players and to understand the ‘why’ in player behavior, game companies need to do actual psychology testing of their game via surveys and lab testing. The New York Times recently ran an article on the importance of “small” data (titled “How not to drown in numbers”). In it, the article highlights the fact that Facebook not only measures clicks, but asks people “why?” via surveys.

With regards to video game companies, one that has been very successful in using ‘small’ data is Riot Games, who produce League of Legends. Part of the success driven by League of Legends comes from maximizing enjoyment via using user data to decide how to make future changes in the game. They pilot potential future game content with the help of psychologists. The most obvious use of data by Riot is their player behavior team that has significantly reduced negative player behavior in the game, which they continue to develop today. Their older GDC talk about how their ‘small’ data collection improved the game can be found here. Riot’s data collection taught us that it was possible to reduce negative player behavior, even if it wasn’t possible to completely eliminate negative behavior.

The interesting part of Riot is that they extended this value of data collection into every aspect of the game’s design. This includes surveys about the amount of money people were willing to pay for services, such as new skins (or what types of new skins players would want). If something appears in the game, it is likely a combination of designer’s ideas and people who tested the potential impacts of those ideas before they were implemented.

Riot’s approach differs significantly from companies who design primarily based on their own developer’s guts with little or no integration of surveys or other usability data. Usability testing can help to see whether or not people are having fun. There is no way to measure fun other than to actually interact with users via usability testing  (whether survey-based or observation of players). The best design is user-centered design, rather than trusting that other people will use a product in the way that designers intend. Design needs to care about usability and player experiences. For people interested in usability testing, Carol Barnum has a book titled “Usability Testing Essentials: Ready, Set… Test!” there are also plenty of other resources to learn about how to measure player behavior and enjoyment.

Regardless of the methods, ignoring the experiences of players can result in designing without knowing what users will do in the future, or how players feel.  While it is unlikely that any designer can make everyone happy all the time, understanding players’ thoughts, motivations, and goals can improve the design of games. User testing allows for understanding how people will react to things that are not yet released out in the wild. While tools like feedback forums are helpful, the use of actual targeted user testing and survey data gives much better samples and avoids the echo-chamber problem that can happen inside offices or discussion forums. Frustrating players by making them feel like their thoughts and feelings don’t matter ends up hurting companies. Interacting with users via actual experiments and surveys can help to make sure that data being used to inform game design in meaningful ways. In the rise of big data via analytics after the game’s release, we can’t forget the importance of measuring whether or not people are having fun now, or could be having MORE fun in the future. A little psychology driving user experience testing can go a long way in improving game design. I just wish more game companies invested in psychologists as a core part of their design teams.

Posted in Research on video games, Written By Lissanna

3 comments on “Breaking the echo-chamber: usability and data driven design
  1. RohanV says:

    I’m not entirely sure I agree with this argument. For example, take movies. A lot of people think that movies are getting “focus grouped” to death these days, and thus are becoming much more homogeneous and bland.

    Henry Ford once said, apocryphally, “If I had asked customers what they wanted, they would have said faster horses.”

    In some ways, it is the duty of our artists to exercise *their* judgment, to give us something new and the change the way we view the world. For games, sometimes that means inconveniencing us, forcing us to adapt and learn new playstyles.

    • Lissanna says:

      User research has to be about more than just focus groups. It has to be about doing actual psychology research. You can do lab tests where they do A versus B and you see whether A or B is better. In Riot’s case, they put things out in the world and do research to see how they can make those things better. Focus groups aren’t the right tool for every job, so it requires actual real scientific testing to know how users will interact with a product, or how to make products better.

      The thing causing problems in the movie industry, however, is the industry itself and not the people watching movies.

    • Bob Tilford says:

      I agree with Lissanna. User research rarely involves focus groups, and it’s never just about asking for someone’s opinion and then implementing it. User research is really about gaining insight into how people interact with a game.

      For example, if someone designed a game around an innovative and unusual method of control, user testing might reveal that people struggle to pick it up, but that other people who’ve spent more time with the game really enjoy it. One option might then be to rework the way the tutorial introduces the control scheme to help newcomers learn the ropes.

      At the broadest level, the aim of games user research is to help games meet their designers’ intentions.


Featured Blogs