Blog Archives

Breaking the echo-chamber: usability and data driven design

One thing that game companies have a lot of is data. They record tons of ‘big’ data using their various analytics. However, this only tells the company what people have done (where did they go? What did they click on? How often do they log in? How much money are we making?). This big analytics data says nothing about how people feel (Why did they do something? Did they enjoy what they did? What changes would make people happier?). The use of usability testing to generate data can help break up echo-chambers that are likely to form within companies or within feedback forums.

To understand the motivation of players and to understand the ‘why’ in player behavior, game companies need to do actual psychology testing of their game via surveys and lab testing. The New York Times recently ran an article on the importance of “small” data (titled “How not to drown in numbers”). In it, the article highlights the fact that Facebook not only measures clicks, but asks people “why?” via surveys.

With regards to video game companies, one that has been very successful in using ‘small’ data is Riot Games, who produce League of Legends. Part of the success driven by League of Legends comes from maximizing enjoyment via using user data to decide how to make future changes in the game. They pilot potential future game content with the help of psychologists. The most obvious use of data by Riot is their player behavior team that has significantly reduced negative player behavior in the game, which they continue to develop today. Their older GDC talk about how their ‘small’ data collection improved the game can be found here. Riot’s data collection taught us that it was possible to reduce negative player behavior, even if it wasn’t possible to completely eliminate negative behavior.

The interesting part of Riot is that they extended this value of data collection into every aspect of the game’s design. This includes surveys about the amount of money people were willing to pay for services, such as new skins (or what types of new skins players would want). If something appears in the game, it is likely a combination of designer’s ideas and people who tested the potential impacts of those ideas before they were implemented.

Riot’s approach differs significantly from companies who design primarily based on their own developer’s guts with little or no integration of surveys or other usability data. Usability testing can help to see whether or not people are having fun. There is no way to measure fun other than to actually interact with users via usability testing  (whether survey-based or observation of players). The best design is user-centered design, rather than trusting that other people will use a product in the way that designers intend. Design needs to care about usability and player experiences. For people interested in usability testing, Carol Barnum has a book titled “Usability Testing Essentials: Ready, Set… Test!” there are also plenty of other resources to learn about how to measure player behavior and enjoyment.

Regardless of the methods, ignoring the experiences of players can result in designing without knowing what users will do in the future, or how players feel.  While it is unlikely that any designer can make everyone happy all the time, understanding players’ thoughts, motivations, and goals can improve the design of games. User testing allows for understanding how people will react to things that are not yet released out in the wild. While tools like feedback forums are helpful, the use of actual targeted user testing and survey data gives much better samples and avoids the echo-chamber problem that can happen inside offices or discussion forums. Frustrating players by making them feel like their thoughts and feelings don’t matter ends up hurting companies. Interacting with users via actual experiments and surveys can help to make sure that data being used to inform game design in meaningful ways. In the rise of big data via analytics after the game’s release, we can’t forget the importance of measuring whether or not people are having fun now, or could be having MORE fun in the future. A little psychology driving user experience testing can go a long way in improving game design. I just wish more game companies invested in psychologists as a core part of their design teams.

Posted in Research on video games, Written By Lissanna

A quick note

Hi all,

My guild is recruiting: Undying Resolution currently has some open spots for Ranged DPS classes for our 25-man raiding. We are hoping to get in 1 to 2 more ranged DPS players in the next week or two to help keep our roster full for going into the summer vacation months. We are 11/12 TOT. We are raiding three nights a week, since we recently added a third night of 25-man raiding to our schedule. So, we are now raiding 8 to 11 EST on Wed, Thurs, & Sun (US – Elune – Alliance). Adding the extra raid day was really beneficial to us, so we’re gaining faster progress speed than we’ve ever had before! We are looking for people with some MOP-level raid experience. Check us out!

A quick update about me: I have been traveling the last week for work. I attended a work-related conference for the Society for Research in Child Development this last week in Seattle, Washington. At the conference, I presented some preliminary results from the Autism face processing study that was included as part of the research I described in my crowdfunding campaign last fall. The data I presented was some of the functional MRI brain data from the baseline (before the intervention) time point of the intervention study, since we are still working on completing the follow-up assessments from the intervention this month, with lots of other exciting research things happening this Summer/Fall. However, this means I may be a little slow in posting blog content until the semester ends in a few weeks. Sorry for falling behind in my blog posting! I will try to get a post up actually related to druids “soon”.

Hearthstone! One of the more interesting things to happen at Blizzard in recent months was the announcement that they will be releasing an electronic card game related to WOW, called Hearthstone. While this is somewhat old news, I wanted to highlight how much fun I think it will be. You can expect to see some Hearthstone related content here on Restokin when that game is in beta & release.

Looking for guest bloggers! Are you a druid that enjoys writing about WOW? Well, I’m opening back up my “voices from the community” series to guest bloggers this summer. If you have an idea for a World of Warcraft Druid-related post that you would like to write and have posted on Restokin, feel free to send me an e-mail: lissanna70 at Please tell me about yourself (info such as how long you have been playing, an armory link, links to any previous blogs/youtube/etc if you have them), and what druid-related post ideas you are interested in possibly writing.

Resto Roundtable at the Team Waffle Podcast: The Team Waffle Podcast recently had a resto druid roundtable. The lineup included Jarre, Hamlet, Sodah, Jasyla, and of course Arielle as the host/moderator.

BONUS: Resto druid buffs in newest 5.3 patch notes. It looks like they finally put in some more resto druid buffs in the 5.3 patch notes. Importantly, this includes a larger player cap on Tranquility for healing in 25-man raids (will hit 12 instead of 5 in 25-mans), a range increase for our other AOE healing (mushrooms & Swiftmend’s Efflorescence – 10 yards, up from 8), and a reduced cooldown on Ironbark.  Healing Shrooms should also do more healing, and thus may actually be useful “soon”. These should be nice quality of life fixes for 25-man raiders in particular, who have been struggling this tier with feeling useful.

Posted in Research on video games, Restoration Healing Trees, Written By Lissanna

The psychology of boss design part 1: Information overload

The design of PvE raid bosses in World of Warcraft is a complicated process. Each person on the boss design team gets to create their own encounter(s) in a raid dungeon, sometimes with collaboration on some of the trickier bosses. Over the course of the game’s development, the fights have become more and more complicated. The bosses have more abilities, requiring more movement, and more coordination as a group. These bosses are requiring greater memory demands, greater multi-tasking, and faster reactions to things happening in the environment. These actions all happen while we also complete a complicated series of button presses or mouse-clicks related to filling a specific role in the raid (tanking, healing, or damage dealing).

Over this series of posts, I will talk about how the Mists of Pandaria raid encounters are pushing the limits of human memory, reaction speed limitations, and visual perception abilities (for good or bad!). In this blog series, I’m going to talk about several problems that plague boss encounters, where the fight mechanics are breaking several core psychology principles (related to memory, reaction speed, and visual perception abilities). I will also explain how these principles matter for being able to learn boss encounters in WOW, especially  as it impacts LFR versions of these encounters (where we can expect players to have spent less time researching the encounters in advance).

Can you remember all the mechanics?

First, we are going to talk about Memory abilities and how it impacts our ability to learn how to kill bosses in WOW.

  • Short-term Memory: Your ability to remember items over a short period of time (Wikipedia definition). Science cites 5 to 9 items (7 + 2) as the range for the maximum numbers of unrelated words or digits you can hold in mind. In the cases of WOW, you could think of this as the maximum number of unrelated boss mechanics that a player in the raid could remember if your raid leader listed off boss mechanics and you didn’t spend time to memorize them before the fight. Once we pass around 7 boss mechanics, you probably couldn’t actually recite all the mechanics back to your raid leader (and most people wouldn’t even get all 7 right). In addition, you naturally tend to remember the first and last numbers from the list better than ones in the middle. Basically, your ability to remember new information is limited.
  • Working Memory: Your ability to both briefly store information and use that information to achieve a goal (Wikipedia definition). In this case, not only remembering those boss mechanics, but responding to them appropriately during the fight (actually flying to nests, 1, 2, and 5 while also healing your party members, rather than just remembering that you need to fly to nests 1, 2, and 5). The ability to both remember the mechanics and use that information to perform the fight correctly is using a system with very constrained and limited resources.
  • Long-term Memory: Your long-term memory ability is much less limited than short-term or working memory abilities. You can remember thousands of vocabulary words, math, physics, what to do at your job, etc. If the boss strategy requires too many elements, your goal prior to arriving in the boss room is to study and memorize all the boss mechanics and strategies. Also, with practice (many, many wipes on the boss), you can learn to remember an almost unlimited number of boss mechanics, ability lists, or whatever.

The problem of increasingly more complex fights in WOW:

  • It has become apparent quickly to me in Mists of Pandaria that the boss encounters in raids are quickly passing the “sweet spot” in our short-term and working memory capacity. Thus, most boss fights, especially for normal and heroic modes, involves simply memorizing a choreographed “dance” for each fight (committing the boss strategy details to long-term memory in advance by reading boss strategy guides and videos), and then practicing that “dance” with other people in the encounter until you have fully learned the fight “dance”.
  • Many people doing fights in LFR don’t spend the time outside of the game learning the “dance,” and the fight mechanics can’t be done with only using short-term memory and working memory abilities when you first encounter the fights. This is why Blizzard has to either totally trivialize the encounters (so you don’t have to remember any of the strategy involved at all), or players generally have a miserable LFR experience. This also applies to more “casual” guilds that may not have the time to commit to serious advanced studying of fights ahead of time. My own guild makes people read and sign threads before arriving, and several of the TOT raid encounters have exceeded our own memory capacities based on the need to really spend an hour or more memorizing the boss details to understand the fight in advance.

Average number of boss mechanics by raid dungeon

I pulled up the Wowpedia boss mechanic page for each boss of each raid dungeon described below. I counted the number of boss abilities and computed an average “memory score” per dungeon. I have also included the minimum and maximum memory score. In the case of Patchwerk from the original Naxxramas being the lowest memory score, he ranked as a 3: Hateful Strike threat/health requirements, range requirements to prevent kiting, and the frenzy/enrage mechanic that caused most of the wipes.

Table of average, minimum, and maximum memory scores by raid dungeon (in previous tiers, some were combined, such as BOT and BWD into one mean for the two major dungeons in that tier. I left individual scores for MV/HOF/TOES):

Molten Core5.547
Original Naxx6.333333333310
SSC and TK7.9317
Mount Hyjal4.846
Black Temple8.666666667314
BWD and BOT11.4717
Throne of Thunder12.91666667919

and as a graph for easy viewing:

Conclusions and recommendations:

There is a fairly consistent trend for increases in memory demands over time since Molten Core. In addition, the type of overly complicated fight, which used to be the “end” boss of each tier, is now being placed early in the raid dungeon, causing roadblocks for new guilds trying to get some early progress through normal modes.

Throne of Thunder is really an outlier in terms of the memory demands placed upon average raiders. What is actually more concerning, however, is the huge jump in memory requirements between the previous raid tier (MV, HOF, and TOES) having fairly straightforward mechanics, and Throne of Thunder’s huge list of conditional requirements that need to be remembered. Even the first TOT boss, Jin’rokh, which is the least memory intensive has huge conditional requirements on every mechanic: run out when you get the ball, but not over another spark or you will wipe the raid, and not through the water or you will wipe the raid, and also not where the water will later spawn or you may cause deaths later in the fight. Even for average raiders, this starts to be information overload when the raid leader tries to explain the fight.

Suggestions for LFR design and the Dungeon Journal: For the LFR version of Jin’rokh, you still have 14 bullet points in the dungeon journal (though under my memory scoring strategy, this fight had a memory score of 9 – since earlier raid tiers didn’t have a dungeon journal). The LFR journal just makes it possible to ignore some of those mechanics and still live (though there is absolutely no clear indication of which points will still kill your raid members or not). If someone read you all the names of all the points and asked you to repeat just the names of them back to you, you couldn’t actually do it after only hearing the list once.

For LFR versions of fights, rather than keeping approximately the same number of mechanics to remember and just making mistakes less deadly, it may be necessary to remove a greater number of mechanics from the LFR versions of fights. In general, I’d recommend to keep the number of points on the dungeon journal for LFR fights below 10. Then, the people without raiding addons or watching fights in advance would have an easier time learning the scaled down version of the fights. The raid designers do this some, but as the number of boss mechanics increases, the memory demands for LFR versions of fights needs to stay in a range that people can handle. However, if you look at the dungeon journal, it is pretty much uninterpretable for people running LFR OR normal-mode encouters, and this is not really either a helpful or informative tool. In the case of Durumu, I gave him a memory score around 17, but realistically, his raid finder page has 28 different key terms with descriptions.

Posted in Mists of Pandaria, Research on video games, Written By Lissanna

Video game research!

Our crowdfunding for science campaign has now ended! Thank you everyone!

There was a great TED talk in November about the importance of making better educational video games. Video game researcher, Daphne Bavelier, from the University of Geneva, talked about research on how playing video games can change cognitive abilities and brain development (for the better!). Dr. Bavelier’s scientific work has been a hugely inspirational to me. I was able to attend a talk she gave at Penn State in 2010 about her video game research. The end of the video is particularly inspiring where she talks about the need for gamers and scientists to come together to improve the quality of games with educational goals. My goal as both a gamer and a scientist is to make educational interventions that tap into the motivational aspects of being fun! In honor of my last day of crowdfunding for my very own new video game research project, I thought I would share my inspiration with all of you:

We will soon be returning to our regularly scheduled WOW-related posting. Starting next week, I hope to have more WOW-related content to share with all of you about mages and druids. Thank you all for supporting me and my research!

Posted in Research on video games, Written By Lissanna


Featured Blogs