Session 7: Quality Assurance; Playtesting by Genevieve Conley (Riot Games); Focus Testing Results

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK. Welcome to class. The theme of today's lecture is all about testing. So I'm going to give the intro to testing lecture where I talk about testing and how, what it is, and why we do it. Give you some basic testing tips, and let you know what we expect you to be doing in terms of testing for your projects.

And I'll spend a little more time on focus testing, because that's the part people are frequently least familiar with. We have Genevieve Conley from Riot, and she's going to give a lecture related to testing. But more about what Riot's doing and stuff like that. So hopefully that will be fun.

We'll take a break. And then since we just told you all about testing and how important it is, we will go ahead and run a focus test in class. We'll give you some time to get things set up, probably about 10 minutes to plan your test. Then we'll set things up and we will run testing.

We're going to ask that everyone on your team test at least two games and that everyone on your team observe at least one person playing a game. How exactly you're going to make all of that happen in flow is your problem. And then we will come down.

We'll give you some time to let your team compare observations and talk about results, and then we'll ask you to come down for a standard in class, two minute no visuals presentation. Monday, project two is due along with your design change log and focus test reports and your individual write-ups, and the project two presentation.

This is a five minute presentation. There are presentation guidelines on Stellar. You can download and look at them. Once again, we're primarily looking for postmortem and reflection. We want to hear about your process.

We want to know what you learned. We want to know what you did right, what you did wrong, and what you're going to do in the future. And we want it in five minutes. Grades for project one are up, along with comments about your individual postmortems and some comments about the overall project stuff. Overall, we were really impressed with the quality of project one. So keep it up.

But do you stop and read the comments so that you guys can make sure to do good jobs on your individual postmortems going forward. Anything to add? All right.

So people say testing. People say quality assurance. What are they? Quality assurance is actually just a whole process. It involves a whole lot of different types of testing. But in general, it's the systematic process checking to see whether a product of service being developed is meeting its specified requirements.

The most important part of that definition really is systemic, which means you're doing regular, repeatable testing. Looking at making sure that all the portions of your game or project are working. And requirements, making sure you're meeting that it doesn't just "work" as far as you think it does. It works according the requirements of the game, of the project, of whoever you're shipping to and et cetera and so forth.

So you need to know what your standards are to be doing good quality assurance. Without testing, your project will fail, and will take more time, and will take a lot more effort. You can make games without programmers. There's a lot of graphical user interface engines out there. Some of them are getting older and older, but they're out there and you can use them. GameMaker, for example.

You can make games without artists. I bet several of you are figuring out how to do that right now. But you can also make games completely without assets using text engines like Inform 7. You don't really need designated producer or distinct game designer on a small team. One person can really do it alone.

But you're never going to make a good game without testing, and the more systemic and planned your testing is, the more results you'll get from less effort. So who gets to do all this testing? That's really everyone involved in the project.

Do not assume that because you are the key programmer or the key designer or the key whatever that someone else will check your code and make sure it's working after you've integrated it. You really want everyone on the team to consider it to be their responsibility to keep the game working as well as possible all the way through, and also to take the time to go in and check other people's stuff to make sure that it went in.

Remember that this is one of the cornerstones of using Scrum as well, where before a task is actually done, someone else has gone in and verified it. Well, someone else is going to be someone on the team, which means you're all going to have allot some time to doing it, and you're going to have to think about that. Speaking of which, anybody have testing as one of your tasks on your sprint task list?

Anybody? Woo, you win. It's going to take some time. And it's going to take some time that probably otherwise could be used for making code. So go ahead and remember that now and think about that as you're going into your last week of work, that you need to allot some time for testing.

So types of testing. What is the terminology I'm going to use so that we all understand what I'm saying when I say it? I divide testing up into three major areas, and there isn't a lot of formal methodology on this, especially not in the game industry. So I tend to talk about technical testing, play testing, and user testing.

When I say technical testing, I mean getting your game to run. Specifically, I often mean getting your game to run exactly as how you intended it to run. So if the designer claims that the peach mobile should go up to 30 miles per hour and fire eight peaches per second, if you go into the game and you discover that it goes 70 miles per hour and fires two peaches per second, that's not working even if it makes game awfully fun with it like that.

When you start thinking about whether it's fun or not as a developer, then you're moving into the area of play testing which is where you, the team, are playing your game trying to figure out, hey, all those requirements and constraints and plans we made, are they working? And are they working well, and is a player going to enjoy it?

You're play testing your game as a developer. You're checking to make sure it works. In that case, yes. The fact that the peach mobile is currently going 30 miles per hour and it's so slow that no player is ever going to want to use it. That's a bug. That's a really big bug.

But it's a play testing bug if your specs say it should be going 30 miles per hour, not a technical testing bug. And it is good to know the differences so that you can know what you're talking about and why you're talking about it.

Finally, there's user testing, which is getting someone who is not on your team to play your game. You, the developer, while you do play testing, can pretend to be an actual player. But you are not an actual player. You will never be an actual player for your game. You know it too well. You are too close to it.

You have spent too many hours playing it. So you really need to get someone who has never seen your game before, ideally, to come in and sit down in front of your game and play it. We're going to make pretend when we run our focus testing workshop that your classmates here are actually good testers. Tell you a secret, they're not.

They've seen your game because they've play tested it before. They've heard you talk about it. They heard you pitch it. They might even have worked on it if they switched teams.

So they're not actually your ideal testers, but they are very convenient testers. So we're going to use them today. And for your next projects, we're going to encourage you-- and, in fact, require you-- to go out into the cold, cold world and find actual testers who haven't seen your game.

People in your dorms. People in your other classes. Friends, people at the coffee shop. Whoever you think will do the best job of giving you the information you need. User testing is another one of those terminologies that means a whole lot. So for my convenience, I tend to divide it up into two kinds of things, focus testing and "user testing."

Where "focus testing" is specifically when you're sitting down and you're playing the game, looking at how it plays. Is it fun? Are people enjoying it? Is it challenging enough? Are the mechanics interesting? Is your game a good game, and getting that information back from your players.

When user testing, I tend to use more for looking at the usability of your game. You may have a really, really fun game, but if it takes your players 12 minutes to get off your starting page because they can't figure out what they're doing or how to do it, that's a pretty serious usability issue. And you need to figure that out as well.

So just so you know that those are those two user types. It's all user testing, but having some granularity to find the difference is a little useful. So the basics for technical testing, which hopefully we can go through pretty quickly. I suspect that if you guys have done programming, you've done this before.

If you've had classes that wanted code that worked, hopefully they've helped teach you what the good things to do are. A lot of this sounds an awful lot like good software practices lecture that Andrew gave, because a lot of good testing is, in fact, a good software practice. So have some technical standards.

Have a bug database. Or, given the size of these projects, at least a bug Excel spreadsheet with a list of everything that you've noticed. Build and test before checking it in. Have daily builds. Have a daily play-through.

Planned feature cuts. Scope. When you realize that you're not getting everything done on Friday night as you're working on your game, start cutting then. Don't wait until Sunday night to decide what you're going to cut.

Cut as soon as you realize that you're blowing your estimates. That will allow you to choose what you cut and to adjust changes in your game to reflect the things you cut. It will also mean that testing your game to make sure-- it will introduce a lot fewer bugs when you do it in a controlled manner than when you do it in desperation, which is why I put it under testing.

Finally, code freezes and asset freezes. These are great, and no one likes to use them because it means you're going to waste an entire day not working on your game. Who wants to do that? On the other hand, has anybody here turned in a project that they've made a last minute change to Sunday night, it was due Monday morning, and then it didn't run at all? I got some honest people out there. Good.

The advantage of taking that whole day to test your game is you turn something in that works on Monday. Often, you turn in something that works a lot better on Monday because you've taken the time to test it. You've found the bugs. You've had a friend help you review your code changes, and you really got it working.

If you want to make a good, polished, solid game, you will think about it and give yourself a cushion for project two. That's probably maybe a day. It's a two week project. If you give yourself much more cushion than that, you're not actually working on it. But actively give yourself that deadline and give yourself some time to play your game.

 

So that's the basics. Some advanced tips. It's all pretty simple. I think you've all heard this all. How we maintain your feature list. Agree on how features are going to work, so talk with each other. Test those features as soon as they go in.

 

This may not be worth it on these games, because they're small. When you hit your project four game, you may want to have one of these. Think about having a standardized test plan that runs through all the buttons and the branches and the pages and the options of your game. I realize that when you've got randomness as one of your core design goals, you can't predict how your game is going to play.

But you should be able to take a look at your game and know most of the states that it's going to get in. And you can create a checklist that says, hey, have we exhausted all the possibilities that a player will encounter? Have we tested them all in the last two builds, yes or no?

And the answer should probably there be yes because one of the things that's very common with projects that have a lot of highly integrated systems working together and depend on emergent behavior-- which is most games-- is you can introduce a feature over here that changes the way that a feature that you implemented two weeks ago works. And you'll never know that it did that until you go through and you test and you find that something that was working is now broken.

So it is worth it to take the time to go back and check those features you know nobody touched. Bug reporting. Because so many people are so bad at this, I'm going to take the time with a captured audience to tell you how to do it. Be kind to your fellow engineers. Be kind to yourself, if you end up getting your own bugs.

Write good defect reports and not bad ones. Good bugs include what happened, what should have happened. Because it doesn't do me a lot of good if you tell me that the peach mobile is going 30 miles per hour if I don't know why that's a problem. How do I reproduce it?

If it's obvious to reproduce, you don't need to include it. But if it took 38 steps, include a text file. How common is the bug? How serious is the bug?

Is this something that the team needs to jump on right now, or is this something that Sunday afternoon you're going to look at and say, eh, ship it. Because some bugs you look at and you say, eh, ship it. But you want to know that it's there, and you want to know how it's going to affect your game before you just decide to ship it.

Include some supporting data if it's going to be helpful, and it often is. If you're trying to describe something visually, give them a screenshot. If you can run it in the debugger and catch some debugging code, great. Save the files. Know the version of the game you were using.

Be ready to help your team member who has to fix your horrible awful crash that happens one out of every four times, and you're not sure how you made it. Be willing to help them recreate it, because it can be really hard to recreate bugs.

Finally, know the difference between a criticism and a critique, and make sure that your reports are critiques of the game and the bugs and not criticisms of your fellow developers and their ideas. And when I say criticism, what I mean here is an unsupported opinion usually delivered in an unprofessional tone. These purple elephants are junk and it's totally stupid that they fly.

A critique is a statement of opinion about the game backed up by examples and hopefully facts. The purple elephants make the game too easy because they can fly. I can skip almost a third of the game play in the bottom area of the screen. There's a problem. There's a reason why it's a problem. And there's a fact that you can talk about, and you can have opinions.

And it doesn't matter that the purple elephants go in the game or not. You can just talk about what they're doing in the game. You don't have to talk about the person who put the purple elephants in the game at all.

So user testing. I think we've actually covered this one to death. But if I did a lecture about testing and didn't stop and talk about why you're doing it, it would be a little odd. Once again, every time we say iterative design, that means you make a design, you test it, you get the data, you move on without actually testing it with users. You don't get real data, et cetera and so forth.

So I think that drum has been well and truly beaten. But here's a warning we haven't really talked about yet, which is test-driven design. Does not mean do what your testers tell you to do. Test-driven design means gather data from your testers, analyze it, and act on it.

90% of all MIT students will tell you that your game is too easy. They're wrong. 90% of all people who like to play first person shooters will tell you that your game needs a gun. If you didn't plan on having a gun in your game, it probably really doesn't need a gun.

Testers are often full of solutions and suggestions to make your game just like the game they wish they were playing. But you're not trying to make that game. You're trying to make the game you're trying to make. So when you gather your data, put it through the filter of what your vision of the game is and what you intend your game to do, and don't let your testers drive your game. Let your vision and your data drive it.

So how. How do you get ideas from testers? And how do you actually turn them into something useful? So you all remember the experimental method. It's actually really pretty similar.

Going to form a hypothesis, or ask a question. Figure out how to get the data to that question, collect it, and analyze it. So forming a question. What do you need to know about your game that your team can't answer? The more specific you can be, the easier it is.

Sometimes it is hard to get specific because you've got a game and you've got a whole lot of interlinking systems, and it's hard to figure out which one is making things not quite right. But do your best. In there a particular mechanic? Are people having a hard time or an easy time getting started on your game?

How engaged are people in your game play? Are they playing the game and grinning, or are they just sitting there and hitting the keys? How many of your players complete your game in about the time you expected them to? How many of them complete too quickly or take too long?

So when you're running scientific experiments, you try to standardize your conditions as much as possible to get data that can be compared with each other. When you're collecting qualitative data, which is what most focus testing data is, this is actually pretty hard to do but you want to try to do it anyway.

So some of the ways you can do that is by make sure you're giving each player the same set of starting information by having a script to introduce your game, to give them any information about the game they're going to need because your game is in too early to have a UI. Games often don't have UIs that are particularly useful while they want to be tested early.

That's OK, just tell your players what your UI "should" tell them. You can double this up with user testing by giving them pictures of the intended UI, and you can see if they can figure it out from there or not. But that's advanced testing tricks.

Have a dedicated observer. Don't depend on their memory for what they're looking for or what they see. Give them a list of specific things to look for that you think will answer the questions you need to answer. Give them space to write it down with. And give them the autonomy to stand back and just take notes while someone else offers your player assistance or introduces them to the game.

And finally, you really can't observe everything. Think about what your players might be able to tell you that you can't see. Write down a few questions for them to answer afterwards if you need to. Keep it short. You don't really want to make somebody write an essay while they do your testing.

If you can encourage your players to talk to the game or to talk out loud while they play, that's the greatest thing ever. The best people I've ever heard play games are middle schoolers because they connect their brain straight to their mouth without any filter, and they're just there.

They're telling you what they're doing. They're telling you why they're doing it. They're telling you why this particular aspect of your game is the worst thing ever, or why this particular aspect of your game is the greatest thing ever. And you can just sit there and listen to them and you know why they're doing things.

Adults are a lot quieter. We often don't want to explain why we're doing something. Often it's because we don't know we're supposed to be doing. And we don't want to admit that I don't know what I'm doing. I'm just going to poke stuff until I figure out what's going on.

So that's where you want to watch people. And if you really need to, you can lead and say, why did you do that thing? Or even better, what are you thinking? People will often answer those questions. But you really want to keep those prodding questions neutral.

Don't say, why did you do that crazy thing? Or, what are you looking for back there? But just, what are you thinking? What are you doing? What's your plan? That sort of thing.

Once you've got your data the team can review it together to notice trends, spot common problems, and so on. And that's where the things that four or five or a majority of your testers did or said or ran into, that's what you're going to be pulling out and using to think about how you're going to fix those problems. Or if you noticed a trend that you hadn't noticed before that people were really liking an aspect of your game you haven't thought about, it's something you can work on magnifying so that you can share it with everyone.

So for those of you who fell asleep while I was talking, the too long didn't read version really is who are you testing with, because your audience will affect your data. What are you testing with? The state of your game will affect your data. That shouldn't stop you from testing with early builds, but it does mean you need to adjust your expectations and make sure your players are ready to play with it.

And finally, why are you testing? Only you can answer that question. We can give you a lot of reasons or suggestions, but if you're running a focus test it really needs to be because you're looking for data that your team intends to use. And you can always use data to support your game.

You really can. Even if you don't realize that you need it, you do. And I keep saying "and gather data." And I talked about observations. I talked about surveys. I talked about interviews, which are our three major tools unless you are going to go to all the work of videotaping and audio recording and getting releases from your testers, which I really don't recommend. That's more work than you should be doing.

But the three major tools really are observing people and taking notes, asking them a standard set of questions afterwards-- either on paper or in person-- and thinking about what kind of data you can gather best with each of those methods. If you have 20 different levels you want someone to rate at a one to five as to whether it's good or not, you should just give them piece of paper and when they finish each level, they can check a number between one and five.

If you want to see how long it takes someone to get started playing the game, you probably just want to watch them with a timer. How long does it take them to get from that first screen to getting into the game? And finally, if you want to get a general impression, sometimes talking to people helps bring out things that you didn't think of.

You can get some insights you wouldn't have thought to ask for by talking with people. But be a little careful talking with people, because people don't want to tell you bad things to your face. They'd rather write the bad things down on a piece of paper where they don't have to admit that they said bad things about this game.

That covers less for middle school students, as I mentioned earlier. But most people don't really want to say mean things to the person who made the game, because we don't want to make people feel bad. So you need to balance that out when you're asking people questions about well, how did you like the game?

The answer is usually, oh, it's great. Yeah. So having run through all of that, let me fight with my computer and bring up a couple of examples. That's not the right form yet.

So this is a script created by students by a team that had a dedicated QA lead person. I do not expect your sheets to look anything like this, but I want you to see an example of what someone did when it was their job and they were taking time to get things ready. So you can see how serious we are.

And so you've got your observer instructions. And what's day's goal? Finding out if the feedback in the game works, along with a couple of other things. And there's a testing script written out pretty clearly along with some basic instructions.

Since these teams had a dedicated focus test lead, they had someone who'd spent a lot of time thinking about testing and a lot of people who'd spend a lot of time working on the game. So the lead tester was in the position of trying to make sure that the team understood how to run tests, because they weren't experts at it either. So I just embedded all of the good advice right into the script.

And there you have it. And then there's a sheet to write notes down. They've got the big questions that they're worried about, so they can write down. They've got space for other notes. And it means that whatever they've got, they've got.

And then finally, it follows up with a fairly short questionnaire trying to get some answers to questions they're wondering about. The specifics are always going to vary from game to game. Looking at the focus test sheet from one game isn't going to help you wildly with yours. But you can think about the way the questions are worded and the number of questions so that you're not overloading people.

 

I believe these are up on Stellar as a sample if you want to download it and look at it as an example. Finally, the paper we expect you to do. So this is the focus test report which is also download-able on Stellar, I think in either Word or Rich Text Format so you can use it. When we say we expect you to turn in a focus test report, this is what we're expecting you to turn in.

If you take a look at it, it walks you through the whole process and there's a space for most of this stuff. So what's the goal of the focus test? Record what happened at the focus test and what are you going to do afterwards.

When you are planning your focus test for this afternoon later in class, I recommend you download this and use it as your sheet to record notes on, because it'll help make sure that you get the data that we are asking you to get. And I think that's actually most of what I have to say about testing. Are there any questions?

PHILLIP TAN: I have a question. What's currently using some bug database, and if so, what are folks currently using? Or let me rephrase that. Who has used a bug database before? What are some of the ones that you're familiar with?

AUDIENCE: Chehra.

PHILLIP TAN: Chehra?

AUDIENCE: Bugzilla.

PHILLIP TAN: Which one?

AUDIENCE: Bugzilla.

PHILLIP TAN: Oh, Bugzilla. Yeah.

AUDIENCE: Buganizer.

PHILLIP TAN: Which one is it?

AUDIENCE: Buganizer.

PHILLIP TAN: Buganizer?

AUDIENCE: Yeah.

PHILLIP TAN: Oh.

AUDIENCE: I use [INAUDIBLE].

PHILLIP TAN: Yeah.

AUDIENCE: [INAUDIBLE] That's the [INAUDIBLE] database, yeah.

PHILLIP TAN: And as Sara mentioned, you can also just use a Google spreadsheet or something.

AUDIENCE: [INAUDIBLE] is a great bug database too.

PHILLIP TAN: There is always this intermediate step between someone entering a bug into a bug database and then deciding that they're going to fix it, right? So priority, somebody actually says I think the bug that the characters are too short or something, and then the art director says no, not a bug. Will not fix that.

So just because you're entering something in a bug database doesn't necessarily mean that your team's committed to actually spending time on that. But when a team does decide that, OK, this is a bug. Maybe because it crashes the game.

Maybe because it's actually not how we want the game to perform. It needs to be entered into your task list. So that means your task list is changing and you need to be able to put that in a priority.

Possibly, this bug is more serious than some of the planned tasks that you already have on there. So you need to rearrange that so that [INAUDIBLE]. Something else that Sara mentioned very briefly, but I want to stress, MIT students are actually horrible testers. I've heard this many times from many different companies here and [INAUDIBLE].

MIT students are not a representative population of the general public. You look at systems differently. You look at screens differently. So even though most of your testing that we're going to be doing this semester, I imagine that a lot of testing you're going to do on your own is going to be with other people in your dorms, for instance, and that's fine for this class.

But don't think that it's normal to use even university students as your only test audience. That's really not representative of the general public. MIT students in particular.

PROFESSOR: MIT students do [INAUDIBLE].

PHILLIP TAN: The final thing, as Sara mentioned, user testing in two contexts. One for overall category of you are testing with users and then it's broken down to more specifically, focus testing and user testing. And you may be wondering why user testing is used and that [INAUDIBLE] two different things, as Sara said, it's good to be more granular.

What we normally refer to as usability testing, as Sara mentioned, has only fairly recently become the standard name for it.

PROFESSOR: Yes. Yeah.

PHILLIP TAN: Back in the '90s and even the early 2000s, user testing and usability testing were basically the same thing, because we were talking about websites and enterprise software. We were not talking about games.

So the only kind of user testing that you'd be doing, putting a piece of software in front of someone, getting feedback, was usability testing. That's why they had the same name for so long. But now there's focus testing, which is very, very applicable not just for games but also for things like user experience. I had not heard that term user experience [INAUDIBLE].

So where it's not just can the player figure out what to do, but the person actually having the desired experience for playing the game. That's why now there's multiple names for that.

PROFESSOR: Thank you. Then I think I'm out of here. I'm going to sneak off with my computer.

GENEVIEVE CONLEY: Hi, guys. My name is Genevieve Conley. I'm a researcher at Riot. And some other terms that you might have heard for that, user experience researcher, user researcher, or the way we were just talking about it, like user tester basically.

And a little bit about me. I graduated from MIT in 2010. I got my degree in Course 9. And I actually worked at the MIT Gambit game lab on a couple games there. So that's actually where I got my start doing user testing, was in the summer projects we would run play tests with local community members, have them come in, and that's where I actually did my first user tests.

So I work at Riot Games. How many of you guys have heard of League of Legends before? OK. How many of you actively play League of Legends? OK, good. So I'll give a little bit of context on the game just so I'm not losing anybody. And I'll try to make sure I explain all of our jargon, because sometimes when you're in a game company you just start using the language like everybody speaks that language.

But a little bit about Riot is we make League of Legends. And our core mission is that we aspire to be the most player-focused game company in the world. And this is going to be relevant to what I'm talking about today, because you can't really be player-focused if you don't talk to your players.

So what this means at Riot is we think about all of our decisions in terms of how much player value it adds, or how much player pain something negative causes. So this is like-- I don't know if any of you familiar-- we have an eSports scene, and right now we're in the middle of our world championships. And we created a video and a song that would basically, we hoped to be the anthem of the world championship.

And this is something that a couple people at the studio thought was really important, so they just went out and made it. And the point was just to make the players pumped up about worlds and to give them a song that represented how they feel. And this is just a small example of something that happens at Riot where we really just want to do something that we think players will think is cool.

And one of the ways if we do that is through play testing and user testing. And I'm going to use a little bit different terminology, so I'll make sure that I tie it back into the way that Sara's been talking about it as we go. So a little about League of Legends. It's generally a 5v5 game. It's a team oriented game.

So you play as a member of a team. And you usually play a specific role or position on that team, just like basketball. And you play as one of many champions. And it's online and it's competitive. So like I said, we have an eSports scene where you play against another team.

And you go basically across this map, try to take objectives, and ultimately blow up this thing called the Nexus which is a giant crystal on the other side of the map. And you play as a champion. And these are player characters that each have an individual set of skills, different play styles, and there's over 120 of them in our game.

So even though each game is session based-- you don't level up your character necessarily over the lifetime of playing the game, each session is a new session-- there's lots of different ways to play because we have lots of different characters, which I'll talk to in a little bit. So one of things that I talked about is it's really hard to be player-focused if you don't talk to your players.

And one of things at Riot is that we all play League of Legends. In fact, some of us play it a lot. Some of us play a lot at work. But just because you play your own game doesn't mean that you actually know what it's like for all of your players. And using MIT students is a great example. If you guys just play with MIT students, you might not know what it's like for people in other countries, for example.

So one of things we try to do is make sure that we understand our players broadly and know that we are our players, but we are a small sub-segment of our players. The way we use research is it equips us to understand player beliefs, their needs, their desires, and their relationship with the game and with other people in the game. And what this allows us to do is design experiences.

So there's specific experiences, interactions that we're hoping to design. So in general when we're creating a feature, we have a set of goals that we're trying to achieve. And what we want to do is we want to help designers. developers, QA, artists, we want to help them achieve these goals. So we want to know exactly what they're trying to do and then get them feedback on how well those goals are being communicated to players, how well they're actually able to interact with those, and then what their reception is as a result.

So the way I'm going to break this down is play tests and user labs, which is essentially the way we were talking about it before, is play tests and user tests. So play test is the backbone of creating your game. It's really, really difficult to create a game without play testing it. So I'm glad that's really part of how you guys are designing your games.

And play tests basically allow us to quickly gather information about a game or a feature to see how it's working. And typically, this is done with resources that are readily available. So people in your dorms, people in your class, or at Riot, we often do it with other Rioters, and even on the team.

And some of the benefits we can get out of this is we can identify technical issues. Now, Sara talked about a specific type of this, which is quality assurance testing, which is less what I'm talking about. I'm more talking about testing with your team to just quickly identify technical issues that might prevent you from playing that game.

You can also look at things like balance. So in a competitive game like League of Legends, you play against other players. And if one type of character is really overpowered or has a significant advantage over another type of character, we can quickly identify that through play testing.

So for instance, our designers have two play tests a day where they test their in-progress champions to see if there's anything basically game breaking like that that they can quickly identify and iterate on. We can also look at things for fun. I mean, this is the most obvious one. But it's really important when you're making a game, if the game is supposed to be fun, that it is actually fun.

And in our case, we do want to have a fun game. So we have a whole group of designers and developers that have a team which is known as the play team and they make temporary game modes. So I don't know if anybody played recently, but we had a big in-game event called Shurima, and we had this game mode called Ascension.

And when they were developing that, they were often several times a day-- many, many times a day, in some cases-- testing every iteration that they made to that game mode to make sure that it was actually fun. And a lot of times what you'll find is that your game isn't necessarily fun sometimes. And that's OK, because each time you make a change, you're getting closer to that fun experience.

One day, you'll be playing your game and like, oh my gosh, it's actually fun now. This is great. Let's do this. So it's a really good way to get that feedback.

And another specific area that we're looking for a lot when we play test is clarity. And clarity is a word that we use to mean basically how clear is the information-- either the visual information or the mechanical information-- how well is that being communicated to players? So this could be something like with UI, if you just have a picture and you show it to players or to your dorm-mates, your classmates, how easily are they able to identify what a button means?

Or which things are clickable and not clickable. Those things are all really, really important in a game like ours where we have five people on a team, 10 people on the map, each with their own set of abilities and visual effects. And there could be a lot of visual clutter if we didn't think about this type of thing.

And then the way we typically get these, at least at Riot, is like I said, team tests. That's the example I used with the designers and the play team. They're just playing within their team frequently to make sure that they can quickly and easily give each other feedback.

Quality assurance test is another type that's a very specific implementation of this, which Sara spoke to. And in house "guerrilla tests"-- and I always have to be careful how I say "guerrilla," not "gorilla." "Guerrilla."

This is when we bring other Rioters who aren't part of a team who maybe haven't had exposure to that particular feature. This is particularly useful for things like clarity or usability tests, because we don't necessarily need to have that perception or that sentiment that is really unique to players. What we need here instead is just see, hey, how easily is this usable for a reasonably tech savvy person?

But even then, of course, there are some cultural differences. There's some hardware differences. So it's a first phase when you do them as guerrilla testing. And you always want to make sure that you follow up with real players after the fact.

So user labs, this is what I really focus in. So I use a lot of different methods at Riot, but one of the key ways that we're able to get player feedback is by bringing players to Riot to have them play the game. So this is a way to help us understand how real players will actually approach a feature. What we get out of this is really, really important.

We get reception and interest. Like, how stoked are players going to be about something, or what we call the table flip reaction. Which is like, we're working on an update to Summoner's Rift right now. And it was early on. It was really important for us to know, were players going to be really upset that we were changing something that's so key to the game.

So we made sure to loop them on very early in on the process to see how they were responding to changes we were going to make. We also look for things like usage patterns and errors. And this is important that we get a broad swath of different types of players to look at this type of thing, especially with things like we just updated the client. Making sure that the changes we made to the interface weren't going to confuse players-- experienced players who were used to our old version and new players who would be coming into our game for the first time.

We also look at needs and desires. And this is not like Maslov's-- Maslow's hierarchy of needs. It's more like, what are things that players really want from our game? And understanding that can be difficult sometimes because, like I said, we all play our game.

We're very entrenched in this experience. We have our own experience in the game that we feel very strongly about. But understanding that other players might have different needs and desires, and understanding what proportion that represents within our broader player base, these are really important things to understand.

How we get this is a lot of different ways that I'm actually going to go into in a little bit more detail. And each one has a specific application, but there are a few that we use pretty much in every user lab, which I'm going to go through right now. So the first is something we just talked about, which is just observation.

And I bring this up because it's something that anybody can do. You don't need a dedicated researcher like myself. You guys should and will be doing this. And so in this case, I have Chris, who is a developer at Riot, sitting in for a player. And I have [? Jisan, ?] who is another researcher at Riot, sitting over his shoulder watching him play.

In our case, we typically just have one researcher in the room both taking the notes and doing the observation, just because of resource limitations. And because we've done this a lot and it's our full time job. So in this case, [? Jisan ?] is watching Chris interact with the client. And you can see in his hand, he's actually got the script-- or what we call protocol-- in his hands so he can remember what things he should be looking for, what things he wants to say.

In this case, we don't generally try to use it as a script. We try to use it as a memory refresher. That way, players can feel really comfortable with us and feel like they have a relationship and can have a comfortable conversation. One of the ways we achieve this is through what we call the Think Aloud protocol, which is what Sara was referencing, which is basically we try to make sure that they have this direct link from their mind to their mouth.

And there's a couple ways to achieve this that I'll speak to, but the goal of this is we want to know exactly what the game is communicating to players, and then how players are processing that and creating an internal model. And the best way to do that is just to step back and let them tell you exactly what they're seeing. And if you aren't quite sure what they're seeing, then you can follow up, which I'll get to in a second.

So here's an example of a player who's seeing our character Lucian, who is a champion who was released fairly recently in the game. She's seeing him before he had been released in a work in progress state and giving us feedback on what she sees.

[VIDEO PLAYBACK]

-I can see that he walks faster than the normal--

[END PLAYBACK]

GENEVIEVE CONLEY: Sorry, we forgot to plug in the audio. I think you guys could hear it all right. All right.

[VIDEO PLAYBACK]

-[INAUDIBLE].

GENEVIEVE CONLEY: Oops.

-I can see that he walks faster than the normal character would, but his animation doesn't make it seem so. It seems like he takes a really big stride, but I was imagining him to walk really fast. But it seems like it looks like that way.

[END PLAYBACK]

GENEVIEVE CONLEY: So this is an example where the player has just brought this up to us that she expected him to move one way, and he's actually operating in a different way. You can think of this is as like, basically a violation of expectations, which is often what we're looking for especially with our characters. Does what you see and what you hear about a champion match with how they actually play?

So one way you can do this is to just let them talk. And it is really important not to be giving feedback on whether or not you think they're correct, especially if they do have a question. It's also really important not to be judgmental, right?

So the way you phrase your questions, even if you aren't actually being judgmental, sometimes players might think that you're judging them. So one of the contexts we give players upfront is always for us, since we actually didn't work on the game or the feature that they're seeing, we can say, we didn't work on this. It's really important that you give us honest feedback. I'm not going to go home and cry at night, so I want you to give me totally honest feedback.

And we also let them know that sometimes we're going to ask them questions, and we're asking it so that we can understand what they're thinking because that's what we're interested in understanding. We're not asking it because there is a right or wrong answer. So one of the ways that makes it easier to make sure that we understand what their mental model of the game is is a follow-up with some probing questions.

So the player in the previous example was pretty clear about what she was seeing and why it violated her expectations, but sometimes players don't necessarily know why it doesn't actually fit what they expected. Or they might know that something's frustrating, but they're not necessarily giving you the answer why.

So one of the ways you can follow it up is either with probing questions, which you ask in the moment, or you can follow up with interviews after the fact. So here's an example of [? Jisan ?] following up with a player who just saw Lucian for the first time.

[VIDEO PLAYBACK]

-How was that play [INAUDIBLE]?

-I thought it was fun. I thought it was [? different. ?]

-What makes it fun and something different about it?

 

-So he, I guess, normal [INAUDIBLE] skills, just increased the intensity. A lot of his skills based off of move speed and [INAUDIBLE] speed, and I guess he feels [? confident. ?]

[END PLAYBACK]

GENEVIEVE CONLEY: So this is an example. A lot of times you'll find this, is that when you ask that initial question, you get a short response. And what we're always trying to do as researchers is invite a conversation where they're just really comfortable telling us exactly what they feel and giving us a lot of detail.

So in the first question, he asked a really broad opening question, which was "how was that?" He didn't even ask, "how did that feel," which might lead in a certain direction-- although that would be an appropriate question as well. He left it wide open for the player to take us where they wanted to talk about it.

So the player mentioned that it was fun and something different. So [? Jisan ?] wanted to get more depth into what does that actually mean, because that alone doesn't give us enough direction to be able to provide to our designers to say what was good, what was working. So he mirrored exactly what the player said back at him.

He said, oh, what was fun and what was the thing that made it feel different? And then the player, that way we're not actually putting our own spin on interpreting what we think they're saying is fun or maybe what's different. We're just mirroring exactly back what they said so that they can give us more detail.

You have to be careful with that, because you also don't want to sound cold and clinical like you're sitting in an office somewhere writing down everything they say, which you will be. You want to make it feel like it's a natural conversation.

So we're really lucky because we get to work with League of Legends players so we already have a connection built up with them where we can talk about it, but it's really important that you make that as hey, I want to know what you think. You're really valuable. Let's just have a conversation about it. And it can be really challenging sometimes.

So this is one that you guys are already pretty familiar with, too, is part of interviews that I've clumped in here is basically the questionnaire. Sara gave you guys some great context about when this might be useful. Here's an example of when we use it in League of Legends research.

So this is an example with Azir, who is a character who just came out in League of Legends. This is actually from quite a long time ago when he was in the concept phase. And this is something we do actually with most of our concepts, is we put them in front of players before they go into development to see, hey, is this something that feels like it fits in the world of League of Legends?

Is it clear where maybe the source of power is? And does this look like a champion you would actually want to play? Now, we don't use data to drive decisions at Riot. I want to make that really clear. We use it to inform.

So just because something wasn't resonating with players in our labs wouldn't mean that we would cancel a character. It's just a way for us to get feedback for our designers who already have their experience and the experience testing with other Rioters to help them make informed decisions. So what's useful about a questionnaire in this case is that we can compare across champions over time and across different roles or positions and see what is resonating and what does good look like.

We also use this as a jumping off point. So we start with a questionnaire, and then we go into a more detailed interview about it to make sure that we're getting enough detail to be able to provide useful information to the designers. So you guys are also probably pretty familiar with the idea of experiments in general. But part of research testing can be devising experiments.

So we often hire people from psychology backgrounds or research backgrounds in academia because they have experience setting up experiments. So this is like the classic one. I don't know if you guys have already covered this, the Engelbart experiment. OK.

So this is foundational human computer interaction. It's one of the most cited studies. And it's English, Engerlbart, and Berman basically were looking for a way to improve a way to interact with a screen or text manipulation. And what they proposed was something that was radical and was called the "mouse." Maybe you've heard of it.

So that thing in the right, I know it's not a really clear photo, but that is one of the first early prototypes of a mouse. And what they wanted to do is they want to see, is this better than some of these other interactions out there. Qualitative experience alone might not be a very strong way to prove it in this case, especially when you can collect quantitative data to back it up.

So what they did is they used the time that it took to complete a task, a very specific task that was given across all these different ways of manipulating text. And they saw how many errors the user made in this case. So you can see that the mouse did pretty well in terms of time overall. It didn't take the participants very long to be able to complete the task.

But it had very few errors, which made it overall the best out of these different ways to select text. And you can see some of the other ones on here were things like knee control, joystick, a graphicon, which is like an arc drawer, a light pen, and a joystick. And all of these were ways that we could have gone instead of the mouse.

But based on some of this research, they were able to improve their design and now you can see I'm using one right now. And as part of the experimental approach, I broke it out, but there's a specific way you can look at it, which is using physiological techniques. And there are many, and I'm just going to cover two.

And typically, these are used for very specific types of research. They're very exciting. So oftentimes, you'll see people want to use them on everything. But they're very particularly useful for certain cases.

So this is an example of eye tracking, which is basically fundamentally, we put a camera up that's able to specially track where eye movements are. And there are more sophisticated ways you can look at this-- and I'm sure a bunch of researchers at MIT could tell you a lot more about it. But in this case, it was pretty simple.

We were just looking at where people were looking on the screen and how often and how long they were looking there. So you can see the red parts of the map here are the parts where there are more intense gazes or more gazes overall. So unsurprisingly to probably most of the League of Legends players in here, or you guys as gamers in general, the mini map has a lot of gaze.

The bottom part, which is where the abilities are displayed, also has a lot of gaze. That has a lot of information that's really relevant to players like the cool down timer, basically when the ability is going to be ready again. And the center of the screen is where most of the action happens.

So this type of information is helpful for us. If we want to, for instance, introduce new information on the screen, we'd want to know, hey, are we going to introduce this in a place that's going to get a lot of traffic naturally. But also is it going to interrupt things that that gaze is there for a very important reason, and maybe we shouldn't interrupt that. So we can help us to inform those types of decisions.

There's also many other types of physiological, and I'll just hint at one right now which is things like galvanic skin response, anything that has to do with physiological arousal, either for stress or excitement. These can help with situations where maybe players wouldn't be able to vocalize why something was frustrating or exciting.

Or it can also help combat some biases like recency effect, where if you ask a player at the end of the lab or at the end of a play test what was interesting or what was frustrating, they might preference things that were at the end. Or they might preference things that were really emotionally significant for them, but might forget smaller things that are still maybe small frustrating things that build up over time.

So none of this is really that interesting if I don't give you an example of how we actually do at Riot. So here's a case study. And I actually presented this last year as part of a talk that we did at MIT. So I apologize if a little bit is re-run for you guys. I'll try to spice it up a little bit as I go.

So I'm going to be talking about Lucian, who I introduced to you guys earlier. He is one of the player characters or champions in League of Legends. So where we always start as researchers, and part of the job that I haven't really alluded to yet is we spend a lot of time talking to our designers, to our developers, to our artists, to our producers, to make sure we understand the goals they're trying to achieve.

And I don't know how many of you guys got to catch August Browning's talk last night, but for the champion designers, they start with a very specific idea of what goals they're trying to achieve. What is this player character going to feel like? What are you going to feel like when you play it?

What is the mechanics supposed to feel like? What is the theme supposed to feel like? How does this resonate with players? How does it fit in League of Legends? These are all really important questions for them that they're constantly asking themselves.

And in the case of Lucian, he was supposed to feel fast, sleek, speedy. He was like a gun ninja. Like pew, pew. No? All right. Trying to see if you guys were awake there.

And he's a range damage character with finesse game play. And he basically has the high skill ceiling. And what that means is basically he's harder to pick up, but when you master him it's a really rewarding feeling. So that's a trade off between accessibility, maybe how quickly a player can pick him up, or what level a player can pick him up, for a reward of how rewarding that feels when you do master that champion.

And some of the challenges in this were, this guy, his theme is pretty different. His equilibrium style, if you've ever seen that movie. And that's not something we had necessarily done that much in League of Legends before, so we wanted to make sure, does he even fit. And we also wanted to know how do you let a player feel super fast and speedy in what is essentially an RTS style game. It's a challenge.

So this is what the test plan eventually looked like. It's not necessarily where we started from. But we started with concept testing, and then we went through actually three lab studies with iteration in between based on the feedback we got from players. And after he launched, we did some post-launch evaluation.

So step one, just like I showed you earlier, was to do this concept testing. And we got some pretty good feedback based on that. His fantasy was good with players. They liked the mood.

[THUD]

They felt like he fit. Are you all right?

AUDIENCE: Yeah.

GENEVIEVE CONLEY: And he made sense for the role of an AD carry, which again is like a range damage dealer in this case. So he had high appeal scores and the theme felt really strong, but one thing we very quickly identified-- it's a little hard to see on this one-- but his guns were different colors.

So players expected the guns to do different things based on their colors, which was not what we were planning. So this was very good early feedback before we got very far into development where we could pivot and make sure that the idea of the two guns doing the same thing was conveyed. So based on the feedback we got, we were able to give that to the designers and they continued their development.

And then after a little bit of time, we were able to take it into the lab. So we started with a Lucian that was designed to be a speedy range assassin. He had a double shot and a utility kit. And when I refer to kit, just to be clear, that means his set of abilities. And characters in League of Legends generally have three abilities and then an ultimate ability, where the ultimate is like the super big bad power that you get after some leveling of the character.

So I blew the reveal here, but in the first lab we found out that he didn't really feel that sleek and fast. He felt stiff. He planted and then fired his double shot, which made him feel like a turret. And people did like his visual theme. They thought it was compelling, but it just didn't match his play style.

So we took this back to the designers we said, hey guys. Here's the videos. Here's the notes. We usually make a highlight reel based on what we see to support our evidence so that the designers can get enough feedback from that, as they are our subject matter experts and really know how to interpret what they see. And we said, OK. Let's go back.

You guys take some time, do some changes. We'll take it in the lab again. So we did lab study two. And this time, we had a different Lucian to bring in. This time, he was more mobile. He was basically a range [INAUDIBLE], running gun, double shot guy.

Well, we had a little bit of a mixed bag of results here. He did feel a little bit faster. But some poses and animations felt awkward. And he was high speed in the numbers-- which I'll talk about in a second-- but he still felt slow for some reason. But worst of all, we got quotes like this one about the ultimate, which was "It worked in the way I did not want it to work," which is a pretty good sign that something's wrong.

So we took all this back to designers again. We said, hey guys-- and the animators and the artists in this case, of course-- and we said, he feels slow. His ultimate doesn't really have the usability that people expect, but the double shot theme is working, so we're definitely on the right track and players are still very excited about this champion.

So now we're in a bind, and this is often the case when you're working with research from players because they're not always able to articulate exactly what's wrong because they don't have the context that you do. So this is why it's so important for us as researchers to pair with our designers, our artists, our producers, and our players to be able to get the right kind of data to make informed decisions.

So here was our problem. The frame delay between his basic attacks-- which is actually very, very low in this last lab. Usually that means things feel fast. It's a pretty standard way when we're developing characters to be like, OK, this will feel faster. He also had a very high base movement speed, which means that the time it took him to run from point A to point B was pretty high relative to the rest of our champions.

And if that weren't enough, he actually had a dash ability which means that he could just jump basically from one point to another. All of these things should have made him feel fast. But players were saying he didn't feel fast, so there was something wrong.

So our designers and our artists and our producers took all of this information back, looked at the videos, play tested the heck out of it, and came to this conclusion. They increased the ability responsiveness and they changed how animation flowed from one state to another. So I'm not an artist, but what this means to me is that they are basically able to make the transitions from animation a, animation b, or ability a to ability b, and make those feel more natural. Not like me.

And they also gave more feedback to the double shot. And again, this is where subject matter expertise is so important because our animators, our visual artists, our designers knew this bag of tricks to be able to apply to it. And finally, they were able to make the ultimate ability clearer.

So they were able to the telegraph better what it was actually supposed to do so that players would have more accurate expectations. So we took it to lab study number three. We put it in and finally, the game play anesthetics fit the concept, and players were quite happy. But not everything is always happy ending, everything is perfect.

There was still some confusion about how the ultimate was supposed to work. We found that players wanted more control of it. Basically how the ultimate currently works is he locks his arms forward and shoots in that direction. So if you stray left to right, he continues shooting in that direction which, for some players, was pretty confusing. Now what they expected based on some of our other champions, and it wasn't able for them to learn it in one session.

So one of the things that we have to take away from that is labs have some external validity problems, which basically means that we're sitting in a lab. There's not other players around. They're often not playing on live. There's a lot of things that are different in our test situations than real life.

Usually, you wouldn't try to master a champion in one session. So this is one of those cases where we put it out to our designers and they decided, and we supported, that he was going to be a high skill ceiling champion, which meant that players were going to need some time to learn how to play him. And that just because they didn't learn him in one session didn't mean that they wouldn't learn him over time.

 

So this is what Lucian looked like when he did go live. And it helps if I play the video.

[VIDEO PLAYBACK]

[WARFARE SOUNDS]

GENEVIEVE CONLEY: So he's in the top there. Pew, pew.

[WARFARE SOUNDS]

 

GENEVIEVE CONLEY: So you can see his dash there.

[END PLAYBACK]

GENEVIEVE CONLEY: So yeah, sometimes a little confusing if you don't play the game to have to watch 10 champions all in one little team fight like that. But ultimately, the important part's there. He's using his ultimate and he dashed at the end.

Ultimately, ended up actually delivering on player expectations for this champion.

[VIDEO PLAYBACK]

[END PLAYBACK]

We don't need to watch that again. So what did we learn? Well, we knew going into it that Lucian was going to be a difficult champion to get right. And we also learned that, you know what, it's OK to take a champion back to the lab multiple times if we really feel that it's worth the effort.

We knew that his theme was compelling, but there was something that wasn't just matching up between his mechanics, his visuals, and his theme. And this is one of those things is that everything on paper can be totally 100% right, but if players don't agree with it or players aren't feeling those goals that you're trying to hit, there's going to be a problem.

So we're able to collaborate with our subject matter experts, with our players, in order to get the experience right. And I mean, I'll say it again. Just because the mechanics stacked up correctly does not mean that it necessarily will in the feeling, which is why you play test it in house, why you play test it with other people at your studio or at your school, and ideally, why you test it with the people who are actually going to play your game.

And then we also learned that you can sometimes have these trade-offs. It goes back to what you hear from players. Sometimes they're going to make suggestions and you have to look for the note behind the note. It's like, what is actually the problem?

You guys are the subject matter experts. You're going to be able to look at what they're saying and never take it as, from players, a criticism. Always think about it as if they said something wrong, there's something that has to be causing the problem. What is it underneath their note that's actually causing that problem? Let's look for that thing.

And we're still iterating after launch. That's one thing that's typical with our champions is that yes, we do play tests with players. Yes, we do play tests in house. But once we launch the champion, there are thousands and thousands of play tests with real players on live. And that's when we learn about things that happen maybe at scale or under very specific situations, or for things that we didn't account for, for these external validity concerns.

So we keep iterating. And in the case of Lucian, we actually continue to collect metrics and user sentiment through things like surveys and through game data to understand how he was doing. And based on this, we're able to make more iterations, fine tune, and hopefully make him a little bit more accessible.

And today, Lucian is well regarded, widely played. And if you watch the professional matches in the world championship, he's use quite frequently. All right, you guys, I hope I didn't bore you. My name is Genevieve, again.

If you guys ever have any questions, you can feel free to reach out to me at gconley@riotgames.com. My Summoner name in game, if you play, is RiotCapernoited. And you can also find me on Twitter there. I'm always happy to answer questions. Thank you guys.

[APPLAUSE]

 

AUDIENCE: We get a 10 minute break.

PROFESSOR: So 10 minute break. Come back at 2:20 and be ready to start working on your focus test stuff. OK, welcome back to class. Now that you've had a chance to be talked at a whole lot, we're going to go ahead and ask you to actually do.

The plan is to give you about 10 minutes to meet in your groups, plan your test. I actually strongly recommend you just grab a copy of the focus test, either in the Word version or the RTF version of off Stellar so you've got that as a guide. Sit down, plan your test and how you're going to run it.

We'll give you 10 minutes for that. Then we'll have about 45 minutes for people to circulate and test. And one of the things you should also be planning is when your team members will be out testing other people's games and when your team members will be observing.

Just given the number of people in class, the number of games we have, and the number of people on the team, chances are good that you aren't going to be either observing or testing all the time. You're probably going to have to wait. But go ahead and circulate, take the time to watch other teams doing focus testing.

Try not to watch a game in progress that you're going to focus test. If you realize that you're just sitting around waiting for a chance to focus test, let the team know that you're in line next. I'd rather you pulled out a book and read it than watched someone else play a game you're about to focus test. It will be more useful for the poor people who you're testing for if you haven't already seen somebody do all those things.

That doesn't mean hide in the corner and read a book because there's too many people wandering around testing games. We do expect everybody to participate both sides. Observe and test. So try to balance that out a bit.

We'll circulate and keep an eye on things. And if it looks like at some point nobody's actually testing, you're all sitting around twiddling your thumbs, we'll end it early. I don't know how long this testing session will take because we haven't had such large teams on this project before. So it's a little harder for me to know what the amount of time is.

After that, we'll give you some more time to talk in your own teams and come up with the what are the results of your test to analyze your data. And then we'll ask somebody to come down and for each team give us the summary of your focus test report.

How did your focus test turn out? What did you learn? Do you have a plan for what you're going to do with it, or are you still figuring out how you'll deal with it? Those are all reasonable things given you only had 10 minutes to look at your data.

Thank you. It is 2:23. At 2:33, we'll go ahead and start focus testing. Actually, I'm going to give you 15 minutes to get set up, because I just realized you probably need 10 minutes to talk and then probably five minutes to make sure you have a machine up and running. So I apologize, it's going to be 15 minutes to plan and prepare for your test. So that's 2:38 at this point.

[CHATTER]

 

AUDIENCE: Yeah. I think he just wants to--

PROFESSOR: OK. You should be ready to go now, I hope. And actively go ahead. You should have a computer ideally set up to run your game. Who is actually ready to run a focus test?

PHILIP TAN: Everyone who is manning a demo computer, a computer ready to run the game, put your hand up. All stand up so that we can see you.

PROFESSOR: Yeah, go ahead and stay up so we can see where you are.

PHILIP TAN: I'm not seeing any hands from these teams over here. OK. All right. All right. All right. As I said that, someone is standing up back there. How about the Sparkly Redemption team? OK, all right. There's someone there to run a game. All right.

PROFESSOR: So close all the laptops that are not actually running the game would be my advice, and then let's go ahead and get started.

PHILIP TAN: All right. Start playing each other's games. Go.

PROFESSOR: Yep.

[CHATTER]

 

PROFESSOR: Next time, we'll ask them to have two or three.

PHILIP TAN: Yeah.

PROFESSOR: As part of the Stellar thing. He was talking about the--

AUDIENCE: I got the FOA for free.

PROFESSOR: OK. It's OK if you don't have a plan of action as you're looking at your results as long as you've gotten ideas to what you're seeing out of your data. It's hard to look at your data, get it evaluated, talk about it, and figure out what you're going to do about it all in 10 minutes. We are giving you a short amount of time because we're running out of time.

That said, let's get started with Sparkly Redemption.

[APPLAUSE]

AUDIENCE: So we had a small hiccup when we started where we didn't have-- part of our core game is collecting sparklies, and we didn't have a version with that when we started play testing. So we actually got to test out some different movement controls that we were trying out, which actually turned out to be good because the controls that we liked, which was right clicking with the mouse and then left clicking to shoot, a lot of people had a lot of trouble with that.

So it was a good thing that we messed up and then got some information from that. Then we got a version with sparklies working, and there was a big bug where if you killed all the monsters, they never respond. So that bug has now been fixed in the last 10 minutes, so that was good.

Other than we, we got just random things like people staying in the corner and just shooting, as opposed to just moving around the map like we want them to. So we have a few problems that we have to figure out that. Right now, we don't know the answers.

[APPLAUSE]

 

AUDIENCE: Hi. I'm with Lost Underground. As a refresher on our game, the idea is that you are lost underground in the darkness trying to find an exit. And so as part of our game, we have this sphere of visibility around you. And as we play tested, we got a lot of good feedback on what that visibility means to people and how we can actually use it to make our mechanics and our [INAUDIBLE] design more compelling.

Another big issue we ran into with people play testing was our collision boxes are basically pixel perfect. And so people would get caught trying to go through the map, which was really helpful for us, because although we already knew that this problem existed, we actually learned a lot about how people were actually trying to make these decisions.

And we learned that instead of using the idea of the entire [INAUDIBLE], a lot of players were actually looking at where the feet were planted and trying to move based on that. So when we go and fix it, we'll make sure to put the collision box around where the feet are. We also got a lot of good feedback on the items that we have in our game and where people are actually focusing their attention.

[APPLAUSE]

 

AUDIENCE: [INAUDIBLE].

PROFESSOR: Oh, Plunder Winds. Yeah.

AUDIENCE: So our game involves moving around a grid and running into random encounters. The favorability of the random encounters are indicated to the player by numbers and colors on the board. So one of the biggest things that we got from testing was that people didn't understand the role of the numbers and colors played or how they were related or if they were related.

And they began by moving around randomly and just waiting for things to happen to them. There's also a mechanic in our game that involves moving with the wind and against the wind. And so moving with the wind doesn't count stamina, but moving against the wind does.

Or moving sideways of the wind does. Moving against the wind is not possible. People had trouble figuring that out as well. So one of the biggest things we're going to try and do is more visual feedback to the player, help them figure things out as it goes.

Also, people felt like there should be more long-term consequences instead of just planning for the next move or two. And so we're going to see how we can integrate that into our game.

[APPLAUSE]

 

AUDIENCE: Hi. Blind Aliens. So we got a lot of great feedback from the use play testing. So one of the main things we realized is we needed more clear goals and more long-term goals.

So for example, our gun automatically reloads, and that was a little confusing for people because they didn't really see that it was reloading. But instead of doing something like that where the gun automatically reloads, we're thinking of adding bullets around that you have to go and collect so it's more short-term goals and also long-term strategy. How do you save your bullets up, when do you use them all.

Another thing we noticed was that our game was way too hard at first, and way too easy after five minutes. So we decided we need to focus on ramping up the difficulty. Instead of starting with three aliens attacking you, maybe only one. And then maybe after five minutes, instead of only having three aliens, maybe 10. [INAUDIBLE] difficult.

So another thing we focused on was our AI. Not AI. User interface. So I talked about how there was a reloading thing, it was flashing in the corner, but no one saw it so no one knew that their gun was reloading. They thought it just wasn't working and they were frustrated that they were clicking and nothing was happening.

So we need to make that more visual. Along with our aliens, when they're half-dead or they're still attacking you. People thought they were dead. They'd run into them and they'd die and be frustrated.

So instead of maybe turning them darker, maybe turn them red or something like that to make that clearer. So thank you guys for all the feedback.

[APPLAUSE]

 

PROFESSOR: Modudice.

AUDIENCE: Hi, we're Modudice. So a lot of the feedback we got was really based on our UI. And it's because we're modeling a 3-D die using 2-D graphics. So a lot of people have different opinions about how that should look, and no one thought that the way it does look is OK. [INAUDIBLE].

And so we really got a lot of opinions. Maybe you could do it this way. Maybe you could do it this way. And that really helps, because now we can really build up a discussion of how it should look. How we can make this mental model in people's heads match what they're seeing on the screen, which is the real hard challenge that we're facing. So thanks for the feedback.

[APPLAUSE]

PROFESSOR: Beaver Evolution.

PHILIP TAN: I told them it was going to be the problem, someone needs to go up.

AUDIENCE: So through play testing, we really realized that we weren't conveying to the user exactly why all the instructions worked clearly and what was happening at each turn. But otherwise, the game mechanics worked pretty well.

So in Beaver Evolution, you can choose whether to build, populate, or evolve, but it wasn't clear to the player exactly how many beavers you can populate or how many dams you could build, or what natural disasters could happen. And then between each turn, when a natural disaster happened, there wasn't that much engagement between the player and the game being like, oh, this drought happened and then this is a consequence of that.

So I think going forward, we're going to really try to make it easier for players to understand what happened during that turn, and then what their choices are in terms of picking it. So a lot of it is clarifying what the instructions are, what the game mechanics are.

[APPLAUSE]

 

PROFESSOR: And finally, Comcastic.

AUDIENCE: Well, the two main pieces of feedback that we got from players were about the UI of the game. And we had some issues with that because our game has a lot of complex UI requirements and we didn't have time to implement all of those prior to this test session.

But we did get a lot of feedback about UI improvements that would make it easier for players to see what was going on with the game. For example, highlighting houses that are currently being served by a [? source ?] center. The other main complaint that we got was that the game is nebulous right now.

There's not a whole lot of challenge or clearly defined goal. And so one thing that we're really going to look at going forward is tightening the focus of the game, addling more tension, and making the goal of the game more concrete. And since it doesn't fit very well into an established genre, we had some more complex design decisions to figure out how to make this a fun and engaging game.

And so that's pretty much what we're going to be focusing on going forward, would be the UI and adding more challenge and tension to the game.

[APPLAUSE]

PHILIP TAN: I have a few words. Do you have anything to say?

PROFESSOR: You go for it.

PHILIP TAN: OK. So I just want to remind everybody that you started on this project less than 10 days ago. OK. And you just finished your first playable prototype, and you did a good test. So give yourself a hand.

[APPLAUSE]

 

Let's see. Things to keep in mind. For those of you who are making mouse-driven games, you want to make sure that you have enough mice to have for your next play test which will be probably project three. And you've only got another five days, if I'm not mistaken, when this is done. This is exactly a two week project.

So if you haven't started cutting features, this is probably the time to do it. Let's see. Something that, of course, we did notice and you all felt it was that there weren't enough stations for playing the games. For future play tests, for project three onwards, we are going to expect at least half as many game computers as you have at the average team size.

So I'm not sure whether we're doing with teams of six or teams of eight. We're doing teams of six for project three, which means each team should be prepared to have at least three computers running with the games. Something else that we're also going to make it a lot clearer where the game stations are is that the computers that are not being used for game play should actually just be closed.

Take notes using paper, because it gets confusing to figure out which are the game stations and which ones are just like people taking notes and which ones are the ones where people actually coding on the sly trying to fix last minute problems. You really should have all that sorted out before the play test, not during the play test.

 

So next time when we're doing play test, just make sure that half your team, basically, is ready to go with the game. And I would suggest more than half your team should be able to run the code, just in case someone falls sick, right? And doesn't bring their computer or their mouse.

Rest of the class is for you to work on your own team. If anyone hasn't signed into the attendance sheet, we still have it up here. And I want to have everyone give the folks from Riot a big hand for coming and giving their feedback.

[APPLAUSE]

Thank you.

GENEVIEVE CONLEY: [INAUDIBLE].

PHILIP TAN: Oh, yes. They have lots of information about internships. Oops. Oh, whoa. A very, very large stack of documents and positions that are available. And a deadline, October 15th, which is not that long from now. So definitely check those out as soon as you can.

Let's see. Actually, while we're on internships, there's one other thing that I wanted to mention. I sent out a message for Europe over in AeroAstro. They are looking for game developers. Even better if you also know MATLAB, but they just want game developers for a high school science competition that they're running.

It's like first robotics. It's a robotics competition, only in zero gravity. So if that sounds cool, you should definitely respond to them quickly because Europe deadlines are coming up. And that's it. See you [INAUDIBLE].