Postmortem Analysis

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from 100 of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

RICHARD EBERHARDT: Hi, Pablo.

PABLO SUAREZ: Good to be back.

RICHARD EBERHARDT: Nice having you here. So we wanted to talk about a number of high-level postmortemy things for how the class went. So we've got some high-level things we wanted to talk about. We want to talk about the collaboration between MIT and the Red Cross/Red Crescent group and your folks-- how student preparedness went, in particular, comparing those teams that got a lot of support, like client support, like Snap, of course. But also, some of the other teams got a little bit more, and then the unsupported teams, which aren't exactly unsupported, but they got less, so that's largely the Heatwave Team, I think was the main one that we want to talk about there.

So collaboration what do you all think about how we talked to each other? How we met with each other-- things like that?

PHILIP TAN: I like the way how we started in the beginning of the semester. We had to talk-- and we had some Skype issues. But I think the idea of kicking off the entire class with the initial presentation of this is what your final project is going to be. It's going to take us a while to get there, but you come into class understanding this is what it's going to be. And Pablo being the person to describe why this is going to be important, I think, set lot of expectations well.

So I think later on, when we actually got to that project, the degree to which each team actually could have been supported actually became more of an issue. The teams that you could directly talk with-- and you were going to be the primary user of the game-- clearly benefited. But you were right there from the beginning. These are the teams that, maybe, they did receive support, or some of them didn't receive support, but weren't talking with people who had been there right from the start, I think maybe had felt a little bit more disconnected from the main thrust of the class. At least when they got to that final assignment.

So something I like to think about for future classes is the extent to which-- if we're going to repeat this format-- where the primary client comes in right at the beginning to set the tone for the entire class. How do we represent more of the variety of different things that you can do right at the beginning? So that when students think about what they're going to be doing throughout the entire semester, they are actually keeping full scope of the project in mind, not just the one example that might be presented to them on the first day of class.

PABLO SUAREZ: And we also may want to have something like an engagement meter. Because I wasn't aware of the fact that some teams were feeling disengaged. I would show up, some people have questions, some other have less questions. And from my perspective, those who have questions I spent time and it was about the same chronometric time per group. I could see that some were more lost, but I wasn't aware that what was needed was more fire in their belly. So if we could--

RICHARD EBERHARDT: I couldn't see that either.

PABLO SUAREZ: Right.

RICHARD EBERHARDT: The students never reported it to us.

SARA VERRILLI: Yeah, I was going to say, when we asked the students to do their postmortems, they were really clear about how this had been a big motivation, and interest had been a big problem for them. But they never mentioned it to us during check-in meetings when we were doing play testing, when we were working with them. So that's the sort of thing that we need to-- probably one of the things you do is probably to warn them about that. These are hard topics, and they can become boring topics very quickly. And they need to step up and let us know when they think they're facing a brick wall.

RICHARD EBERHARDT: I think the clues were there. If we had means and a way to remind ourselves, like a checklist or something simple like that, just to say, are you engaged? What's your motivation? And, again, that's assuming that we even want that to be our job. I think for anything client based, it has be our, probably, our job.

SARA VERRILLI: I think that when we're asking them to work on projects where they have relatively little control over the topic, I think we actually do need to. Because I think part of it was I don't think the teams knew how much freedom they had to deviate from the topic they had chosen and the specific. They thought they had chosen this thing, and now they were stuck with it. Even though one team was like, we chose a topic that just wasn't deep enough. There wasn't enough there. And if they had come and talked to us, we could have given them the freedom to well, go change to a new topic, or how can we go--

RICHARD EBERHARDT: Go deeper.

SARA VERRILLI: --how can we go deeper in it? Right. But we were not sufficiently warned.

PHILIP TAN: But that's where rapport with someone who knows a lot about the domain can help. And if they don't feel like they have that rapport or someone who they can readily contact to ask even seemingly silly questions, then we're not going to be able to get down to actually ask the important questions. Because sometimes they can't tell. They can't necessarily tell what looks like flagging motivation on a part of the team or what just seems like end-of-semester pressures on the students, right? They haven't been working on enough projects to be able to tell the difference. And so we do need to help them through that, I think.

SARA VERRILLI: By that you mean the students, then?

PHILIP TAN: The students. I mean the students.

PABLO SUAREZ: And one of the things you may want to consider is we know that both in the real life-- in the future, in their future, as well as in future courses-- there will be limited ability of those who are experts to interact with the game design team. The game design team, you took the effort-- and I thought it was very well-balanced-- to have some who know programming, some who know narrative, some who know user interface, and so on. They will, by default, not have expertise on heatwaves or cholera or etc or climate science. They cannot assume that all they need to know-- including the appetite for engagement-- will be only in the client.

MIT is a bubbling powerhouse of engaging stuff. And if they could, as part of what they are told-- they will have to do in the future, and what they have to do in this course, to reach out to engaging people, fellow students, and junior faculty or whoever may give them some time-- to just go and say, what is exciting about this? What are the cutting-edge areas? What are the institutional challenges. So that they also have to fuel their own appetite.

RICHARD EBERHARDT: Having an assignment or having part of that final project being like here's the research component that we expect you to do, but we didn't put an underline over it.

SARA VERRILLI: We didn't make it explicit that they needed to--

RICHARD EBERHARDT: We did when it was in trouble. Like for Saving Gora Gora, they were in big trouble. We told them a number-- you told them, we told them. I think you'd be surprised what they came up with. Because they actually, I think, they took the baton and went with it.

SARA VERRILLI: They did that [INAUDIBLE].

PHILIP TAN: Nice.

RICHARD EBERHARDT: The other cholera team did not, and they were given the same-- almost the exact same-- feedback. That is the thing.

PHILIP TAN: I think there is a huge social barrier between the undergraduates and the rest of the institute. The noble laureates, and even the post-docs here, that's a lot of social pressure keeping students away from that kind of interaction. You're all right. All of these things do exist here at MIT, but some of this is reinforced by professors themselves-- professors who keep their doors closed when they in their office, things like that. And so I do think we have to do give them the license, and sometimes that's pointing them to specific people.

RICHARD EBERHARDT: Yeah, giving them resources like here's a list. One of these might pan out for you. The others might not. And that's OK, just do what you can, kind of prime them for it.

PHILIP TAN: It's hard to be a teenager in this space.

RICHARD EBERHARDT: Actually, since we're talking about the student preparedness in that research like what we saw between what we consider the supported teams versus the unsupported teams, which may or may not be fair-- could be the motivated teams versus the unmotivated teams. Snap, in particular, really--

PABLO SUAREZ: [INAUDIBLE] correlation.

RICHARD EBERHARDT: --really well-supported. They actually reached out to you all for help with the game. You actually even did testing for them that was actually coming from you and Jana, yeah, because you were running workshops.

PABLO SUAREZ: Right.

RICHARD EBERHARDT: How did those workshops run-- how did you interact with the team?

PABLO SUAREZ: So, first of all, I think it's important to distinguish not only what you are calling supportive versus unsupportive, but also Snap versus everything else. Because Snap was a game concept that preexisted the course that needed refinement and digital interface. So from that perspective, it's not surprising that they have something clearly defined that they could run faster and deeper.

And the group was a group that engaged and went deep and went fast. The interaction we had with them, also, very importantly, we have events that we could use the product. So it was much easier to give feedback saying when we do this, we expect people that would get bored, why don't we try this other thing.

RICHARD EBERHARDT: It's milestones, which we give them milestones. But there is no weight behind them. These ones have an actual, if it doesn't work by then, someone's going to have egg on their face, right?

PABLO SUAREZ: And from that perspective, we know this, but we can try to do more of giving students more proportion of work that we know-- that we, the clients know-- that we have milestones. And that is limited by how much of the real world gives us milestones to abide by. But we can try that.

I do not feel comfortable with the label supportive versus unsupportive because it sounds like some got time and support and some others didn't. I think it's more about the form of the relationship and how the time was used or the opportunity to use the time was used. Because I have no doubt that you supported the team when you knew support was needed. And as far as I know, the same was true on our part. I think it's more about having sensors about when support is needed that is not being delivered.

RICHARD EBERHARDT: I think when we're talking about them in this way, I'm using it based on how they describe their own relationship and their own experience. So that the teams who we are calling unsupported are the ones who, in their postmortems, called out and said, we didn't feel like we got client support. They got the same amount. It was a different way, different quality, different quantity.

SARA VERRILLI: The way to put it is the same amount of support was there for them. One other way to put it is they didn't know how to reach out and get it. They didn't use the resources that were there because, at some level, we weren't pushing them at them.

And a lot of MIT courses tend to encourage students to solve all their problems themselves. They don't want them reaching out. They want them-- figure this out. And so it may come as more of a second-- it may not come to them as first nature to go, hey, we need to go and reach out for more help and grab that--

RICHARD EBERHARDT: And so that's actually the same thing. So what we saw with the Snap team is we wanted-- the feedback they were getting from us was look at your front end, look at your UI. We wanted to see a lot of changes on the UI. Because they were working on these milestones, they were really worried about the back end so they were doing a lot of work in the back end.

They were getting two different messages, and they chose client message rather than instructor message. Totally OK and totally viable. We took that into account. But I think the main issues we had with them is when they got our message, it was, they heard it early on. They didn't react to it until it was too late-- had they done a little bit extra in the beginning.

So one thing I think we need to think about with the class is thinking about how are we helping them understand how to prioritize feedback? Right? What's more important? Could be because here's something that's actually going to get used. It could also be here's a lesson we want to make sure they get. I think they got the lesson. They just got it in a harder way.

PHILIP TAN: Sometimes they get a lesson at a point of time that they can't actually execute on it. But hopefully they can do it for their next project after class. Something I want to suggest, actually, because we've been talking about how it's been difficult for us to figure out what's going on inside the teams. We can see the games and the testing, but the team dynamics, we don't find out about that until near the end of the class.

And something that we used to do outside of this class, but in our research work, was to get all off the scrum masters together on a regular basis to just do an exchange of information with each other. They don't even have to be necessarily reporting to us-- although we can moderate those sessions. It's more about them sharing their experiences with each other, and then it may make it more obvious that there are resources out there that some teams are taking advantage of and some teams aren't. Of course, we can use those situations for teaching, but also for teaching each other.

SARA VERRILLI: Creating some group, group support.

RICHARD EBERHARDT: We talked about that before this semester started, but we couldn't figure out a way to do that fairly. I already feel producers bear the brunt of the work in the class. They're the ones we see all the time. We don't see the students working on programming or design as much. Design a little bit more, but programming, we just almost never see them.

SARA VERRILLI: And we let the teams use the producers as their face-- so giving all the presentations, giving all the reports, answering questions when we ask questions in class. Think about how we convince the teams to share out that work a little bit more evenly.

PHILIP TAN: Well, perhaps, maybe, we can reduce the extent that we require these producers to do presentations in class, and replace that with these more closed-off sessions where they don't feel like they have to be on stage and put on the best face for their team. They can actually just talk about this is the problem I'm having with my team.

RICHARD EBERHARDT: So one thing I also wanted to make sure that we talked about what was it like for you? How much of your time did we demand? Was it a lot compared to other projects? Was it a little bit? Was it about moderate?

PABLO SUAREZ: I want to start by saying one of the things that I really appreciated was the time we spent-- both face-to-face and emailing and so on-- before the beginning of the course, to have understanding of what you and your students can offer, what we need and can offer, and to manage expectations, as you said. I think that was very well done. And it's not always the case. So with the students and professors and Red Cross, in general, we [INAUDIBLE], so thank you. That was really appreciated.

I think that amount of time that I personally had to give was about as much as I anticipated, about as much as I think is reasonable to expect for the future. We do have Jana, in particular, and a few other members of the team-- of the Red Cross team-- who were available to give a hand. And I think we needed to improve because I did make it very clear that I couldn't get five emails per day asking questions. But to have some kind of ability for the right contact when either engagement levels are too low or mega confusion levels are too high or sense of purpose is diluted and so on. So to refine that.

I think it would be better to think of times when there can be slots of decided interaction between each team, and someone from the client organization that doesn't have to be during class. Because I would always interfere with your plan. You gave us one opportunity that I thought was very useful, but a little bit late in the development process and also very crammed. By the time we said, OK, these are the kind of things you can do, there was no time for them to think and reflect on, OK, we can do A or B or C. Show that to us, and then us providing feedback.

So even if it were one more chance via Skype but scheduled, planned, and if-- it is in the nature of our work to have to travel, and it is in the nature of our work that travel timing changes outside of our control. So where there's a default date, and if things have to be changed, it is changed. So to know that there's going to be an instance of feedback that may be too little, too late, still but to anticipate that.

RICHARD EBERHARDT: Yeah, part of that is the students, we don't always know the student's schedules. So it could possibly be planned in the beginning of the semester. But planning it before the semester would be really difficult. Unless we just say it-- this is the time that you're going to have. If you don't take advantage of it, you don't get it.

PABLO SUAREZ: Right. It has to be we propose it's this week. This is the default time slot. If not, that time slot, agree on a different time slot. And then it's up to the students.

RICHARD EBERHARDT: So you're actually saying you would have talked to them a little bit more.

PABLO SUAREZ: If the problem was lack of engagement or perception of lack of support.

RICHARD EBERHARDT: So it's really that planning and scheduling. So as long as that's in place, then the time is there, because it's been planned. OK.

PABLO SUAREZ: And also I think it would have been nice for them to have access to, say, someone from the Ghana Team to look at the color, someone from the Heatwave Team in Buenos Aires-- to at least to get a flavor, there's another human being there that eventually can use their game. And I didn't do enough to help them envision the user-- either the Red Cross user or the community level or policy or government user.

PHILIP TAN: I think we need to state that strongly that it's not just the information that you're providing. It's the fact that there is a real person who can really benefit from the work you're doing is much more powerful than some sort of abstracted audience.

RICHARD EBERHARDT: So we provided plenty of documentation, but that's not the same as a human being.

PHILIP TAN: That's not the emotional drive that you're going to get-- oh, someone is counting on me doing good work.

RICHARD EBERHARDT: So maybe it might just be something for us to talk to the Terrascope folks to see how they do that.

PHILIP TAN: Well, they specifically get people who can commit on regular meetings. And the idea is-- my understanding is that it's for the domain knowledge primarily. But you get that side benefit of here is a real human being who's excited about this problem, or maybe really concerned about this problem, and that can influence the motivation of a team.

PABLO SUAREZ: And also, I don't know to what extent we can keep cramming what students have to do-- I know they already have to do way too much-- but to look at how that kind of field is currently being communicated in the game universe and outside of that. So PowerPoints or documents, they're, across the board, generally boring and unsuccessful-- even if they have the right content. So for them to get a sense that if someone really needs to create awareness on that topic, and this is the best they have in what exists in the real world, how about trying to give something that makes people want to do more through their game?

PHILIP TAN: I think saying, this is how low the bar is, can we get it slightly better, is less motivating than saying, what's the best you can do?

RICHARD EBERHARDT: But it's also choosing something that's going to get used. Why go in a direction that's-- if this is what those people are thinking about, if you give them something completely not like that at all, they're probably not going to know what to do with it.

PHILIP TAN: But I mean, I can imagine things like evaluating current methods of teaching about a certain concept, to be part of that research process that we've been talking about-- it's understanding the problem.

RICHARD EBERHARDT: Cool. All right, so I think it's about time to let you go. But thanks so much for coming in.

PHILIP TAN: We thank you. Huge gratitude.

Free Downloads

Video

• iTunes U (MP4 - 48MB)
• Internet Archive (MP4 - 48MB)

Subtitle

• English - US (SRT)