« More about Nine Worlds | Main | All Kung Fu games... »
April 11, 2006
Evaluating Play formatively
In Game Design as Process I presented a process flow that had a lot of dotted lines from play abck to the earlier design steps. I strongly believe there are a variety of ways that play chanegs the campaign as you go. I have a lot of fuzzy ideas in my mind about theory, but I think I'll start talking about one of the tools I've found useful.
Formative evaluation is the process of gathering feedback to make a process more effective. The emphasis here is on the collection and analysis of data and the revision of the game documents as a result of actual play.
While formative evaluation will help refine the goals and objectives of the game as they change through play its really crucial that this process be broader to capture play strategy, mechanics and social contract issues as well.
I find that the formative evaluation works best when anchored in the design documents. What I’m normally trying to answer following questions:
- Are the goals of the campaign (the design document) working?
- Is the Social Contract adequately capturing what is happening in and around the play space?
- Are the mechanics adequately representing the goals? Do they include the appropriate sub-skills?
- Are the campaign goals clear and readily understood by the group? Does everyone understand what we’re trying to do?
- Is the campaign being managed efficiently in the manner its being meditated? Is everyone’s needs being met?
My biggest question always boils down to how often to do this. I feel like it tends to be only when one of the above questions starts being problematic and I’d like to be more proactive.
Be prepared to obtain information that indicates that your campaign is not as effective as you thought it was. But through this continuous feedback cycle I find that the game can be made healthier and more viable in the long-run.
Comments
So how do we get this information from the players? I feel that these questions, in themselves, are a bit too high level or too broad to simply hit players with. I do think they are a good way to come up with the direct, specific questions to ask players.
One way I could see this working is: the GM has separate sheet with 5 sections, on for each part of the game (basically for each of these questions). Whenever some event in the game suddenly feels "wrong" or "odd", the jot down what was happening in what seems like the right category, and keep going. Later, ask a question about that specific event, but skew the question towards that category, trying to get to the core of the game (while tying it to specifics).
Questions I could see myself asking players:
1. Your character chose to not to puruse that criminal. I was surprised, because I was expecting more of how you choose to deal with criminals, but you went back home instead. How do you feel about that goal, pursuing justice?
2. You seemed a little confused or put-off when the other player narrated the ghost into the scene. Was that a hiccup? What do you think?
3. I saw that you really didn't like when you whiffed that Hacking roll. Was that okay, just a frustrating failure, or was there something about it that wasn't set up right?
...I'm sort of blanking for specifics for 4 and 5 at the moment. But in any case, I feel like it's easier to address specific incidents in play (tied to specific larger parts of the game) rather than trying to look at the big big picture.
Posted by: Dev Purkayastha at Apr 13, 2006 11:57:46 AM
Hi Jere,
How many things do you change in a game at a time?
I have a fairly firm rule to change only one thing at a time, because if you change multiple things, I can't figure out which change to attribute any improvement/worsening to. This is based on my experience doing homebrews, but I hadn't really thought about hacking the non-mechanical stuff until very recently. (This is because of the way Nine Worlds is playing out -- I think we need to tweak the informal style of play a bit to make the game really work.)
Posted by: Neel Krishnaswami at Apr 13, 2006 2:52:36 PM
Neel,
I find that its best to address changes to fix one cause (or set of related causes) at a time. For tantaene we had a bundle of interrelated issues that we addressed all at once. It would have been possible to address one of these alone but given the situation it was desirable to look at them as a package.
I think it is important to look at the non-mechanical aspects of a game in the same light as the mechanical aspets, since frankly I feel the mechanical is such a small slice of the pie (that we often seem to give higher value to).
Posted by: Jere at Apr 14, 2006 9:26:27 AM
Dev-
For my most recent time of doing this with tantaene I developed a matrix with the areas of goals, social contract, play strategy, mechanics down the left and on the top the areas we were concerned about (Initiation, conflict framing, Drama, Exoteric and Esoteric). I then had a few sample questions in each to start conversation going. Once I got conversation going I actually just ahd to take lots of notes and try not to be too apologetic.
I like your guided questions on specific events too. I might call that the summative evaluation of the game, but thats the 6 Sigma geek in me talking.
Posted by: Jere at Apr 14, 2006 9:36:38 AM
My basic checklist when I do this (it never goes according to plan)
- Arrange for the whole group to participate. It doesn’t help if someone is missing.
- Adequate time should be arranged for
- The group needs to be motivated to participate
- After every play sessions it’s a good idea to make notes about how play went and its relationship to the play documents. It helps if everyone does that.
- Personally play a minimal role in the discussion (this is so hard and not always called for)
- Come up with some starting questions relating to the various game documents.
- Summarize the data collected
Posted by: Jere at Apr 14, 2006 9:42:30 AM
What if you had Players write down two questions after every session:
1. Most confusing/unexpected part of game today.
2. Most cool/exciting part of game today.
Seems like you could even decide to have a meta session where folks could comment on comments.
Posted by: Arref at Apr 14, 2006 8:26:58 PM
In my experience, most players aren't eager to spend potential gaming time on feedback, though occasional state-of-the-game commentary-type sessions are a good thing.
Posted by: Jeffwik at Apr 15, 2006 4:11:57 PM
