Valve's "Secret Weapon"
How the Studio Behind Portal and Half-Life Uses Playtesting
About a year into the development of Portal, playtesters kept giving Valve very similar feedback. Something along the lines of "that was a great tutorial, I can't wait to play the actual game".
Uh. Slight problem. That was the actual game.
Despite playing through 14 or so hand-crafted puzzles, players were clearly missing some vital element that would tell them this was, indeed, a video game. And so after a lot of discussion, Valve decided that the game needed an antagonist: someone to push back against the player, to provide motivation to keep moving, and to put these puzzles into context: this was all training in order to defeat a boss.
The end result was, of course, GLaDOS: a loopy AI overlord with a biting, passive-aggressive wit. She teases you, taunts you, tricks you. And the whole game leads up to a climactic one-on-one showdown in her central chamber.
"Well, you found me. Congratulations. Was it worth it? Because despite your violent behaviour, the only thing you've managed to break so far is my heart."
GLaDOS
Thanks to sharp writing and killer voice acting, she has become one of the most iconic video game villains of all time. But, says Valve's Robin Walker…
"Her genesis begins with a straightforward process of us trying to solve the core gameplay problem in Portal".
Robin Walker, Valve
Now if you ask me, the fact that GLaDOS may never have existed if it wasn't for this feedback goes to show the value of playtesting.
Playtesting Portal
Not to be confused with quality assurance - that's for rooting out bugs. Or focus testing - that's for market research. No: playtesting is simply watching people play through a chunk of the game - sometimes with a questionnaire or interview at the end - and then using what you see and hear to drive changes in the game's design.
For example, if Valve sees that players are routinely getting themselves killed when trying to redirect deadly, bouncing energy balls - perhaps they should make a change - like having it so you can only place portals on walls that are way above the player's height.
And so on Portal, this approach was used to touch up almost every aspect of the game.
Valve used playtesting to make sure the learning curve was perfect, they used it to remove moments of frustration, to make sure players noticed key objects, to improve the pacing, to tweak the difficulty, and to ensure the storyline was coherent.
Playtesting even led to the game's iconic visual style, with those sterile white walls and floor.
Originally, the game featured cluttered and grungy environments, but playtesters found it hard to identify the key puzzle elements. In one test, a player spent half an hour trying to push a shelf onto a button - while completely ignoring a nearby box.
Kim Swift - one of the key developers on the project - says…
"Probably the most important thing that we learned since coming to Valve is playtesting".
Kim Swift, Valve
You see, if you didn't know, Portal started life as Narbacular Drop: a student project made at Digipen University in Washington. They were invited to show the game off at Valve's headquarters and about halfway through the demo, CEO Gabe Newell stopped the presentation to offer the entire team a job.
Their goal was to remake the game in the source engine, and within the Half-Life universe. And to help, Valve introduced the small student team to its game development process.
Here's how it works. You start with a goal: perhaps, in this case, to make a puzzle that is clearly readable, and satisfying to solve. You then take a stab at reaching that goal, by designing something - in this case, a test chamber.
Next, you evaluate whether your design reached that goal by doing a playtest. And if it doesn't meet the grade, change the design and repeat.
Valve keeps going with this iterative process until it is "no longer excruciatingly painful to watch the playtests", says developer David Speyrer.
"Fantastic! You remained resolute and resourceful in an atmosphere of extreme pessimism."
GLaDOS
Perhaps, given the subjects of its most famous games, its apt that Valve sees game development as an engineering problem or a scientific study. Valve's former in-house psychologist, Mike Ambinder, says…
"We see our game designs as hypotheses and our playtests as experiments to validate these hypotheses".
Mike Ambinder, Valve
Now, it should be said that playtesting is by no means exclusive to Valve. And indeed, every game developer sources player feedback at some point in the process. But the difference is that Valve is obsessed with it, to an almost religious degree.
Ambinder calls playtesting "the most important part of the game development process". Newell calls it "our secret weapon". And a designer who joined the studio in 2018 said “I always heard that Valve does a lot of playtesting before I came to work here, and I was not prepared for the amount of playtesting this company does".
Valve does playtesting early. Swift and her team started running playtests of Portal after just one week of starting to build the game at Valve, even though they only had one half-finished room to show people.
And then do it often. Portal was playtested practically every week. They'd do a playtest on Friday, discuss the results on Monday, apply the lessons the rest of the week, and test again on Friday.
It sounds extreme. But I think this devotion to playtesting makes a lot more sense when you learn about the disastrous development of Valve's very first game, Half-Life.
Valve’s History
About two months before Half-Life was supposed to ship, Valve realised that the game... well, sucked. The developer said…
"You couldn't play the game all the way through, none of the levels tied together well, and there were serious technical problems with most of the game. As a whole the game just wasn't working."
Ken Birdwell, Valve
The only real option on the table was to scrap the whole thing and start from scratch. But this time, they would do things differently - which lead to two game development philosophies that are still present in the studio today.
One is the idea of the cabal: where the staff is split up into tiny, multi-disciplined teams who each take complete ownership of a small chunk of the game.
But the other is frequent playtesting, from the very earliest point of development. This time, if the game sucked, they wanted to know right away.
So, about three months into development of the restarted Half-Life, they started playtesting. They found random gamers in video game shops, and contacted people who filled in those little registration cards you got in old PC boxes. They'd sit them down in front of the game and just watch them play.
Each test would result in dozens of things that needed to be fixed, changed, added, or deleted from the game. Like, after watching some playtesters break every crate in a level, Valve realised it probably needed to stuff some of those boxes with goodies like ammo and health packs.
And this new process obviously worked: this version of Half-Life was a huge success, and is now seen as one of the most important and influential games ever made. And, as such, Valve would use playtesting extensively in all of its future games.
On Half-Life 2, Valve planned to give out the gravity gun towards the end of the game, but players loved using it so much, the team decided to make it available much sooner. On Left 4 Dead, playtesters found it hard to find teammates who were in trouble, which led to the x-ray outlines of your fellow survivors. And on Portal 2, a paint type that let you walk on walls was scrapped, when it made multiple playtesters feel queasy.
The arrival of Steam allowed Valve to go further - converting millions of online players into post-launch playtesters. When Steam's data showed that lots of players were getting stuck in Episode One, they released a patch to reduce the difficulty of a tricky siege battle.
Plus, playtesting was instrumental when exploring an all-new technology, virtual reality, for Half-Life: Alyx.
For instance, Valve learned that a player's tolerance for standing around watching people talk was significantly lower in virtual reality, so had to speed up the game's pace.
"Player behaviour helped us navigate VR development. You could almost think of the player as another designer. There is barely a moment in the game that wasn't improved by what playtesters told us and showed us through their play behaviour".
Christine Phelan, Valve
Playtesting Tips
So, from Half-Life 1 to Half-Life: Alyx, playtesting has been an invaluable tool for Valve. So I want to share some tips from the developers, to show you how to playtest more effectively.
Though it should be stated: Valve is not like other companies. From its unorthodox organisational structure to its near infinite resources, there are very few companies on Earth who can copy what Valve does.
And also: this approach may work best on super linear, extremely hand-crafted story games - and other techniques, like heat maps, may work better for other types of game.
But, with those caveats out of the way, perhaps their experience can still provide some useful guidance.
Tip 1: Test early
"Playtesting is where we make the vast majority of our most important changes to our game. So we try to do it as early as possible on the project, to get the most value out of it."
David Speyrer, Valve
When you identify a problem early on, you have time to go back to the drawing board and develop an effective solution to the problem. But do it too late, and sometimes the only way to fix a problem is with a flimsy band-aid solution. Like, uh, I dunno, having the characters just tell you the solution to a crummy puzzle.
Valve may start testing within a few days of prototyping a mechanic or designing a level. Even if it looks hideously ugly, with programmer art or bright orange textures on all the walls. But this actually helps them avoid wasting time on other aspects: there's no point investing heavily in art or audio if the game mechanic is gonna be completely changed - or cut from the game entirely.
Tip 2: Test often
Valve typically tests every single week, to make sure that they are constantly iterating on player feedback.
This also leads to a huge amount of data. Each Half-Life 2 chapter had about 100 playtesters, for example. And that mass of information can be invaluable.
By having a huge number of people look at the game, Valve can look for common trends in the feedback, and avoid tweaking the game based on a few weird outliers.
And the more experience you have seeing people interact with your designs, the better you'll get at preempting this feedback in the future.
"After you've watched a couple hundred playtests, you start to develop a much better sense of what are successful and unsuccessful design strategies".
Gabe Newell, Valve
So, some famous lessons learnt at Valve include "players don't learn when stressed" and "players don't look up".
Tip 3: Shut up and watch
A playtest should try to simulate the actual experience of playing the game. Which means the observers need to stay quiet for the duration of the test: no hints, no guidance, no answers to burning questions.
"Nothing is quite so humbling as being forced to watch in silence as some poor playtester stumbles around your level for 20 minutes, unable to figure out the 'obvious' answer that you now realise is completely arbitrary and impossible to figure out".
Ken Birdwell, Valve
Post-game interviews and questionnaires can certainly happen - those were instrumental in solving the GLaDOS problem after all - but you'll often learn more by watching players than by talking to them.
Swift says "they may tell you later that they like the game but you'll really tell by their body language whether or not they actually enjoyed themselves".
It's also a common game design trope that playtesters like to offer potential solutions to the problems they face - but as they don't have insider knowledge of your vision or constraints, those solutions are usually better left ignored.
Tip 4: Designers should run playtests
Valve does not outsource testing to a special department or leave it with a publisher: playtesting is done by the people who are actually responsible for the design and execution of the level or mechanic in question.
This shows the developers exactly what needs fixing, and can give much-needed motivation to improve the game. And player behaviour might inspire new puzzle solutions or ideas for the designers watching.
Like, in Half-Life Alyx, players would instinctively cover their real-life mouth to stop Alyx from coughing, and alerting a gigantic blind zombie named Jeff.
So Valve turned covering your mouth into an actual game mechanic.
Tip 5: Get the right people
Valve gets feedback from a huge variety of different playtesters, from fellow staff members to little kids to pro gamers. But its learned to always have a target audience in mind when making changes.
Like, when Valve was figuring out how to make a climactic boss fight against GLaDOS, they initially got feedback from hardcore FPS players, who said the level needed more action, more challenge, and more skill.
But that idea was a dud: when facing this tricky boss fight, says Erik Wolpaw…
"The vast majority of playtesters who had gotten used to the slower-paced, cerebral nature of Portal were just frustrated, confused, and dissatisfied".
Erik Wolpaw, Valve
They still wanted to satisfy those hardcore FPS players, but did so through optional content, like Portal's advanced chambers and challenge maps.
Tip 6: Challenge your assumptions
Okay, so maybe the GLaDOS boss fight didn't need to be a test of skill. So, perhaps it needed to be the most complex puzzle in the entire game? Surely that would be a suitable climax.
But, here's the thing - playtesters thought the game's mid-point escape, that bit where you use a portal to avoid falling into a pit of fire - players thought that was incredibly climactic and satisfying. Even though it's basically the easiest puzzle in Portal.
But Valve realised the time pressure, the visual impact, and the high drama all made it way more exciting to players. So maybe they were overthinking the final boss, too. "We've been holding on to this idea that we need a complex puzzle at the end and it simply wasn't true", says Swift, and so they settled on a pretty simple sequence.
It's very easy for game designers to make wonky assumptions about how players will act - I recently made a whole video about doing this in my own puzzle game.
Perhaps the most important tip, though, is that playtesting feedback is just data, and it's up to the designer how that data is interpreted, filtered, and applied.
For example, Half-Life 2 originally had a very short introduction before Gordon Freeman grabbed a gun and started shooting. Playtesters loved it. Who wouldn't be excited when jumping into the action? And so, given the feedback, it would be easy to keep the game exactly like that.
But, despite the positive sentiment, writer Marc Laidlaw says Valve decided to keep working. They thought it would be better to delay combat.
"We wanted you to witness the cops doing something horrible, and feel like what you were doing was a response to that - not that you were just a killing machine".
Marc Laidlaw, Valve
And they thought it would lead to a better emotional pay-off if you had to wait a much longer time before finally getting Gordon's iconic crowbar.
Conclusion
Here's the thing: playtesting is as useful as you make it.
If you just bend the game to the whims and desires of every playtester who comes through, you'll end up with a dumbed-down game, or bland, design-by-committee sludge.
But if you go in with a clear goal. A specific game you want to make, for a specific audience you want to entertain. And then use playtesting to validate whether you are hitting that goal, you can use feedback to unlock the incredible game you are trying to make.
New Game +
I've got one last Portal story, before you go.
So, the invention of GlaDOS is actually not the most dramatic change that Valve made, in response to player feedback.
During an internal game jam, held shortly after the release of Portal 1, a bunch of Valve developers came up with an experimental puzzle game called F-Stop. In this game, you use a camera to take photos of objects. Then, you can spawn that object into the world - perhaps at a completely different scale.
Gabe Newell loved the concept, and said it should be developed as a follow-up to Portal - meaning each game in the series would feature a different piece of technology developed at Aperture Science. But, after nearly a year of development, playtesters were very vocal in their feedback: Portal without portals just didn't work. And so Valve decided to scrap the game and start Portal 2 all over again.
In recent years, we've been able to see what F-Stop might have looked like, as fans have used leaked assets and code to make various recreations of the game. But Valve never returned to the concept. If you want to play something similar, check out Superliminal, or wait for the upcoming photography puzzler, Viewfinder.