Accreted Drivel

Steven Tattersall's personal blog

Double Doors

Created on: 2013-12-05

Posted in programming debugging development war-stories

Here's another war story. Every developer who's done 15 years in the industry will have similar.

A very late night; we'd been crunching for several months, waistlines were expanding, tempers were fraying, the engine was a bug factory, tools were overly ambitious, and the game itself had died by a thousand cuts.

(I'll write more about this project one day. There's always one project that makes or breaks you, and I think this one did both.)

In the office it was sweatier than Merv Hughes's budgie-smugglers. The air was thick with the excess calories from takeaway and the aircon shut down at six.

Every day we would cut a disk to ship up to Liverpool for QA. Every day the courier van would arrive at 6am to collect it and hurtle up the A1 like a loon, and we'd take a handover photo, in case this was The Final Night. It had gone on so long that we had 2 shifts; Julian the lead would stay through one night, and I'd do the next. I still have one of the photos. Our faces are puffy and swelled, balloons pricked only by patchy facial hair.

So maybe this was the night. Bugs were down to single figures and we've stomped those during the day. The testers were on their playthroughs of the latest build before I started off a file copy, went over to the cutter machine and cut a disk. (This was PS2. I think we had just started to investigate continuous builds.)

I'm in the middle of reading some coding articles and monitoring the baseball scores as the builds chunter away.

"Steve? You'll want to look at this."

I looked. There is a door that shouldn't be open. The game's broken again. It's an A-class bug. You can just run through part of the game without doing anything.

Must Be Fixed Tonight. Somethere there is a comfortable bed, my bed, floating away into the distance.

The courier clock starts ticking. To be ready for the Man in the Van, the build must start in the next few minutes.

Why is a simple door not doing the right thing?

First, door logic never ends up being simple. It's usually given to junior programmers who make a hash of it ("how hard can it be?") Since the logic had to be written in an incredibly inflexible custom FSM token language (at the time, no "and" operator for transitions), unless the door is doing something very simple, your state machine starts spiraling into many twisty code paths, all similar. Lesson One: *riends Don't Let Junior Friends Write Door Code.

Second, our engine is a bug factory. We've massively overestimated the power in a PS2 and now to keep it running at a decent rate, game entities are summarily killed as soon as they are just out of view, with hilarious consequences. Imagine the entity's existence in the collision system vanishing, so you can shoot through doors that are just out of sight. That kind of thing. The engine doesn't handle shutting down some bits and not others, resulting in a constant arms race between the designers, who are turning things off faster than Nosferatu in a tanning salon, and the programmers with their brains dribbling out of their ears, trying to find ever more precarious ways of patching the problem.

Third, no-one ever wrote their door logic to be re-entrant. So if you deactivate an entity, and reactivate it, lots of logic goes wrong and door hang open. Or sometimes pop into existence open, then close, with a mysterious creaking noise floating in from the distance. (I should have recommended that testers play without music. We could have identified these earlier. "Log all mysterious door creaks!" One for the future.)

So we know about doors. Doors bad. Suspect number one: bad door code.

It's time to pull up the state machine for the door. Lots of the doors have custom state machines. Don't ask me why. It seemed like a good idea to someone. Usually because the designers wanted something very very slightly different from another door, say rotating a different way, playing a different sound effect or having different rules to open, pausing slightly.

This state machine is a horror.

Tick tock tick tock.

This door acts like a proximity-operated door for the first section of the game, then after some puzzles are solved, the door shuts behind you. Then the player does some more stuff, watches about 50 cutscenes, defeats a disappointing boss, and after this event the door opens again. It's like it has to do one door's job for a while, then a different door's job afterwards.

This is a state machine that requires a bit of thought but should be doable, even in the not-even-a-scripting-language state machine syntax we have to use. But this code is beyond spaghetti, it's more like pasta that's been stuck on your pan for days. It's been patched, scraped, nurdled into a congealed form like nothing ever seen. It's really 2 separate state machines, one for each event, copied twice and with random links in between, all assuming that it will never be interrupted.

Tony and I stare at the code. We try to draw a graph of states, then throw it away. It might be fixable, but it's more probable that we encounter the heat death of the universe before we check all the special cases.

And I don't have time for a rewrite. It takes about 25 minutes to test each time, the state machine syntax is error prone and I need to cut the disk RIGHT NOW OMIGOD THINK.

Tick tock tick tock.

... has to do one door's job for a while, then a different door's job afterwards...

Just a moment, is it a sliding door? It is. Brilliant, fire up the map tool!


If you play the shipped game and nurdle your character into an alcove, you can just about tell that there are 2 doors. They both use the same model, and are mapped in the exact same spot.

One door uses some standard proximity-operated state machine code, and the other uses some other standard "open on event" code. The event controlling Door 2 is set until the first puzzles are solved, forcing it open all the time. Then when the puzzles are done, the event is cleared and the door shuts until the second section is completed. Meanwhile, Door 1 does its proximity operation until the puzzle solving, then it's forced open by setting a different event.

The doors had to be sliding ones that disappeared into the walls, otherwise you would see 2 doors if one swung open but the other didn't...

Tick tock bing! It works and the disk goes off for subs. (It fails of course, for some other reason).

That game was released 10 years ago this month.

Back to Index