Level Design And Build System Thoughts

I've been discussing with Bernd a lot lately how we could improve our build system and level design process over the next months. Turnaround times for complete and incremental builds, and also the "local turnaround" of level designers are starting to become critical for a "big" project like Drakensang. A complete (nightly) build is now at around 11 hours (this includes recompiling and rebuilding everything, building an installer and uploading the result to an FTP site). An incremental build (during the day) takes at least half an hour and up to 2 hours (without building the installer and uploading). To put this into perspective, Drakensang has about 7000 textures and about 4500 to 5000 3D models (I don't have the exact numbers at hand because I'm not at the Labs right now), the runtime data for the whole game is currently about 4 GB in size.

For level designers, there are 2 separate turnaround-time problems: updating the work machine with the new data from the last nightly build (this may take somewhere between half an hour and and hour or so), and the time it takes to test a change in the actual game (we're working with Maya as level design tool right now, not an ingame editor).

We have a few holy dogmas at Radon Labs:

• Daily Build: everybody must work on the most current data which is at most 1 day old
• "Make Game": creating a complete build must be fully automated, and happen on a central build machine
  
• The Toyota Rip Cord (don't know if this is translated correctly, it's the "Toyota Reißleine" in German): if there is no working build, production essentially stops until the problem is identified and resolved (and the responsible person has been ritually tared and feathered).
  
• One Tool For One Job: don't use several different tools for the same job (all 3D modeling is done in Maya for instance)

We also have a some other secret dogmas in our canon, but they don't affect the build system or level design work flow so I won't utter them here :)

We could easily chicken out by giving up daily builds for instance. But this would most likely create "Ivory Tower" pockets inside the company. Tendencies like this happen all the time, they are dangerous for the project, and must be fought the instant they show up.

Instead we stepped back and thought about how a perfect build-system and a perfect level-design system would look like. The whole problem is really 3 separate (but somewhat related) problems:

1. reduce build time
2. distribution of build data to workplaces
3. reduce turnaround times for level-designers

Point (1) is relatively easy. I think the only worthwhile improvement can be gained by distributing the workload across several build slave machines. We already invested serious optimization work into our Maya exporters, so there's not much more to gain there. Setting up a distributed build system is an interesting job, but not too complicated if you have control over all build tools.

Point (2) is more interesting. The question here is "do we really need to distribute all the build data to all workplaces?". That's 4 GB of uncompressed data per day per workplace, but a level-designer typically only needs a fraction of that data during a normal work day, which typically looks like this:

1. level designer comes in at the morning, and pulls the most current build data from the nightly build
2. level designer cvs-edits the files he needs to work on
3. level designer works inside Maya and several specialized tools, like dialog and quest editors
   
4. level designer needs to check his changes in the game frequently (involves starting the game)
   
5. in the evening, level designer cvs-commits his work and goes home
6. the build machine creates the new nightly build for the next day

There are several problems here:

• in the morning, a lot of time is wasted to just update the runtime data of the workplace machines
  
• the local turnaround time to check changes in the game is too long (somewhere between 1 and 3 minutes)
• when the level designer checks in his work in the evening, subtle collisions may occur with the work of other level designers (this is especially critical in "persistent-world-games" like Drakensang)

Above a specific project size and complexity, level design becomes more and more frustrating because more and more time is spent waiting for results.

Now here's the actual point of the whole post: What if level-design would actually be fun? We could improve the fun-factor a lot if the level designers would immediately see results, and could directly work together with others. What if level-design would be like mixture between a Wiki and a multiplayer game?

Here's how we think our level design should work in the future:

1. level designer comes in the morning, and starts the game in level-design mode
2. the game notifies the level designer that an update is available, updating only involves pulling a new executable
   
3. the game connects to a central game server, which holds the actual game data in a database, and the graphical/audio content through a network share
4. the level designer creates, places and destroys game objects directly in the game, all changes are distributed via the game server to other level designers working "nearby"
5. to test the changes, the level designer presses a play button, and after a few (very few!) seconds, the editor will change into game-mode (it is very important to strictly separate the edit-mode from the game-mode, because application programmers should never have to care about level editor stuff)
   
6. the ingame level editor is augmented with specialized tool windows written in C#, some of them generic (i.e. a nifty table view), some of them project-specific (i.e. an inventory editor)
7. in the evening, the level designer shuts down the machine and goes home

So we would give up Maya as level design tool in favor of a "collaborative ingame level editor". The collaborative/multiplayer part sounds like a gimmick, but it's actually very important because it solves the data collision problem. Since all changes are immediately distributed to all level-designers, there's no danger that several conflicting data sets are created (the longer 2 separate data branches are evolving, the more likely collisions will occur which will be difficult to resolve).

Up until a few days ago I would have scrapped the whole idea and declared it as impossible. Implementing an ingame editor which would suit all the different genres we are doing sounded like opening a can of worms. But in the end it isn't that difficult (for a distributed system it's actually necessary to have an "ingame-editor"). We already have a lot of the basic building blocks in place:

• We can pillage a lot of ideas from our current "Remote Level Design" system. At the moment, we can run Maya and the game side by side, and changes in Maya immediately show up in the game, this is nice for tweaking lighting parameters for instance.
• Game data already lives completely in a lightweight local database (SQLite). This gives us a lot of advantages:
• a game entity is completely described by a simple set of named, typed attributes
• a game entity always corresponds to a single row in a database table
• all "other data" already lives in database tables (even hierarchical data, like dialog trees)
• all data manipulations can be expressed with a very small subset of SQL (INSERT, UPDATE and DELETE ROW)
• The only operations that must be supported by the generic ingame level editor must be "Navigate", "Create Entity", "Update Entity", "Destroy Entity", where creation is always a duplication either from a template or from another entity. More complex operations, or different views on the game data, will be implemented in C# tools which are connected through a standardized plugin interface.
  
• With Nebula3's TcpServer/TcpClient classes and orthogonal IO subsystem as base it should be relatively easy to setup the required in-game client/server system
• We are already using some specialized editor tools written in C# (we did some of them in MEL before, and C# is a HUGE improvement over MEL especially for GUI stuff)

The devil is always in the details of course. But I think this is a pretty good plan to fundamentally improve our level design process in future projects.

Game Development Essentials: Bugtracking (or how we ended up writing our own bug tracker)

The last post of this series dealt with version control, now its time for bug tracking. In any non-trivial project it's important to have a good plan, or design document, which defines the final product, and a means to keep track on how close the current state of the project is to the final product. Also, in a team, it its important to define the work items each team member should work on at a given time, and to keep track what work items have been finished, and how this affects the current overall progress versus the planned progress. Finally, during the development of any project, hundreds of unexpected little annoying bugs creep in. These bugs must be collected, analyzed, fixed and squashed.

If this sounds terribly complicated to you, that's because it is. During a game production, many plans and deadlines are made on various levels of abstraction. A simplifed list from most abstract to most detailed may look like this:

• Game Proposal: This is a very abstract document often used while pitching a new game to publishers. A game proposal is somewhere between 4 and 10 pages and gives a quick overview of the game. The point of the proposal is to get a publisher interested and to convice him that the game will make him a shitload of money.
  
• Game Design Document: The game design document is usually created after a project has been signed during the so-called pre-production phase. The design document should be the cook-book for the project. The more details are fleshed out here, the better. The Game Design Document can be several hundred pages thick. Some features however cannot be planned in detail beforehand, especially in game projects, because it may be unclear whether a planned feature "feels right" once it is implemented. What sounds good in theory may totally stink in practice (in fact, very many features fall into this category). So its important to know what things to leave intentionally open in the design document, and its important to add "spare time" for experimentation (however, project managers generally don't like this because it messes up their Gantt charts).
  
• Milestone Plan: the milestone plan structures the Game Design Document into a list of features, spread over the main production time, and breaks down the feature list into milestones. Advance payments from the publisher are usually bound to successfully delivering those milestones on time. The milestone plan's features are usually down to a granularity level like "10 creature models finished", or "The player character can pick up objects.".
• Work Items: Here we are at the most detailed planning level. Features from the milestone plan are split further into single work items. A work item is a chunk of work assigned to a single person. For instance, we could take one of the 10 creatures from the milestone definition above, and split this into several work items, for instance:
1. Design.
2. Modelling.
3. Texturing.
4. Animation.
5. Acceptance.

"Now, wait a minute, wasn't this post about bug tracking, not planning?" - you might rightfully ask. Well the problem is, bug tracking and planning are really 2 sides of the same medal. And that's a terribly hard problem to solve.

There are tools for planning, which all look more or less like MS Project, and there are tools for bugtracking, which are all more or less comfortable database frontends. Now we spent literally years looking for the perfect bugtracker (we already learned to not expect an integrated planning and bugtracking tool). We looked at everything that's on the market, we literally spent several man-months just evaluating what's there! In the end we did the unthinkable: We wrote our own bugtracker. And it was exactly the right thing to do.

Now, how the hell did it come to this?

(Disclaimer: for the rest of the post, when I write "bug", I really mean "work item").

The short answer is: usability. Try to enter a new bug into a tool like Bugzilla or Mantis. It is a nightmare of web forms, filled with dozens of fields which must all be clicked and filled out. Its confusing, takes up to several minutes per bug, and generally isn't a lot of fun. Now, when we were a small company with one project at a time and a single project manager, we already had a system in place which was a joy to work with and where entering a bug only took somewhere between 10 seconds and half a minute. All bugs were laid out in a single table, every line one bug. Bugs could be filtered in an instant, statistics could be generated and several users could work in parallel on the bug database. It was almost perfect. The name of that magic bugtracking tool? An Excel sheet with a few macros!

There were several shortcomings that we soon became aware of:

• It doesn't scale well, once you hit several thousands bugs in a project and more then 10 people, shared Excel sheets started to slow down very quickly.
• No real database backend, which made it impossible to define more complex filters.
• Several projects had to be handled in parallel. Way too much data for a single sheet.
• Direct access of external QA teams was impossible, we had to move bugs back and forth manually between different systems.

So this is where an evaluation odyssey began. We looked at several readily available bugtracking tools and decided to switch to Mantis, while still continuing to look for better alternatives. When we switched to Mantis an interesting effect occured: projects which used Mantis as bugtracker tool had dramatically less bugs then projects which used our traditional Excel sheet! Obviously, the Mantis-projects didn't magically obtain a better code quality. People simply entered fewer bugs because entering a bug in Mantis was more complicated then before and took much more time. Unfortunately we weren't really aware of the importance of this fact until late into the project and we had to start slipping deadlines (something unheard of so far at Radon Labs). This really taught us a lesson how important a good bugtracking tool was for the existence of the company.

We knew exactly what we wanted from a bugtracking tool, we tried for 2 years finding a better solution then Excel (imagine that!), and finally we decided to write our own tool. We hired a programmer with C# and SQL experience and had a first version of a working bugtracker which did everything exactly the way we imagined, and fixed all shortcomings of our Excel sheet within one month!

The requirements for our bug tracker are as follows:

• entering a new bug and filtering existing bugs must be simple, intuitive and fast
• must support multiple projects
• must work over a DSL line
• must support user roles (access rights, users assigned to projects and departments)
• must support multiple projects
• should a true SQL database backend
• it must be possible to extract certain statistics
• must support parent-child and follow-up dependencies between bugs
• it must be possible to add attachments to a bug
  

We decided that it would be best to implement the front end in C#, since GUIs and working with databases is what C# does really well. It's also relatively easy to find programmers experienced in C# and SQL.

The default view of the bugtracker basically looks like an Excel sheet. Each line represents one bug. Every column represents a bug attribute:

• Synopsis: a short summary of the bug (or work item)
• Category: one of Bug, Plan, Story, Suggestion or Task
• Priority: from 1 (most important) to 4 (least important)
• Department: one of prog (Programming), gfx (Graphics), level (Level Design), QA (Quality Assurance) or PM (Project Management)
• State: the current state of the bug, one of: open, fixed, duplicate, in the works, nice idea, obsolete and open
• Creator: who entered the bug?
• Assigned To: who's the bug assigned to?
• Date Created: when has the bug been entered?

The are several other fields which are usually hidden, but can be configured to be visible.

Below the table view is a large text entry field for the bug description, this should contain a detailed description of the bug, and at least the steps to reproduce the bugs. The description field will also contain automatically generated log messages when the state of the bug has been changed. The attachment list contains all attached files, which can be inspected by double clicking on them. It's also possible to save the attachments to the client machine.

At the top of the table view is a row of drop down boxes, which allow quickly filtering the list of displayed bugs (for instance: show me all my open bugs of priority 1 only needs to mouse clicks). More complex filters with boolean operations can be created and saved for re-use as well very easily. There's also a pre-defined standard filter called "My Bugs", which display all open and work-in-progress bugs assigned to me.

Now, how does bug tracking work in practice?

• Alice from QA finds a crash bug in project A (hopefully just an assert() that got triggered). After some trial and error she finds out how to reproduce a bug. She goes into the bug tracker (which is usually open all the time) and creates a new bug in project A, which adds a new empty line to the bug table. She fills out the fields right in place, she knows that the should go into programming, but isn't sure who would be working on it. That's why she assigns the bug to the lead programmer of project A, which is Bob. Since it's a crash bug she will definitely set the priority to 1. Usually she also nags Bob directly about the bug if she thinks the bug should be fixed immediately.
• Bob checks his open bugs and looks at the bug description. From the bug description, looking at the source, and maybe checking previous version of the source code he's pretty sure that fixing the bug is programmer Carl's job.
• Now he re-assigned the bug to Carl and also tells him to have a look at it ASAP.
• Now Carl checks his open bugs and finds Alice' bug. Looking at the description he's pretty sure what's wrong and fixes the bug. Once he's sure the fix works by trying to reproduce the bug following Alice' repro steps he commits the changes into version control.
• He sets the bug to fixed and tells Bob and Alice that the fix should be in the next build.
• Now, in the "official" Life-Of-A-Bug, the fixed bug would be re-assigned to Alice automatically, and when the new build is available, Alice would have to accept the fix from Bob by checking that the bug is indeed fixed, and if that is indeed the case, set the bug to closed. At Radon Labs we omit this final step, and let the bug's life end at the fixed stage.

During production, thousands of bugs are entered into a project's bug database (remember, these are not just critical programming bugs, but all types of work items for the entire team). Often, non-critical bugs will remain unfixed for some time, duplicate bugs will be entered, or bugs are for some reason no longer reproducible. That's why it is necessary that the bug list is maintained and kept tight. Also, sometimes bug priorities must be decided "by committee", maybe the graphics department thinks some bug is highly critical, while the project manager thinks it isn't. That's why the project manager, the lead programmer and the headof's of the graphics and level-design departments gather every one or two weeks to do a bug triage. This is just a short meeting where the list of open bugs is sighted, and re-assigned, re-prioritized, or set to obsolete or duplicate. This is necessary house keeping for keeping the bug list clean, or to set the right priorities in order to hit the next milestone on time.

That's it for a basic overview of bug tracking! However, there's much more to planning and bug tracking as could be written in a single post. So maybe I'll come back to this topic at some later time.