First Nebula3 Snapshot For Download

I have put together a current snapshot of Nebula3: nebula3sdk.zip . Keep in mind that this is all work in progress, and that there's not much to play around with yet (basically only the Foundation Layer, no 3d rendering yet, etc...). But at least you should be able to try out the stuff I'm writing about here. After installation, look for a Nebula3 SDK entry in the start menu and start with the Documentation. Software requirements and compilation instructions can be found there.

Enjoy!

The Daily Fight Against Code Complexity

If you're a programmer, complex code is your worst enemy. Complex code is impossible to understand, impossible to maintain, and impossible to optimize. One could say that programming is a daily struggle against overboarding complexity which constantly creeps in from all corners in a software project.

Here are a few points how complexity happens and what to do against it, the list is by far not complete, and the solutions are not cookbook-recipes. Your mileage may vary dramatically.

• Interface first, code second: This is probably the most important point. When starting to work on a new feature or subsystem, it's tempting to get the most complicated implementation details out of the way first. Resist!!! Instead concentrate on the interface first because that's the only thing that matters to the other programmers which are "honored" to use your creation in the future. Step back, look at the bigger picture, and ask yourself: If I were one of them, what functionality would I expect from the subsystem and what would I want the class interfaces to look like? Repeat this after you designed each class interface: "If I would need to work with this class, would I be happy with its interface or not?". Maybe let your fellow programmers look at your classes. Only when you are completely happy with your class interfaces, start to think about the implementation. The good thing of all this upfront work is, once you have a good class interface, the implementation behind it is usually obvious, and "writes itself".
• Solve the problem at hand, not future problems: Often programmers want to "do it right once and for all" and create super-flexible subsystems which solve all types of similar problems which may show up in the future. Forget it, never works. Just accept that you cannot foresee the future. Simply write clean code which solves exactly the one problem you have at the moment. Only care about future problem when they actually show up. Then reuse existing code if it makes sense. Writing reusable code is something completely different then writing code which intends to solve imaginary future problems.
  
• Pocket fluff adds up over time: Some subsystems have a trend to grow complex over time. The reason is that programmers that use the subsystem need to add little tweaks, like adding a few new methods, or making a method virtual, or making a method public or protected. This is almost inevitable, and it would be wrong to completely forbid such tweaks. However, it is very important to not let this get out of control! When a subsystem requires many of such tweaks, then this is an indication that something is wrong with the subsystem's design, and a refactoring may be in order. Don't be afraid to cut back superfluous functionality, or to throw the entire subsystem away and do a clean rewrite with the new requirements in mind. The earlier such a refactoring happens, the better. Also, if you intend to do such tweaks to a subsystem, first check back with the guy who maintains or originally wrote the subsystem so he knows what's going on.
  
• Newbie programmers love complex stuff: Unexperienced programmers have a strong tendency towards overcomplicated solutions. Even worse are unexperienced prodigy programmers, because they are convinced they are good, when in reality they still need 10 years of real-world-experience to be really good. There's no simple solution to this, experience cannot be injected but must be learned over many years.
  
• You'll need 3 versions to get it right: That's because you can plan and design ahead as much as you want, you won't know what you actually wanted to achieve until the result is in daily use. Only then will all the little imperfections, design issues and interface bugs show up. When this happens, don't hesitate to throw away and rewrite as soon as realistically possible (often minor nuisances like a "milestone plan" stand in the way, but fix the problem as soon as possible or it will haunt you until eternity).
  
• Know what's out there (and use it): Unexperienced programmers tend to waste too much time reinventing wheels (and making them square instead of round). Sometimes it actually IS better to reinvent a more specialized wheel, but for all the boring stuff, don't be proud and just use what's already there. Make sure you have a good toolbox of utility classes that simplify your daily work, like various container classes, or a good string class, and know how to use them! If you find yourself writing more lines code with mundane, repetitive stuff then actual feature-related code, then you need a better toolbox of low level utility classes!
  
• Don't rely on UML (too much): UML is good and fine for doing the very first design steps and getting a grasp of a problem. But don't use UML to design your subsystem down to the method and member level. You will most likely end up with terribly complex class interfaces on the source code level. What looks simple as an UML diagram is often still too complex as source code. Remember that your fellow programmers don't use your subsystem by drawing UML diagrams but by hacking lines of code into the machine! In fact, if your subsystem can only be understood by looking at its UML diagram, you already lost.
  
• Know your target audience and their work flows: When working as a programmer on a game project, the "target audience" for a feature may be other programmers, level designers, graphics artists or (most importantly) the gamer. Concentrate your first design efforts on that. How will the audience use your new feature? What will their work flow look like? Sit together with a level designer, graphics guy or tester and let them show you how they work or how they would like something to work. Don't expect them though to tell you exactly what to do. It's your job as a programmer to formalize their (usually completely unrealistic) expectations ;)
  

The Nebula3 Scripting System

Nebula2's scripting system implemented a script interface to C++ classes, where script commands mapped directly to C++ methods. From a technological view point this was a neat concept, but in the end, the Nebula2 scripting system was too low-level and fine grained for the people a scripting system is mainly targeted at: the level designers who need to script game logic and behaviour.

Level logic scripting usually happens on a much higher level then C++ class interfaces. Directly mapping script commands to C++ methods may introduce a dangerous complexity on the scripting level. Bugs are even more likely then in similar C++ code, because scripting languages usually lack strong typing and most "compile time" error checks, so that bugs which normally show up during compilation in C++ often only show up at runtime when using scripting (however this varies between scripting languages). This was an experience we learned from Project Nomads, which was scripted using the Nebula2 scripting system.

So the lesson learned was: do your scripting on the right abstraction level, and: mirroring your C++ interfaces in a scripting language is kind of pointless, because then you're much better off with doing the stuff directly in C++ in the first place.

Instead, the new Nebula3 scripting philosophy is to provide the level designers with a set of (mostly application-specific) building blocks on the "right abstraction level". Of course "the right abstraction level" is difficult to define, because a balance must be met between flexibility and ease-of-use (for instance, should a "Pickup" command move the character into pickup-range, or not?).

Apart from being too low-level, the Nebula2 scripting system also had some technical issues:

• C++ methods had to adhere to scripting conventions (only simple data types for parameters allowed)
• A hassle for the programmer. Every C++ method that was scriptable needed additional script interface code (several lines per method).
• Only nRoot-derived classes scriptable.
• Object-persistency hardwired to the scripting system (neat concept at first, but the additional dependency made refactoring very hard).

Here's how the Nebula3 low-level scripting system looks like:

• the base of the scripting system is the Scripting::Command class
• a Scripting::Command is completely scripting language independent, and consists of a command name, a set of input parameters and a set of output parameters
  
• a new script command is created by deriving a new subclass of Scripting::Command, the script command functionality is coded into the OnExecute() method of the Command subclass in C++
  
• script command objects must be registered with the ScriptServer singleton before use
• the ScriptServer is the only scripting-language-specific class in the Scripting subsystem, it will register Command objects as new script command, and translate command parameters to and from the scripting language's C-API

This concept is much simpler then Nebula2's and - most importantly - it isn't interwoven with the rest of Nebula3. It is even possible to compile Nebula3 without scripting support at all by changing a simple #define.

Of course, writing the C++ code of script commands would still be as boring as in Nebula2. Here's where NIDL comes in. NIDL is the cleverly named "Nebula Interface Definition Language". The idea is to reduce the repetitive work when writing script commands as much as possible by defining a simple XML schema for script commands and compile that XML description into the actual C++ code which implements a subclass of Scripting::Command.

The essential information needed for a script command is:

• the name of the command
• types and names of input parameters
• types and names of output parameters
• the actual C++ code (most often a single line of code)

Some less essential, but convenient information:

• a description of what the command does and what each parameter means for an automatic runtime help system
• a unique FourCC code for more efficient streaming over binary data channels
  

Most script commands translate into about 7 lines of XML-NIDL-code. The XML files are then compiled into C++ code by a "nidlc" NIDL compiler tool. This preprocessing is fully integrated into VisualStudio, so working with NIDL-files doesn't introduce any more hassle to the programmer.

To reduce file clutter (and resulting compile times), scripting commands are grouped into collections of related commands called a library. A library is represented by a single NIDL-XML-file, and is translated into one C++ header and one C++ source file. Script libraries can be registered at application startup with the script server, so if your application doesn't require or want file access from within scripts, just don't register the IO scripting library. This will also reduce the size of the executable, since the linker will drop the C++ code of the scripting library completely if not referenced.

Finally, Nebula3 drops TCL as its standard scripting language, and adopts LUA for its smaller runtime code size. LUA has become the quasi-standard for game-scripting, so it should be easier to find level designer already fluent in LUA coding.

Nebula3's Zip file system

Game applications often use file archives to reduce file system clutter and improve performance when many small files must be opened and read by the application. While Nebula2 used a proprietary archive format (NPK), Nebula3 uses standard Zip files. This has a number of advantages:

• no self-written tools required to create the archives, just use the zipper of your choice
• simple file encryption supported
• smaller disc footprint
• usually higher read performance because disc bandwidth is often the bottleneck, not decompression speed

The current implementation has a few disadvantages though:

• no write support (not a big deal, NPK's didn't support writing either, and game resources are usually read-only anyway)
• No random access (no seeks), this is a bit more critical, and could be solved with a more advanced implementation. Currently this is circumvented by decompressing the entire contents of a in-zip-file into memory and allow seeking on this in-memory-copy. This approach basically disables all types of file-streaming scenarios (especially streaming audio).
  

Accessing the content of Zip archives is completely transparent to the application once a Zip archive is mounted through the IO::Server::MountZipArchive() method. The IO::Server::CreateStream() method will check whether an URI actually is a file in one of the mounted Zip archives, and return a ZipFileStream object instead of a FileStream object if needed. The application just uses the returned Stream object as usual and doesn't need to care whether it's working with a "real" file, or a compressed file in a zip archive.

Streams, Readers and Writers

The Nebula3 IO subsystem provides Stream objects as generic data storage- and transfer-channels. Subclasses of stream implement or wrap specific data transfer protocols. Stream classes are usually associated with an URI scheme:

• "file:" scheme maps to IO::FileStream class
• "http:" scheme maps to Net::HttpStream class
• "mem:" scheme maps to IO::MemoryStream class

These are the URI schemes and associated stream classes that Nebula3 provides out of the box for now. Stream objects are usually created through the IO::Server::CreateStream() method, which takes an URI and returns a matching Stream object:

// create a HttpStream and a FileStream object
IO::Server* ioServer = IO::Server::Instance();
Ptr< Stream> httpStream = ioServer->CreateStream("http://www.radonlabs.de/index.html);
Ptr< Stream> fileStream = ioServer->CreateStream("home:readme.txt");

An application may associate its own stream classes with URI schemes using the IO::Server::RegisterUriScheme() method.

Streams only provide generic Read() and Write() methods to read and write chunks of memory. To access stream data in a more convenient way, StreamReader and StreamWriter objects are attached to the stream. Readers and writers provide specialized interfaces for reading or writing specific types of data. The Nebula3 IO subsystem comes with the following reader and writer classes:

• BinaryReader/Writer: read/write a stream of typed data elements in binary form
  
• TextReader/Writer: for reading and writing text data
• XmlReader/Writer: for reading and writing XML formatted data

Here's a code example to parse a simple XML file from a HTTP server:

// create a stream and attach an XML reader to it
Ptr< Stream> stream = IO::Server::Instance()->CreateStream("http://www.radonlabs.de/books.xml");
Ptr< XmlReader> xmlReader = XmlReader::Create();
xmlReader-> SetStream(stream);

// start reading...
if (xmlReader-> Open())
{
    // iterate through "books" xml elements
    if (xmlReader-> SetToFirstChild("books")) do
    {
       // read attributes from current element
       String author = xmlReader-> GetString("author");
       int year = xmlReader-> GetInt("year");
       int numPages = xmlReader->GetInt("pages");
       ...
    }
    while (xmlReader-> SetToNextChild("books"));
    xmlReader->Close;
}

Other Nebula3 subsystems may provide their own derived StreamReader and StreamWriter classes. For instance, the Messaging subsystem provides a MessageReader and MessageWriter class for serializing message objects.

The Nebula3 IO subsystem: URIs and Assigns

Nebula3 uses URIs (Uniform Resource Identifiers) instead of simple filename strings to locate IO resources. URI objects are usually created like this:

using namespace IO;
URI webUri("http://www.myhttpserver.com/index.html");
URI localUri("file:///c:/temp/text.txt");

The first URI points to a file on an HTTP server, the second URI points to a file in the local file system. When an URI object is created, it is parsed and split into its components:

• the scheme (for instance "http", or "file")
• the host ("www.myhttpserver.com")
• the local path ("index.html", or "c:/temp/text.txt")

URIs may also contain additional information:

• user info: describes login information for a secure service
• port number: describes a network port number
• fragment: usually describes a location inside an HTML file
• query: a set of key/value pairs often used by web services
  

Nebula3 has built-in support for the "file:" and "http:" schemes already. "ftp:" support is planned for the near future.

For instance, to copy a file from a HTTP server into a local file system you could do somethis like this:

URI from("http://www.radonlabs.de/index.html");
URI to("file:///c:/temp/index.html");
IO::Server::Instance()->CopyFile(from, to);

Assigns have already been used since Nebula1 (actually the concept and name stems from the original Amiga OS back in the 80's), they are aliases (or shortcuts) for file system locations and very handy to abstract the actual location of files in a Nebula application.

For instance, instead of "C:/Program Files/MyApp/export/textures/category/mytexture.dds", you would simply use "textures:mycategory/mytexture.dds". The assign "textures:" would be defined as "C:/Programme/MyApp/export/textures".

Assigns are especially useful to describe locations like the application's installation directory or the user directory of the currently logged in user. For this reason, Nebula3 defines a few standard assigns which are initialized at application startup:

• home: this points to the application's install directory
• bin: this points to the directory where the application executable resides
• temp: this points to a scratch directory which is guaranteed to be readable and writable for the current user
• user: this points to the user directory, in an English Windows, this is for example "My Files/CompanyName/AppName". The user: directory is guaranteed to be writable (unlike the installation directory), and this is the place where config data or save game files should reside.

In Nebula3, Assigns have been extended to work with URIs. For instance, the "textures:" assign could be defined as "http://www.radonlabs.de/myapp/textures", which would automatically cause textures to be loaded from a http server instead of the local file system. Assigns can be nested, so for instance the "textures:" assign could also be set to "home:export/textures", which would automatically resolve to the location "export/textures" in the installation directory.

Getting rid of virtual method calls

In Nebula2, platform abstraction was done through subclassing and virtual methods. For instance, the nGfxServer2 class implemented the platform-independent interface of the graphics server, and a specific subclass (for instance nD3D9Server) implemented the Direct3D9 version of the graphics server, overriding the virtual methods of the base class. Client code would then talk to the nGfxServer2 class interface and doesn't need to care whether rendering is done through Direct3D or some other rendering API.

Depending on the host platform, the performance of virtual method calls is somewhere from slightly bad to very bad, because the additional memory lookup may trash the cache, flush the instruction pipeline, disable branch prediction, etc... Virtual method calls are still the fastest way for runtime-polymorphism, but that's usually not necessary for platform-abstraction, where "compile-time-polymorphism" is often enough.

Nebula3 uses typedef-ing for platform abstraction, and thus eliminates most virtual methods and even enables inlining for frequently called methods without sacrificing platform-independent code.

This is done by first writing a base class which defines the class interface. The base class usually doesn't have virtual methods (except the destructor, since the base class is usually derived from Core::RefCounted). From the base class, a platform specific class is derived, overriding most or all of the methods defined in the base class with platform specific code. Finally, the platform specific class is typedef-ed to the proper platform-independent class name, which is then used by the client code.

Here's an example: let's say we want to implement the RenderDevice class in the CoreGraphics namespace. First, the class interface is defined in the base class:

namespace CoreGraphics
{
class RenderDeviceBase : public Core::RefCounted
{
    DeclareClass(RenderDeviceBase);
    DeclareSingleton(RenderDeviceBase);
public:
    /// constructor
    ...
};

} // namespace CoreGraphics

Note that the class name is RenderDeviceBase, not RenderDevice.

From RenderDeviceBase, a platform-specific subclass is derived, called D3D9RenderDevice, this is the Direct3D9-implementation of the RenderDevice class:

namespace CoreGraphics
{
class D3D9RenderDevice : public RenderDeviceBase
{
    ...
};

} // namespace CoreGraphics

Finally, there's the "proper" class header for the RenderDevice class (coregraphics/renderdevice.h), which performs the conditional typedef:

#if __USE_DIRECT3D9__
#include "coregraphics/d3d9/d3d9renderdevice.h"
namespace CoreGraphics
{
    typedef D3D9RenderDevice RenderDevice;
}
#elif __USE_DIRECT3D10__
#include "coregraphics/d3d10/d3d10renderdevice.h"
namespace CoreGraphics
{
    typedef D3D10RenderDevice RenderDevice;
}
#elif __USE_OPENGL__
#include "coregraphics/ogl/oglrenderdevice.h"
namespace CoreGraphics
{
    typedef OGLRenderDevice RenderDevice;
}
#else
#error "RenderDevice class not implemented on this platform!"
#endif

Client code just works with the RenderDevice class, and is completely unaware that it is actually using the D3D9RenderDevice or D3D10RenderDevice classes. All platform specific stuff is resolved at compile time, and all calls into the RenderDevice are normal method calls, not virtual method calls. This lets the compiler also do a much better optimization job (for instance inlining methods, better link-time code generation, and so on...).