21 May 2009

Maya Plugin

[Edit: I added some clarification to the “cleanup” point  below]

I have started to write a new Maya plugin for Nebula3, which eventually may replace all (or parts of) our current plugin. The actual Maya plugin is only one small part of our asset pipeline, so this is not about rewriting the entire asset pipeline, just replacing one small gear in it. Otherwise it would be a truly Herculean task. In the beginning this will just be a private endeavour, free from time- or budget-limitations, so that no design compromises have to be made. This approach worked quite well for Nebula3 so it makes sense to use it more often in the future.

Our current Maya plugin is stable and fast, but at least the C++ part of it is beginning to show its age, it’s becoming harder to maintain, and since it’s based on Nebula2 code it is a lot harder to do low-level things like file io compared to similar Nebula3 code.

The new plugin will realize ideas I’ve been carrying around in the back of my head for quite some time, and which would be hard to implement into the existing plugin without a complete rewrite. The most important one is:

Separation into a platform-agnostic front-end and several specialized back-ends:
  • The actual Maya plugin will export into intermediate file formats which are completely platform-independent (and probably even somewhat engine-independent). Thus the plugin itself becomes more of a generic 3D engine exporter tool which doesn’t have to change every time a new target platform is supported or an engine feature is added or rewritten.
  • The back-end tools (or libs) convert the intermediate files into the files actually loaded by Nebula3. Those files can (and should) be highly platform-specific.

I’m expecting that the larger chunk of code goes into the platform-agnostic plugin, and that the back-ends are relatively small and straight-forward. The main advantage of this separation is better maintainability. The core plugin can remain relatively stable and clean, while the back-ends can have a higher frequency of change, and the “throw-away-and-rewrite” barrier is a lot lower since only the relatively small back-end-code has to be replaced without affecting the core plugin and the other platform-back-ends (too much). Also, the platform-mini-teams have more freedom to implement platform-specific optimizations into their engine-ports, since they have complete control over their exporter-backend.

The main disadvantage is that the export times will probably be a bit higher then now. A LOT of effort has gone into optimizing the performance of our toolkit plugin (exporting a scene with hundreds of thousands of polygon should only take up to a few seconds), and writing an additional set of output files may effect performance quite drastically. I’m planning to use XML only for the intermediate object hierarchy structure (which depends on the material complexity of the scene, but shouldn’t be more then a few dozen to a few hundred lines for a typical object), and to use binary file formats for “large stuff” like mesh and animation data. But if the XML files are hindering io performance too much, I will clearly go to a performance-optimized binary format, even if human-readability would be a major plus there (in the end, one set of the back-end tools could convert to human-readable ASCII file formats).

If you’re wondering why performance is so critical during export: consider that a project has about ten-thousand 3d models to be exported (which isn’t unrealistic for a complex RPG project like Drakensang for example). If the export time can be reduced by only one second per object, the time for a complete rebuild will be reduced by almost 3 hours! Actually, most 3d models batch-export in much less then 1 second in our build-pipeline, it’s the texture-conversion to DDS which eats the most build-time…

There are a lot of other things a Maya exporter tool should do right to be considered well-mannered:

  • It should support batch-exporting and automation (command-line batch-exporters, means of controlling export parameters on thousand of assets, standardized project directory structures, etc…).
  • It should be designed for a multi-project environment (a modeling artist or level designer must be able to quickly switch from one project to another).
  • It should of course offer a fast and exact preview for immediate quality control (the artist should be able to get an in-engine view of his work immediately)
  • It should not force artists to use archaic file formats. For instance, all texture conversion tools for the various console platforms I have encountered so far only accept crap like TGA or BMP as input, but NOT the industry standard PSD format! Quite baffling if one thinks about it. I don’t know how other companies deal with this, but I think it’s quite unacceptable to keep an extra TGA version for every one of tens-of-thousands textures around, just so that the batch exporter tools will work (for the console platforms we wrote our own wrappers, which first convert a PSD file to a temp TGA file and then invoke the conversion tools coming with the SDKs, but this is REALLY bad for the batch-export performance of course).
  • It should be fault-tolerant: Maya is an incredibly complex piece of technology, and plugins usually have no other chance then only supporting a specific subset of its features. The plugin should not crash or stop working when it encounters something slightly wrong or unknown in the Maya scene, instead it should provide the artist with clear warnings and readable error messages.
  • It should not require too many restrictions in the Maya scene: for instance, a very early version of our exporter tools required the artist to manually triangulate the scene, which is unacceptable of course.
  • It should cleanup the Maya scene during export: It’s relatively easy in Maya to create zero-area faces, or duplicate faces, or faces with a zero UV-area, etc... The exporter should remove those artefacts, and in those cases where an automatic handling is not possible, provide a detailed error log to the artist so that he has enough information to remove those problems manually. [EDIT: this was badly worded… of course the plugin should not modify the actual Maya scene, but instead remove artefacts from the data which has already been extracted from the Maya scene… it’s a bad idea to modify the Maya scene itself during export!]
  • It should optimize the Maya scene during export: For instance, the last time I looked at the XNA Maya plugin it exported a single material group for every Maya shape node, resulting in hundreds of draw calls for our simple Tiger tank example object. This is almost as bad as requiring the artist to work with a triangulated scene. Instead the plugin should try its best to optimize the scene for efficient rendering during export (like grouping polygons by material, sorting vertices for efficient vertex-cache usage, removing redundant vertices, and so on).

Of course this list could go on for a few more dozen points, there’s almost 10 years of work in our asset pipeline, and there’s probably more C++ and MEL code in it then in Nebula3 (which isn’t necessarily a good thing ;)


Tim said...

Hi Floh,

perhaps you should take a look at the Google Protocol-buffers. Its a small lib which generates code to serialise and deserialise data. You only have to write a so called proto file which defines your fileformat in a simple manner. You can use a binary or a human readable format. Perhaps it fits your requirements for the export fileformat.

There also implementations of protocolbufs for other languages. We are using the C++ and C# implementation of the protocolbufs. C# for the tools and c++ for the game.

I hope this is helpful for you.

Greetings Tim

JT said...

Will the Maya plugin be open sourced? I am in the middle of writing my own toolkit for Maya. I was going with the idea of having a file interpreter layer to allow writing to either binary, or text formats (or possibly MD5). The main toolkit just parses and cleans the necessary information needed out of the Maya scene.

isav said...
This comment has been removed by the author.
Floh said...

@Tim: I'll have a more thorough look at this. At first glance it looks similar to our NIDL stuff, so maybe I could use a modified version of this. Thanks for the tip.

@JT: I think that at least some sort of "core Maya export functionality" will be open sourced and normal part of the N3 SDK, but I don't know how far we will go there. Back-end-converters for console platform will definitely not be open-sourced though. I'm aiming for some sort of common plugin core for the public N3 SDK and our inhouse toolkit. But first I want to get some code up and running again before thinking about stuff like this :)

isav said...
This comment has been removed by the author.
André said...

Since you're thinking of doing it in two stages; generic everything-and-the-kitchen-sink export, and then a specific format, did you consider collada for the first stage export?
Reinventing the wheel can be fun... but maybe not the best use of time... :)

Floh said...

@Andre: Yes I did indeed consider Collada for the intermediate file format, but was frightened by its complexity. Writing a 100% Collada conformant exporter or parser looks like more work then the actual exporter code to me.

JT said...

The company I work for uses Collada as our intermediate file format. I would have to say, I am not interested in using it in this context again. I would definitely use it as a conduit to other programs, but it just kills iteration time.

Why do you need a text based format? How often do you need to hand edit or inspect the data? Personally I'm looking towards SQLite towards my next intermediate format. I can get at the data relatively easily (perl, C#).

André said...

The exporter for maya (and others) already exists, and there are downloadable libs to build a reader.
However, I admit I haven't used collada myself :) I just thought that it seems to partly do what you're doing. But if it's as bad as you (and others) say, than that's too bad. Sad even, since that's the whole point of its existance.
Btw, I agree with you that a two stage export is the way to go. It has many advantages, like you mentioned.
Anyhoo, keep up the good work! Looking forward to N3 !