14 Apr 2009

New N3 SDK (Apr 2009)

The new SDK release should be ready by tomorrow (unless there's some last-minute fuckup). For now here's a rough What's New list since the last SDK from September 2008:

Tools:
  • new command line tool: archiver3 - multiplatform wrapper for generating file archives
  • new command line tool: n2converter3 - convert .n2 files to .n3 files (Nebula graphics objects)
  • new command line tool: suiconverter3 - batch converter for SUI resources (simple user interface), currently only useful for the Wii port
  • new command line tool: synctool3 - sync local project directory from build server (only useful with our inhouse asset pipeline)
  • new command line tool: countlines3 - count N3 source code lines and generate Excel-compatible csv-file (comma separated values)
Foundation Layer:
  • brought Wii port uptodate (not part of the public SDK of course)
  • Util::CmdLineArgs renamed to Util::CommandLineArgs
  • Scripting subsystem moved into an addon
  • Remote subsystem moved into an addon
  • new Macros __ConstructInterfaceSingleton / __DestructInterfaceSingleton
  • new standard define __MAYA__ if compiled as part of a Maya plugin
  • new concept: ExitHandlers (see Core::ExitHandler)
  • new low level debug feedback method Core::SysFunc::MessageBox()
  • new concept "root directory" (see CoreServer::SetRootDirectory)
  • various changes to enable using N3 code in N2/Mangalore apps (N2 now sits on top of a very slim N3 Foundation layer)
  • SizeT and IndexT now signed (had to be done for N2/Mangalore compatibility unfortunately)
  • IO::Console is now an InterfaceSingleton
  • various debug HTML page handlers can now sort table content by columns
  • Debug::DebugServer is now an InterfaceSingleton
  • added a minimal Debug::HelloWorldRequestHandler as example for a simple HttpRequestHandler
  • new comfort method Http::HtmlPageWriter::TableRow2(), saves code when creating a 2-column HTML table
  • added Http::HttpClient class, allows send requests to HTTP servers
  • Http::HttpInterface is now an InterfaceSingleton and derived from Interface::InterfaceBase
  • new class Http::HttpRequestWriter
  • new class Http::HttpResponseReader
  • added a "single thread mode" to HttpServer
  • added new classes InterfaceBase and InterfaceHandlerBase
  • moved IOInterface-stuff under IO
  • new class Debug::ConsolePageHandler, displays console output in web server
  • generalized Zip filesystem stuff into general archive filesystem, with ZIP support as one specialization (on Wii, ARC files are used instead)
  • ZipFileSystem is now an InterfaceSingleton (no more per-thread wasted memory for archive table-of-contents)
  • added an "AsString()" method to IO::FileTime
  • new methods in Win360FSWrapper: SetFileWriteTime(), GetAppDataDirectory(), GetProgramsDirectory()
  • moved IO::ZipFileStream class to io/zipfs
  • moved path assign methods from IoServer into new InterfaceSingleton: IO::AssignRegistry
  • new standard assigns under Windows: "appdata" and "programs", mainly useful for tools
  • bugfix in IO::ExcelXmlReader for tables with empty cells
  • new class IO::HistoryConsoleHandler, captures console output into a ring buffer (used by Debug::ConsolePageHandler)
  • moved URI scheme methods from IoServer into IO::SchemeRegistry InterfaceSingleton
  • removed critical section from Stream::Open / Stream::Close
  • new method: IO::XmlWriter::WriteComment()
  • new methods specialized float4-loading methods: Math::float4::load_float3(), Math::float4::load_ubyte4n_signed()
  • vector comparison methods in Math::float4 more intuitive and flexible
  • Math::matrix44() default constructor now sets object to the identity matrix (default constructor still empty in Math::float4!)
  • new constructor from float4 in Math::quaternion
  • moved lots of math functions from scalar.h to platform-specific d3dx9_scalar.h to enable platform-specific optimizations
  • Memory::Heap constructor now accepts initial and maximum heap size
  • memory leap detection for Memory::Heap (doesn't quite work as expected yet)
  • removed global heaps: SmallBlockHeap, LargeBlockHeap, StringHeap
  • added global heaps: PhysicsHeap, AppHeap, StringObjectHeap, StringDataHeap
  • new experimental Win360MemoryPoolClass
  • Messaging::AsyncPort: can now add message handlers to already opened ports
  • added "deferred handled" flag to Messaging::Message (used in the rendering thread to keep CreateEntity messages around until their resources are loaded)
  • wrapped socket and TCP/IP classes to enable "TCP/IP-over-HIO2-Tunneling" on the Wii
  • System::Win32Registry(): split Read() method into ReadString() and ReadInt()
  • System::ByteOrder::Convert<TYPE>() methods now return a value, added new methods System::ByteOrder::ConvertInPlace<TYPE>()
  • new methods Threading::Interlocked::Exchange() and CompareExchange()
  • Win360Thread::IsRunning() now uses the Win32 function GetExitCodeThread() to detect whether the thread is running
  • new method Threading::Thread::YieldThread() (gives up time-slice)
  • new class Threading::SafeFlag
  • new concept: Timing::MasterTime and Timing::SlaveTime, main thread has a MasterTime object, and distributes "main time" to slave threads (i.e. the render thread)
  • Util::Array now has a MinGrowSize and MaxGrowSize to prevent excessive memory waste for huge arrays
  • new experimental class Util::Delegate (doesn't work in Codewarrior unfortunately)
  • new method Util::FixedArray::AsArray()
  • new methods Util::Dictionary::KeysAs<TYPE>(), and Util::Dictionary::ValuesAs<TYPE>()
  • new class Util::PriorityArray (which actually isn't needed anymore I think)
  • new class Util::RandomNumberTable
  • new class Util::Round
  • made many non-essential Util::String methods non-inline
Render Layer
  • new Character subsystem (modular character rendering, fresh rewrite of N2's Character3 system)
  • new CoreFX subsystem (ported from Mangalore's VFX subsystem (visual effects)) -> NOTE: will be moved into addon
  • new CoreUI and UI subsystems (simple user interface system) -> NOTE: will be moved into addon
  • new Video subsystem (video playback, currently Xbox360 only) -> NOTE: will be moved into addon
  • new Particles subsystem (rewritten from scratch) -> NOTE: will be moved into addon
  • new PostEffect subsystem (ported from Mangalore) -> NOTE: will be moved into addon
  • new Vibration subsystem (game pad vibration support) -> NOTE: will be moved into addon
  • new Vegetation subsystem (Drakensang's grass renderer, currently broken under N3) -> NOTE: will be moved into addon
  • new concept: RenderModules, clean framework to add functionality to the render thread
  • new concept: AnimEvents, animations can emit events at certain sample times (i.e. for playing foot-step sounds at the right time)
  • new concept: character attachments (swords, etc...)
  • lots of bugfixes and improvements to Animation system
  • new concept: AnimDrivenMotion, synchronize character movement with its current animation
  • new concept: batched messages, drastically reduce communication overhead between threads by client-side batching of messages
  • new methods: Audio::AudioEmitter::Pause() and Resume()
  • new methods: AudioDevice::SetGlobalVariable() / GetGlobalVariable()
  • added/fixed Xbox360 support to XACT audio classes
  • new class: MultipleRenderTarget, wraps MRT rendering
  • new class: MouseRenderDevice (currently only implemented on Wii)
  • added support for GPU-instanced-rendering
  • RenderTarget: added support to resolve depth buffer into texture (Xbox360 only)
  • RenderTarget: added support to create a resolve-texture which can be efficiently read by the CPU
  • added "late-binding" to ShaderVariableInstance
  • D3D9StreamTextureLoader and D3D9Texture moved from win360 to d3d9, since specialized Xbox360 versions exist now
  • Debug::MeshPageHandler can now display a dump of the vertex data in the web browser
  • Debug::TexturePageHandler now displays the current resource state of textures (Initial, Pending, Loaded, etc...)
  • new class CoreGraphics::MemoryMeshLoader
  • renamed CoreGraphics::Shape to CoreGraphics::RenderShape (because of CodeWarrior problems with identical filenames in different directories)
  • added Multiple Render Target support to Frame::FramePass and Frame::FramePostEffect
  • Graphics::Display::GetDisplayMode() now returns display mode actually set by the CoreGraphics::DisplayDevice (may differ from the requested display mode)
  • Graphics::GlobalLightEntity: all light parameter changes are now transferred to the render-thread-side post-creation
  • Graphics::GraphicsEntity: internal entity handle now only becomes valid after resources have been loaded on the render-thread-side
  • Graphics::GraphicsInterface now uses batch messaging to communicate with render-thread (only 1 message sent per frame)
  • Graphics::Handle is now a smart pointer (fixes problems where render-thread graphics entities were disposed too early)
  • Graphics::ModelEntity: support for AnimDrivenMotion
  • new Input::GamePad methods: ButtonAsString(), AxisAsString(), GetStateAsInputEvents()
  • InternalGraphicsEntities are now registered with the InternalGraphicsServer
  • lots of changes in InternalGraphicsServer and InternalModelEntity which require a proper cleanup doh
  • added support for 2-sided lighting to global lights
  • new classes: Models::AnimatorNode and Models::AnimatorNodeInstance (legacy Nebula2 stuff)
  • new classes: Models::CharacterNode and Models::CharacterNodeInstance, integrate character rendering with model nodes
  • loading of ModelNodes completely rewritten (new .n3 file format, plus n2converter3 tool to convert .n2 files to .n3)
  • new Model::OnResourcesLoaded() method, if Model subclasses need to do initialization work after resources have finished loading
  • new methods to lookup ModelNodeInstances on ModelInstances
  • new class Models::StreamModelLoader
  • lots of other minor changes in namespace Models
  • new class: RenderUtil::MouseRayUtil, convert 2D mouse position into world-space 3D-ray
  • new method: ResourceManager::CheckPendingResources(), returns true when there are currently no pending resources waiting to be loaded
  • SharedResourceServer: several methods now accept a ResourceLoader object when creating shared resources

I also did some interesting line counting statistics recently for the Foundation and Render layers (hopefully the images won't be scaled down too much):

foundation_layer 

render_layer 

"General" is the platform-agnostic code, which is the same for all platforms, "Win360" is the code which is identical between Win32 and Xbox360.

13 comments:

brmassa said...

Just GREAT! I will check Google Code as soon its all commited!

雷宏 said...

Oh, yeah!
Is is a really best gift for the day.
let's just cheers!

雷宏 said...
This comment has been removed by the author.
风之翼CC said...

that's just amazing!
I check this every day and night,thx floh!

Flying Bag said...

This is just awesome!!

P.S. Floh, I've been trying to wrap the nebula3 foundation layer with CLR (.NET). On the first attempt I've got a full list of managed/unmanaged incompatibility errors and I'm not sure what library in the foundation layer I can investigate first. I'll continue to do so but I'd appreciate if you can give me some tips on that. To me there are inter-relations between packages in the foundation layer in quite a few places.

Will checkout too the long waited SDK update.

Thanks

xuebai5 said...

That's Great!!!!!!!!!!!!!!!!!

isav said...

I sware I had a dream about this tonight. Thanks!

Kim, Hyoun Woo said...

Great News! Never ever imagined the update might be done so very quickly like this. :-)

I'm looking forward to see vertex compress related stuff(It was not completed in Sep.2008 as I saw the code)and character stuff.

Also I've update Nebula2 3dsmax Toolkit yesterday(see it on my blog page) Hope it might be useful with upcoming N3 release.

Thank you for the sharing!

Larry said...

Excellent!
Looking forward to the release, Thanks.

Floh said...

@Flying Bag: for a new "port" it's best to start with Memory, then continue with System, Util, Timing, Threading. More complex systems like IO, Net or Http have the most dependencies and should be done last.

A very important method to keep an eye on is Core::SysFunc::Setup(). In the beginning, you should comment-out some of the calls in it (like InitNetwork, the "SetMyThreadName" call.

Drazen said...

Any chance to compile it under linux with gcc ?

Floh said...

A Linux version would require a proper platform-port. It's doable with probably about one or two months of concentrated work. GCC itself shouldn't be too much of a problem, N3 compiles cleanly even with CodeWarrior, which is probably the most exotic and outdated C++ compiler on earth.

isav said...

Floh, could you be so kind to provide us future approach of the N3?