20 Dec 2013

Asset loading in emscripten and PNaCl

Loading data from a file on disk doesn’t look like a big deal in a normal C application:

int main() {
    // open file for reading
    FILE* fh = fopen("filename", "rb");
    if (fh) {

        // read some bytes
        char buffer[128];
        fread(buffer, sizeof(buffer), 1, fh);

        // close the file
        fh = 0;
    return 0;   

When doing a real-world game this simple approach has a couple of problems:

  • blocking: The above code is blocking, when reading from a fast hard disk this is probably not even noticeable, but try loading from a DVD or Bluray disk or some sort of network drive over a slow connection and the game loop will stutter
  • hard-coded paths: The concept of a current directory is often not portable, you can’t depend on the current directory being set to where your executable is. It is better to establish an absolute root location and have all filename paths in the game relative to that (of course how to establish this root location is platform-dependent again, for instance get the absolute path to the executable, and go on from there)
  • can’t use different transfer protocols: the above code works fine for local filesystems, but not loading data from a web- or ftp-server, and operations like creating a new file, or randomly seeking in a file may not be available with other protocols.

It is a good idea to restrict the type of file operations that a game can use, e.g.:

  • do we really need write and create access? An offline game may need to write save-game files and options, while an online game probably doesn’t need access to the local file system at all.
  • do we really need random seek? Randomly seeking in a file can be either impossible (HTTP) or slow because some mechanical device must be moved around, it’s often better to read a file straight into memory and seek there or to avoid such operations at all.
  • do we really need to iterate directory content? again, this can be either expensive (mechanical storage device) or impossible (in plain HTTP for instance)
  • do we really need free-form file paths? Games usually need to access very few places in the file system (the asset directory which is usually read-only, and maybe some sort of per-user writable location for settings and save-games)
  • do we really need access to file attributes? Stuff like last modification time, ownership, readable/writable. Usually this is not needed.
  • do we really need the concept of a “current directory”? This can be tricky for portability, and some platforms don’t have the concept of a current working directory at all

That’s a lot of features we don’t need in a game and which are also often not provided by web-based runtime platforms like PNaCl and JS. It helps to look at the HTTP protocol for inspiration, since that is where we need to load our data from anyway in the web scenario:

  • file system paths become URLs
  • only one read operation GET, which usually provides an entire file (but can also load a part of a file)
  • no directory iteration
  • no “write access” unless specifically allowed by the server
  • state-less, no current directory or current read position
  • operations can take very long (seconds or even minutes)

For a game which wants to load its asset from the web the IO system should be designed around those restrictions.

As an example, here’s an overview of the Nebula3 IO system:

  • all paths are URLs: Not much to say about this :)
  • a single root location: At application start, a root location is established, this is usually a file:// URL pointing to the app’s installation directory, but can be overriden to point (for instance) to an http:// URL. Loading all data from a web server instead of the local hard disk is done with a single line of code which sets a different root location.
  • Amiga assigns as path aliases: A filesystem path to a texture looks like this in N3: tex:walls/brickwall.dds, where the tex: is an “AmigaOS assign” which is replaced with an absolute path, incorporating the root directory.
  • all paths are absolute: there is no concept of a “current directory” in Nebula3, instead all paths resolve to an absolute location at runtime by replacing assigns in the path.
  • pluggable “virtual filesystem” modules associated with the URL scheme: URLs starting with file:// are handled by a different file system module than http://, plus Nebula3 apps can plug in their own filesystem modules if they want
  • stream objects, stream readers and stream writers: this is interesting in the web context only because there’s a MemoryStream object which is used to store and transfer downloaded data in RAM
  • asynchronous IO is really simple: more on that later in this post :)

Since Nebula3 is also used as a command-line-tools framework, the IO subsystem is a bit of a hybrid, which in hindsight was a design fault. There are still all these writing and file creation operations, blocking IO, directory walking etc… which makes the API quite bloated. In a new engine I would probably strictly separate the two scenarios, use the engine as a game framework only, which only supports very simple asynchronous read operations, and write the tools with another framework (or even other language, like python).

Asynchronous IO in Nebula3

Let’s look at async IO in Nebula3 a bit closer since this is the most interesting feature for web-based platforms. This is based on the “non-blocking future” pattern (or whatever you wanna call it) and depends on a frame-driven instead of event- or callback-driven application architecture.

Here’s some pseudo code:

void StartLoading() {
    // To start loading data we need to create an 
    // IO request object and "send it off" to the
    // IoInterface singleton for asynchronous processing
    Ptr<IO::ReadStream> req = IO::ReadStream::Create();

    // The IoRequest is now "in flight" and will contain
    // a result at some point in the future. Because we need
    // to check for completion in some later frame we need to
    // store the smart pointer somewhere
    this->pendingRequest = req;

    // ok, we're done for this frame...

void HandlePendingRequest() {
    // this function must be called regularly (e.g. per
    // frame) to check whether the async loading operation
    // has finished
    if (this->pendingRequest.isvalid() &&
        this->pendingRequest->Handled()) {

        // ok, the request has been completed, if 
        // the file was loaded successfully we get
        // a MemoryStream object with its content
        if (this->pendingRequest->GetSuccess()) {

            // actually load the data from the memory
            // stream and throw the request object away,
            // since all file data is in memory, we can
            // actually use the normal open/seek/read/close
            // pattern on the stream object

            // delete the request object, 
            // remember, this is a smart pointer :)
            this->pendingRequest = 0;

There may be less verbose or more elegant versions of this code of course, but the basic idea is that you start loading a file in one frame, and then need to check in the following frames if loading has finished (or failed), and get the completely loaded data in a memory buffer which can be parsed with “traditional” read and seek functions (and which is very fast since everything happens in memory).

This implies that the engine needs to know what to do while some required data has not been loaded yet. For a graphics pipeline this is quite simple by either rendering nothing or some placeholder while the data is still loading.

But there are cases where the code cannot progress without important data being loaded, or where it would be very tricky or impossible to implement asynchronous IO (for instance when integrating complex 3rd party libraries like sqlite).

If we could simply block this wouldn’t be a problem: the worst thing that would happen is that our game loop would stutter, but on web platforms we cannot simply block the main thread (it is easier on PNaCl where it is recommended to move the game loop into a separate thread, which then can block waiting for the main thread to process asynchronous IO requests).

For Nebula3 I fixed this with an additional application object state called the “Preloading Phase”. The idea is that the app enters this state outside of the normal game loop (for instance while displaying a loading screen), and during this state, populates a simple in-memory filesystem (basically just a lookup-table with URLs as keys and MemoryStream objects as values) with the asynchronously loaded data. When all data has been loaded (or failed to load), the app will leave the preloading phase (and hide the loading screen) and synchronous loader code will transparently get the data from the in-memory file system instead of starting an actual asynchronous IO request. Since all this preloaded data resides in memory this means of course that only small data and few files should be preloaded, and the majority of data should be asynchronously streamed on demand during the game loop. It’s really only a workaround for the few cases where synchronous access is absolutely necessary.

More details about here in one of my presentations: http://www.slideshare.net/andreweissflog3/gdce2013-cpp-ontheweb

emscripten and PNaCl details

Ok, almost done!

For the emscripten and PNaCl platforms I basically wrote a simple Nebula3 filesystem module which fires HTTP GET requests through he respective emscripten and PNaCl API calls, and copies the received data into MemoryStream objects, it’s only a few hundred lines of code each.

The main difference between the two platforms lies in the use of threading:

  • PNaCl works like “traditional” platforms, there are a number of IO threads (about 10, but that’s tweakable) each of them processes one IO request at a time, so that as many IO requests can be in flight as there are IO threads. Those threads also directly handle processing of the received data like decompression.
  • In emscripten, the IO calls (sending a HTTP request, and the callback when the response has been received) is handled on the main thread, but the expensive processing (e.g. decompression) of the received data is handed over to a WebWorker pool (usually 4 WebWorker threads). There can still be multiple IO requests in flight because the IO system doesn’t “wait” for an IO request to finish before firing a new one (but it is still throttled to restrict the number of requests in flight in case a lot of requests arrive in a very short time period).

The actual code implementation is straightforward so I’ll spare you the source code samples. The respective class in PNaCl is called pp::URLLoader, and emscripten offers a whole set of rather specialized C functions which all start with emscripten_async_wget. Both fire an HTTP request (emscripten does an XmlHttpRequest, and PNaCl presumably under the hood as well - this has some unfortunate cross-domain implications), and invoke callbacks on failure or when data has arrived. PNaCl needs a bit more coding work since data is received in chunks (and the receive callback can be called multiple times), while emscripten waits until all data is received before calling the received-callback once.

emscripten has more options to integrate the data with the web page DOM (for instance it can automatically create DOM image objects from downloaded image files), and it also has a very advanced CRT IO emulation layer (so you actually can directly use fopen/fclose after the data has been downloaded or preloaded), but I haven’t looked into these advanced concepts very closely since Nebula3 already does a lot of this layering itself.

There’s a similar filesystem layer for NaCl called nacl-mounts, but similarly to emscripten I didn’t look into this very closely since the low-level URL loading functions were a better fit for N3.

That’s it for today, have a nice Christmas everyone :)

Written with StackEdit.


Lucky Patcher said...

Lucky Patcher

mod apps said...


Arun said...

Thanks for sharing this amazing article and i think you know about lucky patcher and i think it works like a charm. Please post more article like this and i will visit here again.

Komal Jain said...

lucky patcher.apk

Theresa said...

I was looking for this information. Thank god, i have got it from here. best foot spa massagers for home.

Unknown said...

lucky patcher pc

Unknown said...

تحميل المتجر سوق جوجل بلاي ستور google play
download google play store

تحميل المتجر سوق جوجل بلاي ستور google play
Download Google Play store Apk Last Ver

تحميل برنامج لوكي باتشر الاصلي

Dine Kum said...

lucky patcher apk
whatsapp plus apk

The Tech Top 10 said...
This comment has been removed by the author.
Unknown said...

برنامج لوكي باتشر الاصلي

كينج روت


لوكي باتشر الأصلي

Unknown said...

تحميل المتجر جوجل بلاي

العاب بنات ستايل

العاب كرة

العاب طبخ

Unknown said...

Thanks for sharing this post, it was great reading this article! would like to know more! keep in touch and stay connected
Lucky Patcher
whatsapp plus

king root

Unknown said...

I must say you have pretty good guts about coding, do you have any other permanent blog for tutorials ? All i do to learn coding is watch free flix roku and learn.

harish sharma said...

lucky patcher download

moviecounter said...

trinus vr apk

Power clean apk

og youtube

bobby movie

vivavideo pro

Fildo apk

gangstar vegas mod apk

sms bomber

Rayanaly said...

download lucky patcher android worldfree4u

Unknown said...

Lucky Patcher can be downloaded through numerous stages, yet surprisingly, a large number of them will have a ton of promotions

No said...

download lucky patcher app
gb whatsapp

Namra Quadri said...

whatsapp group link
pubg wallpapers

Anonymous said...

Nice blog..! I really loved reading through this article. Thanks for sharing such a
amazing post with us and keep blogging... iot training in chennai | iot training in chennai quora | iot training and placement in chennai | iot training center in chennai | best iot training centre in chennai

sumama said...

peggo pro apk
iptv player latino para pc
walmart credit card login page
worldfree4u 300mb movies
you tv player pc

Nirmal Singh said...

How To Use Lucky Patcher

Emma plamer said...

Your work is just excellent. I really love your work and thanks for sharing these resources.
getmyoffer capital one
wawa credit card
price chopper direct connect

Rajesh said...

thanks for sharing this information
Artificial Intelligence training in Bangalore
python training in btm
python training in bangalore 
python training in bangalore marathahalli
best python training institute in bangalore
python training institutes in bangalore marathahalli
python training in jayanagar bangalore
python training in btm Layout
tableau classroom training in bangalore

Robbie Royce said...

Good information and, keep sharing like this.

Crm Software Development Company in Chennai

Savanah Eden said...

thanks for your extrodinary services , go ahead
Scaffolding dealers in chennai

swathima said...

We help you to get best websites and good ranking in search engines, visit us
Erp software development company in chennai
Professional webdesigning company in chennai
Best seo company in chennai
Crm software development company in chennai

Yadhina said...

Thanks for providing this information .I hope it will be fruitfull for me. Thank you so much and keep posting.web portal development company in chennai
professional web design company in chennai
smo company in chennai

seo company in chennai
sem company in chennai
twitter marketing company in chennai
erp software development company in chennai
best seo company in chennai

veronica said...

Thanks for providing this information .I hope it will be fruitfull for me. Thank you so much and keep posting.web portal development company in chennai
professional web design company in chennai
smo company in chennai

seo company in chennai
sem company in chennai
twitter marketing company in chennai
erp software development company in chennai
best seo company in chennai

veronica said...

Thanks for providing this information .I hope it will be fruitfull for me. Thank you so much and keep posting.scaffolding dealers in chennai

aluminium scaffolding dealers in chennai

shanthumA said...

Nice post, you provided a valuable information, keep going.

Prestashop ecommerce development company chennai
Prestashop ecommerce development company in chennai
Prestashop ecommerce development company
Prestashop ecommerce development company in india

Nandhakumar said...

Nice information keep sharing like this.

scaffolding dealers in chennai
Aluminium scaffolding dealers in chennai
Aluminium scaffolding hire

taishadarelene said...

Wow, Great information and this is very useful for us.

professional bridal makeup artist in chennai
best bridal makeup artist in chennai

Unknown said...

Great information and this is very useful for us.

post free classified ads in india

Sree said...

Nice information Keep going

vito food oil dealers in chennai
freezer with plastic body dealers in chennai

Allison said...

very useful information for us.

best cafe in chennai

Unknown said...

Good information

security agency in chennai
best security service in chennai

jose said...

Really nice post. Thank you for sharing amazing information.
Java Training in Credo Systemz/Java Training in Chennai Credo Systemz/Java Training in Chennai/Java Training in Chennai with Placements/Java Training in Velachery/Java Training in OMR/Java Training Institute in Chennai/Java Training Center in Chennai/Java Training in Chennai fees/Best Java Training in Chennai/Best Java Training in Chennai with Placements/Best Java Training Institute in Chennai/Best Java Training Institute near me/Best Java Training in Velachery/Best Java Training in OMR/Best Java Training in India/Best Online Java Training in India/Best Java Training with Placement in Chennai

Benish said...

Really nice post. Thank you for sharing amazing information.
Python training in Chennai/Python training in OMR/Python training in Velachery/Python certification training in Chennai/Python training fees in Chennai/Python training with placement in Chennai/Python training in Chennai with Placement/Python course in Chennai/Python Certification course in Chennai/Python online training in Chennai/Python training in Chennai Quora/Best Python Training in Chennai/Best Python training in OMR/Best Python training in Velachery/Best Python course in Chennai

Earnmoneyonline said...

Nobody like to pay for advance features of any games don't like to spend money on uc, gems and all. We so irritate if we see any ads while playing game and watching videos. So here i am with a solution of all the problem name as Lucky patcher. Lucky patcher is famous mods app till now which give you advantage to crack any game license, block ads and allow you to remove unwanted apps from your phone. Click now on download lucky patcher to download apk and tutorial how to use it.

DumpsPass4sure said...

I can never name any other exam material than Pass4sure Checkpoint dumps because this is the only dumps which I trust. I prepared from this study material and aced my exam by the first attempt. Pass4sure Checkpoint pdf dumps material provided me all the information about the course outline and I succeeded by solving all the questions in the final test.

AE clicks said...

Ae clicks is an award-winning SEO company in Abu Dhabi, which is consistently ranked among the Top 10 Digital Marketing Companies in Dubai since last few years.Our team of experienced account managers and web specialists work closely with all our clients to develop a better understanding of their business and grow their business online, digital marketing goals, and corporate objectives.

fullassignment said...

Get the programming assignment help by the experts of FullAssignment.com. We have a huge list of academic writers who are aware of all the skills required for curating assignment writing work. fullassignment has a group with aptitude and involvement in online assignment help. For more info please visit our website:
or reach out us on whatsapp - (+1) 669-271-4848

Unknown said...

frozen 2 movie download

Rohit Kumar said...

Alone Status in Hindi

Citaent said...

Wedding DJ, Photo Booth and Karaoke - Caught in the Act Entertainment in Eugene, Lane County, Oregon

Photo Booth - Caught in the Act Entertainment in Eugene, Lane County, Oregon

DJ, Photo Booth, Karaoke and Party Rentals - Caught in the Act Entertainment in Eugene, Lane County, Oregon

Karaoke and Event DJ - Caught in the Act Entertainment in Eugene, Lane County, Oregon

Online Booking - Best Wedding DJ, Event DJ, Birthday DJ, Party Rentals, Photo Booth and Karaoke in Eugene, Lane County, Oregon

Caught in the Act Entertainment - Wedding DJ, Event DJ, Birthday DJ, Photo Booth and Karaoke in Eugene, Lane County, Oregon

Indian App Developer said...

Great blog, but developing an application in react native gives you fast and attractive app as compare to other frameworks, this makes react native best cross platform app development framework.

ALI ADIL said...

interior designing courses in Lahore
interior design course
autocad course in Lahore
3d max course in Lahore
web development courses in Lahore
web designing course in Lahore
web development courses
wordpress course

Big Data Analytics Malaysia said...

If your looking for Online Illinois license plate sticker renewals then you have need to come to the right place.We offer the fastest Illinois license plate sticker renewals in the state.

big data analytics courses malaysia

Abhinavhyd said...

Hey, thanks for this great article I really like this post and I love your blog and also Check marketing analytics certification in Hyderabad at 360DIGITMG.
360Digitmg marketing analytics in hyderabad

Qwerty said...

Al Qur'an Keutamaan Doa Abu Darda RA Syekh Abdul Qodir Jailani Rahmat Allah SWT Malaikat Mazhab Hanafi Shalat Tahajud Shalawat Nabi Muhammad Shallallahu 'Alaihi Wa SallamCara Wudhu Nabi Muhammad Saw

James David said...

Freelance maintenance technicians also benefit from establishing relationships with multiple clients. Working as a freelancer, you’ll gain employment experience with a variety of businesses, boosting your overall skill set and providing you with more work opportunities in the future.

maintenance technician

technical raja said...

67 frienship quotes

# suvichar

anmol vachan

68 suvichar in hindi

Vincent Patrick said...

It is a great post. Nevas Technologies has been providing Dynamics GP (Great Plains) services for over a decade to end clients, ISVs and Dynamics GP partners. We offer full service, one-stop-shop consulting services expertise of implementing and customizing Dynamics GP from version 7.0 has enabled us to build a highly competent team of GP consultants and developers with sound knowledge on the functional and technical capabilities of the solution. We are a leading provider of Dynamics GP (Great Plains) managed services/support.

Lokesh Reshwal said...

attitude shayari

dard bhari shayari

bevfa shayari

friendship shayari

ladki patane ki shayari

hager ali said...

التنظيف المثالي للتنظيف بجازان
السفير المثالي للتنظيف بالدمام
الفارس المثالي للخدمات المنزلية في حائل
شركة الطائي للتنظيف بحائل
شركة الامانة للتنظيف بحائل
شركة النخيل للتنظيف بحائل