It’s been quite awhile since we’ve released a new version, and for the most part that is because we’ve been working hard to chase down some of the more esoteric bugs and building down technical debt, something that isn’t all that sexy to discuss ? .
Some of the newer changes that have been brewing are the build system and asset routing system.
Build system
When we first started building out NodeBB, we ran our app the traditional way: node app.js. Start times were miniscule and life was good. In the years since (has it already been 4 years?), a number of significant changes have made their way into the startup process, slowing things down:
- Integration of the plugin system and its hooks
- Compilation of JavaScript and CSS assets into their respective minified files
- Linking of various bundled assets (sounds, plugin defined modules, etc.)
- Integration with the loader system
- General code bloat to handle different use cases of NodeBB
Most, if not all of these, require some degree of processing, and we ended up having to wait for each of these to finish before opening NodeBB up to traffic.
For example, if we served NodeBB before javascript or CSS assets were finished compiling, then users would be served outdated assets, or worst case, nothing at all. Over time, developing via ./nodebb dev became cumbersome, with server startup times taking up to 30 seconds on single-core machines.
The first thing we did was integrate grunt into our workflow. With selective recompilation, our development build times were reduced, but that masked the real problem, which was that startup itself took quite a long time. While this pain point only mattered during startup, it became a problem because during startup, the forum itself is down and inaccessible. For high traffic customers, even half a minute of downtime could be noticed by hundreds of users.
The problem was even more noticeable if the server itself crashed. Every time it did so, the forum would be inaccessible for up to 30 seconds.
Our latest approach is to sequester the build tasks to their own process, an idea which is not novel in itself.
By compiling new assets while NodeBB is running, we can eliminate the most time consuming tasks from the NodeBB startup process and begin serving users in under 5 seconds. NodeBB itself would still be running during the build process, leaving customers unaffected. In the event of a crash, the daemon would reboot the process and you would be back in business within seconds.
We’ve spent the past several months firming up the build process, in response to bugs reported by our early adopters, and while we were happy to release it for general consumption with v1.4.0, we’re also happy to see that there have been no significant issues so far.
Going forward, we’re looking to capitalise on ever advantage afforded by the build system, and push even more of our startup tasks to the build process. Thanks to hard work by our very own Peter Jaszkowiak, we now build language files as part of our build step, eliminating the need to just-in-time compile language requests.
Assets router
Slated for an upcoming release, we’re doubling down on our use of the /assets route (currently used to hold the compiled language files) by sequestering all uploaded assets (minified files, languages, templates, plugin static directories, etc.) in the /build/public directory, which is directly accessible via /assets on the browser/front-end. We’ve made the decision to maintain backwards compatibility for the upcoming release, and existing requests to assets in / will continue to work (e.g. /nodebb.min.js will still load, although the new route is /assets/nodebb.min.js).
We’ve bitten the bullet and made this change because our handling of the /public directory needed to be improved. There were uploaded assets, compiled assets, and default assets all mixed in one directory, and there was no easy way to direct users to clear out their compiled assets (e.g. "delete these files, these two folders, but don’t delete anything else!!"). In addition, the new assets route will allow a reverse proxy to very easily serve these compiled assets without the use of a complicated (and potentially outdated) location match string.
For example, with nginx, the old method of serving assets would be:
location ~ ^/(images|language|sounds|templates|uploads|vendor|src\/modules|nodebb\.min\.js|stylesheet\.css|admin\.css) { root /path/to/nodebb/public/; try_files $uri $uri/ @nodebb;
}
The new – much simpler – configuration, is:
location ~ ^/assets/(.*) { root /path/to/nodebb/; try_files /build/public/$1 /public/$1 @nodebb;
} location /plugins/ { root /path/to/nodebb/build/public/; try_files $uri @nodebb;
}
The first block configures nginx to directly serve any files requested under the /assets route from the corresponding folder in build/public. The second block configures nginx to load static directories that are served by plugins themselves. Utilising these blocks will reduce the work the NodeBB process has to do, and is advised for high-traffic deployments.
Peter’s changes also bring about slightly lower memory usage (as compiled minfiles are no longer kept in memory) and have simplified the use of IPC between multi-node deployments of NodeBB.
One of the big advantages introduced was the removal of the --local-assets flag, which was added to prevent NodeBB from saving files to disk (namely, nodebb.min.js and other compiled assets). This option was helpful for certain build environments that relied on running NodeBB off of a disk mount that did not have write permissions. Now with a discrete build step, the built assets can be easily mounted (or checked in, depending on various deployment strategies) and we’ll have reduced the number of items written to disk. Ideally, we’d like to get this to zero, with exception of uploaded assets.
While the majority of users will not experience any change in deployment workflow, one important change is that a manual ./nodebb build is required every time you upgrade or activate/deactivate plugins, otherwise any changed assets will not be propagated through to the files served to the client. For convenience, ./nodebb upgrade and ./nodebb reset routes will automatically run the build process, so potential stumbling blocks will be minimized. In addition, the "Restart" button on the ACP dashboard also works to kickstart the asset re-build process.
’til next time!