Why we moved from $EXCITING_TECHNOLOGY to $BORING_STANDARD
Introduction
This article was inspired by the May 2020 blog post Why we at $FAMOUS_COMPANY Switched to $HYPED_TECHNOLOGY, but while that article is satire, this one is completely serious.
This article is the first in a follow-up series regarding our work leading up to NodeBB v2 and beyond. The original article about NodeBB v2 can be found here.
Move Fast, Break Things
When we first started NodeBB, we wanted to capitalize on the newest technology stack available. After all, what's newest is clearly an indicator of market direction, and therefore going to stick around forever!
The decision was a little more nuanced, but by and large, this is why we ended up investing so heavily in our websockets implementation, up until a couple years ago.
Don't worry, this isn't a hit piece on websockets! It's served us very well over the years, and we're not planning to get rid of it anytime soon1.
What happened to exciting?
As it turns out, exciting technology has a couple drawbacks. None that were showstoppers, mind you -- but significant enough to give us pause long enough to consider whether we were using exciting technology for the right purposes. Were we shoving a square peg into a round hole?
Browser and Proxy Support
Websockets are not entirely supported by 100% of clients. Support is nearing 100%, but you'll always have some users who will complain that feature X or feature Y is not working, because they're behind some sort of firewall, or their ISP is blocking websockets, or maybe because Jupiter happens to be aligned with Venus on that particular day.
Conversely, a REST API is supported by 100% of clients. If they can load a website, they can load an API.
We also had an issue with websockets being too new at the time, but this is no longer an issue, since nginx has had websocket support for nearly a decade now.
Websockets also needed some changes to a reverse proxy config to get working. This hasn't changed from day 1.
When you have a hammer, everything looks like a nail
We were so enamoured with websockets that we implemented almost everything with it. We were doing simple call-and-response requests with websockets, simply because it was just so easy to .emit() something on the server, and get it on the client, or vice versa.
We didn't want to put in the legwork to build out proper RESTful routes, especially if we were requesting one-off bits of information, but in the end, that would've been the better approach.
Now that we do have a standardized REST API, along with the associated helpers on the backend and frontend to make implementing routes easier, we're actually in the unique position of being able to move more slowly and deliberately, ensuring that implemented routes make sense from a RESTful point of view.
RESTful opinions
I hear often that REST is a standard that sets out how an API is to be designed. While this is true to an extent, many APIs do not follow these rules rigidly, and some legacy APIs pre-date these opinions.
A problem with websockets was that it was vague enough that it didn't have any best-practices. Do you use periods as separators in your messages, in order to denote namespace, or pipes? Maybe dashes? In the end, it was up to whoever implemented it, and in some places, we had different standards co-existing.
While this pitfall can certainly happen in a RESTful API, it was much easier to build out our API more deliberately and slowly, as by this point, we had reached feature parity. Enforcing things like GET for data retrieval, POST for data creation, PUT for data update, and DELETE for data deletion were easier to enforce as it was an already established best-practice2.
Developer Friendliness
While it was easy for us to get started with websockets, it's not a ubiquitous technology that all developers know how to use. We were approached by several clients who wanted to integrate with NodeBB, but had no option except to build out a websocket client. In terms of developer simplicity, it was an order of magnitude more difficult to do this, compared to sending over our write API docs, and a sample cURL:
curl -X POST -H "Authorization: Bearer mytoken" https://mysite.example.com/api/v3/users --data "username=myUser"
I expanded on this at length in a companion blog post about our Read API.
Very soon after we published the Write API plugin, we started seeing a noticible uptick in integrations and questions regarding integrations, which told us clearly that there was a demand for it. We've had the websocket interface all along, but very very few people approached us with questions on using it.
$EXCITING_TECHNOLOGY Websockets isn't going away soon
The primary use for websockets is for real-time events that require low latency. For example, a browser-based game would benefit from a websockets implementation to achieve near-real-time client-server communication. While the content served by a community forum will never be so time-sensitive as to require sub-second latency, the benefits of having a websocket implementation still apply.
Since it's already implemented in our codebase, our use of websockets to send packets of information to a client is still the ideal use case -- for example, notifying other browsing users when a new topic is posted. While we may investigate other boring technologies like server-sent events, our websocket implementation is still here to stay for the time being.
Ever since we migrated the write API into core, we've been deprecating websocket routes here and there in favour of the RESTful API. This is a long-running work in progress, and we don't expect to fully-deprecate websockets until at least 2-3 major versions from now (if ever).
Notes
Cover Photo Tim De Pauw on Unsplash
1 However, "soon" in the context of NodeBB could mean bloody well anything, from 5 minutes, to 5 years. Your mileage may vary.
2 Ironically, we don't actually use PATCH methods, but we should, for routes that allow you to update only certain values (e.g. user update routes)