Unveiling of the Read API

Developer empowerment has always been at the core of NodeBB:

• Plugins receive first-class treatment, in that there are plenty of hooks enabling plugins to interact with nearly all facets of NodeBB
• Themes play a central role in NodeBB, allowing designers to fully realise their vision without being hamstrung by unnecessary limitations.
• Our open source community is thriving due in no small part to our community forum of developers helping each other

However, one consistent pain point has been our API, and its rather opaque discovery model. More often than I care to admit, we'll receive a query: "Where are your Read API docs?", to which our response is a conciliatory "prepend /api to any route in order to see the underlying JSON". After all, we use the same APIs to render every single page in NodeBB, and if it's good enough for us, then it ought to be good enough for you! However, this sort of solution totally sucks (pardon my French) if you don't have experience working with NodeBB. The core developers all have the full context behind most parts of the code, so we know the full capabilities of the backend, and where to look when we want to edit something. On the other hand, developers new to NodeBB don't have this benefit, and given that we're not a small project, it can definitely feel overwhelming to try to find out how everything works, or where specific logic lives in NodeBB. The lack of consistent API documentation also implicitly enforces a "develop-first" approach, where you simply get started on a feature by diving into the code and figure out the plan and design later. While this approach works really well for hackers¹, it lets down the planners, and we want to take steps to ease development for both types of developers. Another downside of our pre-existing API is that it is not entirely RESTful. The API has been GET-exclusive for the most part. Whatever few places we don't do GETs are for uploads and non-JSON related exports. Almost every action to write to NodeBB is done via websockets, which we are intending to partially deprecate². Lastly, our API is mostly displaying what we need on the page (to render properly client-side). In many cases, that dovetails nicely with developer expectations, but the nomenclature and structure need revising in order to follow API best-practices. A quick example would be, /users shows the user list, but individual users are under the /user prefix (no plural).

What do?

The first order of business is to shine a light on the black box, our opaque Read API. The more documentation we have, the easier it will be to spot opportunities for improvement. More importantly, better documentation empowers developers with better context for their planning. We settled on the OpenAPI 3 specification (formerly known as Swagger), for a couple of reasons:

• A standardised API schema allows for consistent consumption via automated tooling
• We can achieve some spectacular out-of-the-box styling via ReDoc.

A huge thanks to Aziz Khoury for getting us started on this. He singlehandedly implemented a script that allowed for automatic generation of OpenAPI docs (including request and response schemas) by examining the content that went in and out of our community forum. Right off the bat, we had nearly complete, workable documentation! ?? The rest of it³ was done by hand by the core developers, and the document itself will be a living document, that grows with the project. So without further ado, introducing the NodeBB Read API Documentation! ??

So what?

If you look at the actual spec, it's not really all that exciting. It's a bunch of YAML that looks nigh-impenetrable to the laymen⁴. The real cool advantages come in two stages:

1. ReDoc

We integrated ReDoc to style our documentation. We loved the design and found that it gave our API some much needed polish. If you go to our Read API documentation now, you'll see ReDoc as the front-end serving the behind-the-scenes YAML file.

2. Testing Suite Integration

The OpenAPI document itself is tied in to our testing suite, and so every time a change is made to the public API, a corresponding change must be made to the OpenAPI document as well, otherwise the tests will fail. This step is important because it keeps us accountable and ensures that the documentation itself is up-to-date with the latest code. Documentation is often one of the hardest things to keep up-to-date, second only to writing the actual documentation, so you can imagine why a tool is needed to gently (or in some cases, no-so-gently) remind developers to keep the API documentation up to date!

3. Better change tracking

We will now also be able to see how the API changes between version releases, which begets the question of whether this aids us in our steps towards adopting semver.

Let's take a step back...

In March, I talked about taking a step back to view a project under a more objective lens. Oftentimes developers (myself included!) will dive so deeply into a feature or project, and become so invested in its success, that we don't often consider whether it (or its implementation) is a good idea in the first place. So let's take the time now to take a step back and view the OpenAPI spec under an objective lens: I will admit that initial adoption is painful. The integration into our testing suite will introduce a failure point during our development workflow). However, the benefits potentially outnumber the downsides:

• We should now be able to catch bugs where output is changed by accident, or if property types have changed.
• We gain savings in developer frustration, as new developers try to integrate with NodeBB for the first time.

Time will tell as to whether we will discover more benefits, or run into additional downsides. We are hoping that the Read API documentation is a step in the right direction for all parties involved.

What's next?

Now that we have a published OpenAPI spec for our Read API, the obvious next step would be to adopt this for the Write API as well. Our Write API capabilities are currently served by our Write API plugin. Currently, we're in the planning and initial development stages of merging the Write API into core. This is scheduled to land in NodeBB v2. The current documentation is maintained by hand, and does not offer any of the benefits of OpenAPI. Additionally, while the majority of the "write-type" calls in NodeBB are handled via socket.io, we intend to replace these calls with calls to the Write API. A potential stretch goal here would be strict versioning of the API with built-in backwards compatibility. We want to model this after Stripe's API, which allows you to pass in outdated payloads under a version header, and receive the response as though you were still calling that outdated API. ==== ¹ "hacker" in this context, refers to the mindset of figuring out how something works in order to extend or modify it. In modern times, the term has evolved to mean someone gaining unauthorized access to a resource, usually with malicious intent, but I am using it in the strictly exploratory meaning. ² WHAT?!, you say? All will be explained, in a future blog post. Don't worry, socket.io won't go away, we're just reinforcing our commitment to the Write API. ³ Pun not intended. ⁴ For what it's worth, it's pretty impenetrable by developers too. ^{Cover Photo} Sincerely Media on Unsplash