The core idea of The Computer Science Book is that computing is a teetering tower of abstraction. By understanding the design and function of the levels below, we can more intuitively understand the behaviour of the system. Asynchronous programming in JavaScript is an excellent example of this.
We saw in the previous chapter how JavaScript runtimes implement an asynchronous programming model using a task queue fed by background worker threads. Callbacks map neatly to the tasks executed by the runtime. By delving into the bowels of our browsers, we improved our mental model of how JavaScript code is actually executed.
That execution model in turn forms the basis for further abstraction. In this post, we’ll examine how the callback-based programming style makes it more difficult to structure complex, asynchronous operations. It turns out that callbacks are too “low level” and are better used as the building blocks for more powerful abstractions in the form of promises and async/await.
The aim of this post is not really to introduce the various forms of JavaScript syntax – there are a billion blog posts already doing that well – but to demonstrate how they develop new abstractions.
Structuring asynchronous dependencies
Studying the implementation of JavaScript runtimes is not just an exercise in computer science arcana. We know that an asynchronous operation is performed by passing the task off to a background worker and scheduling the provided callback as a task when the operation is complete.
The single JavaScript thread executes the program as a sequence of discrete tasks. From within JavaScript code, we have very limited control over what is executed when. In particular, it is difficult to express logical dependencies between operations in an elegant and composable way.
In other words, how do we express that an operation cannot execute before it has the results of other, independent, asynchronous operations?
In a more conventional, synchronous-by-default language such as Python or Ruby, such problems don’t really arise because the ordering of operations in code reflects both the logical order and the temporal order. JavaScript only guarantees that the operations within a task (e.g. a function body) are executed synchronously. If we want to sequence one asynchronous operation after another (i.e. there is a logical dependency between them), we need to explicitly structure our code so that this happens.
“Sequencing asynchronous operations” sounds like a terribly obtuse thing to do. In fact, it’s incredibly common. Imagine a web server handler that:
- Receives the request body
- Asynchronously validates the body
- Retrieves some data from the database
- Asynchronously generates a response body
- Dispatches the response
JavaScript has gone through a few growing pains as it developed abstractions to help programmers better provide the necessary structure. We’ll start with callbacks, then look at promises and round things off with async/await.
Callbacks
Recall that a callback function forms the content of a new task and provides a way for the runtime to call back to the JavaScript thread when an asynchronous operation completes. Callbacks appear throughout the browser APIs (e.g. setTimeout, addEventListener) but became obscenely ubiquitous in node.js.
Since a callback executes later, when the operation is complete, a sequence of asynchronous operations can be chained by calling the next one from with the previous one’s callback. Code structured in this way often ends up as a pyramid of nested callbacks, affectionately known as callback hell:
function handleRequest(req, callback) {
validateRequest(req, (error, params) => {
if (error) {
return callback(error);
}
fetchFromDB(params, (error, results) => {
if (error) {
return callback(error);
}
generateResponse(results, callback);
});
});
}
handleRequest(req, (error, response) => {
if (error) {
console.error(error);
}
console.log(response);
});
Here, each nested callback represents a new task. It’s possible to make things look a little less hideous by replacing the anonymous functions with named functions defined at the top level. That reduces the indentation at least. No matter what you do, though, you can’t get away from the fundamental weirdness that the program’s control flows through the callbacks. For new developers especially, it is difficult to work out the flow of execution.
Even once you’ve got used to it, callback-based code is difficult to work with. Error handling becomes more verbose – notice all those repeated error checks – and almost anything you might do requires writing yet more callbacks in a seemingly endless profusion.
As an example, it is trivial to write a synchronous function both that returns the value of both input functions:
function both(a, b) {
return [a(), b()];
}
When a and b are asynchronous and expect callbacks, things get more complicated. both now needs to also take a callback and there’s lots of additional boilerplate to ensure that we only call it when both values are ready:
function both(a, b, cb) {
let a_, b_;
const onComplete = () => {
if (a_ === undefined || b_ === undefined) {
return;
}
return cb([a_, b_]);
}
a(aVal => {
a_ = aVal;
onComplete();
});
b(bVal => {
b_ = bVal;
onComplete();
});
}
I won’t blame you if that just made you throw up in your mouth a bit. I’m sorry.
The nature of asynchronous code is that a variable’s value depends on when the current scope is executing. An asynchronous operation’s result only exists in the scope of its callback. That makes sense because, as we’ve just seen, the callback task is only created when the operation completes. We must be careful to structure our code so that tasks needing the result are only scheduled when the result is available.
The problem is that asynchronous JavaScript looks very different to normal, synchronous JavaScript and is difficult to integrate without writing lots of repetitive boilerplate.
Whenever you find yourself writing lots of repetitive code, it’s worth pausing to raise your head from the grindstone and stop to ponder. Is there some kind of abstraction that we are missing?
Promises
The first revolution in asynchronous JavaScript was the promise. They were originally implemented by a variety of competing libraries before being promoted to first-class values in ES6.
A promise (also known as a future in other languages) is a value representing the result of an asynchronous operation. While the operation is pending, the promise has no value. At some future point, when the operation completes, the promise’s value resolves into the operation’s result (or an error in the case of failure).
Since promises are values just like any other JavaScript value, they can be stored in data structures and passed to and from functions. This is the abstraction we’re missing. Promises introduce the concept of asynchronous values.
Imagine a library that performs asynchronous operations e.g. a network request. Without promises, all the library can really do is allow consuming code to register callbacks for when the operation completes, which requires the consuming code to have a certain structure. With promises, the library can return the pending operation itself and let the consuming code do whatever it wants.
Consuming code can register handlers to execute when the promise resolves by calling .then(), which returns the same promise to allow for chaining. Assuming that handleRequest is amended to return a promise, the example above can be rewritten like so:
function handleRequest(req) {
return validateRequest(req)
.then(fetchFromDb)
.then(generateResponse);
}
handleRequest(req)
.then(response => console.log(response)
.catch(err => console.error(err));
The beauty of promises is that, as normal JavaScript values, they can be manipulated and composed just like other functions. Here’s how both might look when a and b return promises:
function both(a, b) {
return Promise.all([a(), b()]);
}
Almost identical to our original, synchronous example! Promise.all takes an arbitrary number of promises and returns a new promise that only resolves when all of the input promises have resolved (or any fails). Since both returns a promise, it can be dropped into anywhere expecting a promise. Complex, asynchronous behaviour is much easier to express with promises than with callbacks.
Yet promises are no panacea. Asynchronous operations have to be sequenced as a chain of .then() calls, which might be better than nested callbacks but are still very different to standard, synchronous code. Worse, if an error occurs in a promise it is rejected, requiring the .catch() handler at the bottom of the chain. This is different to the normal try / catch error handling in JavaScript and can create subtle bugs if you don’t handle everything properly.
We still have to jump through all these syntactic hoops to get things working as we want. Wouldn’t it be better if we could write asynchronous code in the same way we write synchronous code?
Implementation diversion: micro-tasks
Before proceeding, there’s one implementation detail about promises that’s worth knowing.
The neat picture I presented in the previous post of happy tasks patiently waiting their turn in the task queue is complicated by micro-tasks (also known as jobs). These slippery little creatures are like extra high priority tasks.
The main difference between tasks and micro-tasks is how they are executed. If the currently executing task enqueues a new task, that new task won’t be executed until the next iteration of the event loop. Micro-tasks go in a separate queue that is processed after callbacks and between tasks. If the currently executing micro-task enqueues a new micro-task, that new micro-task will be executed in the same iteration of the event loop.
Promises and their handlers are executed as micro-tasks and so the interaction between promises and callbacks can be a little confusing. In this example, what do you think the output will be?
function example() {
console.log("1");
window.setTimeout(() => console.log("2"));
Promise.resolve().then(() => console.log("promise"));
console.log("3");
}
example();
We know that specifying no timeout value for window.setTimeout enqueues the callback so we would expect 1 and 3 to be printed before 2. The promise callback is also enqueued, but because promise handlers are micro-tasks, it wriggles in ahead of the timeout callback, despite being enqueued later:
1
3
promise
2
Treating micro-tasks in this way improves the responsiveness of the system. The whole point of promise handlers is that they are executed as soon as possible after the promise resolves, so it makes sense to give them priority. Sending them to the back of the task queue could cause an unacceptably long delay.
Aside from this implementation detail, though, promises aren’t really anything “special” – they’re just functions. It’s possible to write a (task-based) promise in just a few lines of code.
Async/await
So, after everyone rewrote their callback-based code to use promises, the JavaScript community decided that promises were unacceptable after all. Promise-based code was too hard to debug, they said. It was too easy to forget to add a .catch(), they said. It doesn’t fit well with the synchronous bits of my code, they said.
A new challenger arrived with ES2017: async/await. The idea behind this pair of keywords is to “tag” asynchronous functions with the async keyword. Under the hood, async functions actually return promises. We call async functions and use their return values as normal on the understanding that the return value may not yet have resolved into the desired value.
When we reach a point in the code where we absolutely do need the result, we use the await keyword to create a synchronisation point. If the promise has not yet resolved, the runtime suspends the task until the value is available. The upshot of this is that we can write asynchronous code that looks almost like synchronous code.
The only difference is that we must remember to await the return value when we want the resolved value:
async function handleRequest(req) {
const params = await validateRequest(req);
const dbResult = await fetchFromDb(params);
const response = await generateResponse(dbResult);
return response;
}
console.log(await handleRequest(req));
This asynchronous code looks almost the same as the synchronous equivalent. Error-handling works just the same using the try / catch we know and love.
What’s really interesting about async/await is that it is an abstraction built on promises, which are themselves an abstraction. Understand promises and you understand async/await. Since the keywords are syntactic sugar around promises, we can use them directly on promises too, thus unifying how we structure asynchronous JavaScript. Here’s an example:
async function slowFetchId() {
await new Promise(resolve => setTimeout(resolve, 5000));
return 5
}
We have a callback, a promise and async/await! slowFetchId is designed to emulate a slow, asynchronous operation such as retrieving data from a database. The use of await causes it to suspend until the promise resolves, which only occurs when the timeout triggers after 5000ms.
To see how async/await uses promises under the hood, we call slowFetchId and assign its return value to id. We log what it contains before and after we await the result:
async function example() {
var id = slowFetchId()
console.log("first id: ", id)
await id
console.log("second id: ", id)
}
// Output
> first id: Promise {<pending>}
// Five seconds later...
> second id: Promise {<resolved>: 5}
Since slowFetchId is async, it immediately returns a pending promise. It’s super critical to note that the returned promise represents the whole function slowFetchId. The async keyword converts our plain, boring function into an exciting promise.
Within that promise is the entire body of slowFetchId. When we want the return value from slowFetchId, we use await to suspend the current task execution until id has resolved into a value. slowFetchId is itself waiting on the timeout promise, so it can’t do anything until the runtime triggers the timeout callback, which enqueues a micro-task to resolve the promise and allows slowFetchId to return 5. The promise held in id can now resolve and execution moves on to the final logging statement.
Whenever you see await, you can mentally replace it with a promise:
var id = await slowFetchId();
console.log("id: ", id);
slowFetchId().then(id => console.log("id: ", id));
On the face of it, things seem to be pretty confusing when you have promises inside promises inside promises. In reality, you don’t need to worry about this because nested promises will unwrap themselves. If you want to, you can forget all about the promises. Async/await turns promises into a mere implementation detail underpinning a more powerful abstraction.
The main limitation of async/await is that any function calling an async function must itself be marked as async. Making just one function async might require a whole call stack of other functions to be marked as async too. You might see this referred to as having to “colour” a function after an influential blog post on the topic. Still, while a bit tedious, adding a few async and await keywords here and there is much less work that rewriting everything to use callbacks or promises. While not perfect, async/await is a big improvement.
Conclusion
There was a dark period when callback hell coexisted with various, mutually incompatible promise libraries. Best practices seemed to change from day to day. Thanks to the combination of built-in promises and async/await, things are now settling down. Using promises, we can express the concept of an asynchronous value and with async/await, we can express asynchronous operations in a clean and composable manner.
Armed with an understanding of the task queue, callbacks are little more than a thin wrapper around the execution model of the JavaScript runtime. Promises provide the crucial abstraction of asynchronous values, allowing us to reason about values that mutate over time instead of values that pop in and out of existence of different callback scopes. Async/await forms a new, syntactic abstraction on top of promises that allows us to structure asynchronous code more like synchronous code.
While it might all seem like a mess of conflicting approaches, each new development builds on the previous and studying the lower levels informs our understanding of the surface levels.