Trimming the Callback Tree

Node.js approaches asynchrony by passing around continuations–callbacks used to resume program flow after an asynchronous action is complete. Isaac Schlueter describes their pattern:

  1. Async functions take a single callback function as their final argument.
  2. That callback is called in the future with the first argument set to anError object in the case of failure, and any additional results as the subsequent arguments.

Consider a naive (but asynchronous!) function for copying a file:

function copyFile (src, dst, callback) {
  fs.readFile(src, function (err, result) {
    if (err) return console.error('Failed reading file', err);
    fs.writeFile(dest, result, function (err) {
      if (err) console.error('Failed copying file', err);
      return callback(err);
    });
  });
}

Even this simple function reveals a significant challenge of continuation-style passing: keeping the code growing down the page faster than nested callbacks move it to the right. Fortunately for us, there are several clever utilities that can help bring things back under control.

async

First up is async, a handy library that:

provides straight-forward, powerful functions for working with asynchronous JavaScript

The general pattern of the async library is to implement array utilities (map, each, etc.) that operate on a list of asynchronous methods. Once all operations are complete, a supplied “result” function can be used to resume program flow or handle errors that might have occurred. Each function described in the funcs array takes some number of arguments (usually 0) followed by a continuation callback that async provides. Following node convention, the continuation must be called with an Error (if any occurred) followed by any additional arguments the method wants to pass on.

The resultFunc gets called after all funcs have completed, or immediately after the first time an error is returned. Just like the continuations in each step, its arguments consist of an error (if any) followed by a representation of the preceding functions’ results.

Usage of async.series, for example, looks like:

async.series([function (callback) {
  callback(null, "bar!");
}], function (err, results) {
  if (err) { 
    console.error('something terrible has happened');
  }
  else {
    console.log(results[0]); // "bar!"
  }
});

One of the utilities in async is async.waterfall, which:

Runs an array of functions in series, each passing their results to the next in the array

This is useful when we need to forward some result through a list of callbacks—say, to tidy up the copyFile method:

function copyFile (src, dst, callback) {
  async.waterfall([
    function (callback) {
      fs.readFile(src, callback);
    },
    function (result, callback) {
      fs.writeFile(dst, result, callback);
    }
  ], function (err) {
        if (err) {
      console.error(’Failed copying file’, err);
    }
    return callback(err);
  })
}

This may not look like an improvement for legibility—in this case, it probably isn’t—but notice that async has quietly flattened out the callback chain. Instead of nesting deeper and deeper with each additional function call, each callback is now defined straight down the page. Whether we need to tie together two, three, or n asynchronous methods, we never need to nest deeper than in the copyFile implementation above.

But we’re not done yet! Did you notice the similarities between the definitions of the anonymous functions passed in to async.waterfall? We can take advantage of partial function application to simplify the code even further:

function copyFile (src, dst, callback) {
  async.waterfall([
    _.partial(fs.readFile, src),
    _.partial(fs.writeFile, dst)
  ], function (err) {
        if (err) {
      console.error(’Failed copying file’, err);
    }
    return callback(err);
  })
}

In return for a little bit of async overhead, we have reduced the total count of anonymous functions to…1. If you believe that less code is better code (you should) this is another nice upgrade for the code’s maintainability.

Promises

The error or results passed to a node continuation sum up the set of possible outcomes for an asynchronous method:

  1. It can fail and call its continuation callback with an error
  2. It can succeed and call its continuation callback with its result

Unsurprisingly, callback functions aren’t the only way to represent these outcomes. Another is in the idea of a promise, which, in the words of the Promises/A+ spec:

represents the eventual result of an asynchronous operation

Promises accomplish this by representing the result as one an object in one of three states: pending (operation has not been completed), fulfilled (operation has completed successfully) or rejected (operation failed). Access to this state is provided via the then method, whose arguments are (optional) handlers for the two possible outcomes:

promise.then(onFulfilled, onRejected)

Usage is best demonstrated by example. Using q‘s “_blank”>deferred objects to manage promise resolution, the original copyFile method can now be re-written to return a promise:

function copyFile (src, dst) {
  var deferred = Q.defer();
  fs.readFile(src, function (err, result) {
    if (err) {
      return deferred.reject(err);
    }
    fs.writeFile(dst, result, function (err) {
      if (err) {
        deferred.reject(err);
      }
      else {
        deferred.resolve();
      }
    });
  });
  return deferred.promise;
}

In this case, the complexity of the internal code has actually increased, but the outcome of the function is now represented by the returned promise—no callback required—. We need only provide handlers to the promise for the “fulfilled” (success) and “rejected” (error) cases to the promise’s then function:

var promise = copyFile('./path/from', './path/to');

promise.then(function () {
  console.log('file copied successfully!');
}, function (err) {
  console.error('err', err);
});

Making things even more convenient, promise.then itself returns a promise. This allows more complex actions to be composed from a sequence of promises:

var promise = copyFile('./path/from', './path/to')
  .then(function () {
    return copyFile('./path/from_another', './path/to_another');
  });

Q provides some convenient sugar for putting promises together; rather than writing each function explicitly, we could build the same promise by passing an array of promise-returning methods to Q.all:

var promise = Q.all([
  copyFile('./path/from', './path/to'),
  copyFile('./path/from_another', './path/to_another')
]);

Roundup

For better or for worse, the node approach to asynchronous programming will not be changing soon. Back in August, Isaac S.’s post to the nodejs community made it clear that:

Callbacks will remain the de facto way to implement asynchrony. Generators and Promises are interesting and will remain a userland option.

Further Reading

Hey, I'm RJ! For more learnings about software and management, find me @rjzaworski or sign up for my semi-regular newsletter.

Let’s keep in touch

Send me timely updates on software, product, and process.