3/1/2016

Command-line configuration with confab and yargs

confab is a great tool for configuring node.js applications, but it doesn’t ship with support for command-line flags. CLIs can vary quite a bit from one application to the next, and confab does not make assumptions (or force opinions) about their design.

Fortunately, there are plenty of other argument-processing tools that can be easily integrated with confab. One such is yargs, which (in addition to a pirate motif and a laundry list of other useful features) exposes command-line arguments as the yargs.argv object:

var argv = require('yargs').argv;
console.log(argv.foo);

$ node index.js --foo=baz
'baz'

Since argv is just an object, integrating with confab requires an almost-trivial use of the assign transform:

var confab = require('confab');
var argv = require('yargs').argv;
var pick = require('lodash.pick');

var CONFIG_DEFAULTS = {
  foo: 'bar'
};

var CONFIG_KEYS = Object.keys(CONFIG_DEFAULTS);

var config = confab([
  confab.defaults(CONFIG_DEFAULTS),
  confab.assign(pick(argv, CONFIG_KEYS)),
  confab.required(CONFIG_KEYS)
]);

process.stdout.write(JSON.stringify(config, null, 2) + '\n');

process.exit(0);

No surprises here.

$ node index.js
{ foo: 'bar' }

$ node index.js --foo=baz
{ foo: 'baz' }

Just like that, we’ve bolstered our configuration with command-line overrides. CLI support is missing no longer–we just needed to set it up!



2/5/2016

Evolution Over Revolution

The Porsche 356 (original: Valder137 / Wikimedia)

The Porsche Museum is hard to miss. Towering over Zuffenhausen in the north of Germany’s southern city of Stuttgart, its sharp angles evoke the speed and achievement of the collection within. Inside, exhibits twist chronologically upwards from the company’s founding to the present day, encouraging continuous comparison between each gleaming model and its descendants. The cars are the stars of the show, but it’s impossible to miss the Porsche tradition of “evolution over revolution” that glues the ensemble together.

In practice, the tradition is simple. Build the right platform and refine it to perfection. Reinvention, if it must happen at all, is a dish best saved for after every other choice is exhausted.


Porsche’s story begins in Nazi Germany, when Hitler ordered Austrian automotive engineer Ferdinand Porsche to develop a “car for the people.” 1 The second World War delayed mass-production, but Porsche’s design–the Volkswagen Beetle–would eventually become the most-manufactured car of all time. The ubiquitous Beetle is a far cry from the elite roadsters that would follow, but its aerodynamic curves and air-cooled, rear-mounted engine offered a first glimpse at things to come.

Wandering deeper into the exhibit, the Beetle elongates into the first true Porsche: 1948’s model 356. The body and chassis of Erwin Komenda’s design belong on a sports car, but beneath the long snout the Volkswagen lingers on. Headlights, air-cooled engine; with components hard to come by in the lean postbellum years, the Beetle and the 356 even shared many of the same parts.

After several iterations of the 356, the exhibits arrive at the 1963 Frankfurt Auto Show, where the 911 first rolled off of F.A. Porsche’s drawing board and onto the red carpet. In the 911, 15 years of technological progress had unlocked enabled a car that was bigger, faster, and comfortable than its predecessors. But for all its glamor the streamlined profile and the basic layout–rear-mounted, air-cooled engine–remained the same.

For three decades, the 911 was it. Components were tweaked and styling adjusted, but through oil embargoes and global recessions, changes in management and the evolution of the family business, the flagship coupe remained 2.

The Porsche 911 (original: Michael Barera / Wikimedia)

Success does not depend on revolution. Every 911 cruising Venice Beach shares the familiar features of every Volkswagen Beetle. Both lines have met with tremendous success, despite divergent markets and surprisingly little after a half-century on the road.

For a heavy manufacturer like Porsche, evolution may stem first from practical considerations. Changes in mass production are expensive. New components require new processes, tools, and training. Workers must be hired, supply chains managed–all to bring a single change to market. Set those obstacles in the devastated economy of post-war Europe, it’s no surprise that engineers would reuse Volkswagen parts rather than needing to build their own.

But if change are expensive, complete overhaul is far worse. Not only does the company face the capital cost of new development, but revolutions toss aside years of embodied refinement. When a system goes out with the dishwater, the solutions built into its design go with it. Revolution offers old problems a chance to resurface–albeit in a different guise–in addition to whatever new gremlins slip into the new design.

Designers and engineers love the green field: a blank canvas to fill with works all our own. Before we charge off to reshape the world in our own image, though, we need to be sure the numbers add up. Revolution sings the siren song of a world that is newer, better, and faster. Out with the old! In with the new! And often new is better. Often change does deliver. But change for change’s sake comes with no guarantee.


And so the 911 evolved. Technological improvements allowed the engineers to coax more power from ever-lighter engines. Performance increased, as–all things being relative–did fuel economy.

In 1997, decades of incremental improvements came to an end. That year’s 911 model, known internally as the 996, arrived on showroom floors with an all-new chassis, interior, and updated profile. Its reception was tepid at best. From new headlights derided as “fried eggs” by the German press to liquid coolant flowing in place of the traditional air-cooling in back, the 996 had something for every purist to loathe 3.

The Porsche 911 (996) (original: @crazylenny2 / Flickr)

It takes constant vigilance to keep product development focused on the “necessary” work. Designers love designing. Engineers love engineering. Both love novelty, and novelty, that fickle beast, is happy to blur the finer points distinguishing “different” from “better.” Our nature is to build, evolve, and grow–preferably on the biggest stage we can find.

For the Porsche of 1946, change wasn’t a question of “should”, but a question of “can.” Given a viable alternative, the early 356s would not have reused parts developed for the lowly Volkswagen. But in the dark days after the war, there was no alternative. Deliver the 356, reused parts and all, or don’t deliver anything.

Through the 70s and 80s, flush with cash and talent, Porsche could afford to change. It simply didn’t. Bucking the motor industry’s traditional cycles of death and rebirth, the 911 platform of 1963 changed remarkably little over the years that followed. Not that its the designers and engineers sat idle: each new 911 was lighter, faster, more comfortable, and more reliable than the model that came before. Freed from worrying about redesigning the platform, developers could focus instead on perfecting the driving experience. The foundation wasn’t broken, so no time was wasted trying to fix it.

Radical new designs can be exciting, but customers rarely share designers’ enthusiasm when familiar, long-serving products are reimagined into something new. Redesign comes with good intentions, but even the best have their price. Old habits must be unlearned and new habits must be formed. Existing customers are understandably resistant.

The bigger the change, the more time needed to adapt. Mechanics may be the only ones impacted by a more efficient fuel pump, but when the cockpit is redesigned, everyone knows. Even changes for the better–to drastically reduce a new driver’s learning curve, say–may make less sense after accounting for existing behavior. Revolution means gut-wrenching change.

This is not meant to paint revolution as impossible: only to encourage restraint. Hold it in reserve until those desperate circumstance when there is no viable alternative.


After decades of success with the 911, why did Porsche change? And why the 996?

The 996 is our story, but its setting is incomplete. For Porsche, the early 90s were defined by declining sales and an increasingly outdated public image. In the words of the 996’s designer, Pinky Lai, “tears were flowing.” A new car–the Boxster–was envisioned to boost flagging sales by providing a low-cost alternative to the 911. To keep costs down the two both models would share design and production work, just as the 356 and the Beetle a half-century before. For Porsche, the development of the Boxster meant the difference between independence and a likely takeover bid.

For the 911, there was one more reason to change. After 30 years with minimal alteration, the platform had simply reached its limits. Tighter environmental regulations and the constant drive to improve performance made demands the traditional air-cooled engine couldn’t meet. Times had changed. The car hadn’t.

And so change it did.

The Porsche 911 (997) (original: Valder137 / Wikimedia)

Epilogue

Today’s 911 is a different car than the 911 of 1963, but every detail–from headlights to rear-mounted engine–hearkens back to the models that came before. There’s even a remaining touch of Beetle, if only to show how little the platform’s concept has changed since its conception in 1931.

If there’s a take-home lesson, it’s this: revolution has its place. When all other paths are exhausted, when the list of viable alternatives has run dry–these are the times to throw out refinement and chart a new course. But revolution is exhausting, and expensive to boot. As long as the alternatives remain, we’re better off finding–and evolving–the solid ground underfoot.


Since publishing this article first here and later, on medium, two excellent pieces about the product side of change have come to my attention. They’re Eevee’s “We have always been at war with UI” and Steven Sinofsky’s “Why the heck can’t we change our product?”, and they’re well worth your time!



9/4/2015

Confab: Simple Node.js Configurations

Node.js service configurations run the gamut from sophisticated libraries to ad-hoc lookups of environment variables or arbitrary JSON on the file system. Somewhere in the middle lies confab, a tiny utility for building configurations that are simple, external, and utterly predictable.

$ npm install confab

The core concept is dead simple: a single configuration object is constructed, extended, and validated using a sequence of transformations. With no transformations supplied, the configuration is simply an empty object:

// config.js
module.exports = confab([]); // {}

That isn’t so useful, but confab also ships with a few built-in transformations that can:

  • load configuration data from JSON
  • merge environment variables
  • specify defaults and required fields
  • assign programmatic overrides
  • freeze the results against futher modification.

To set a default port and allow the environment to override it, we can extend our empty configuration with a few transformations:

var config = confab([
  confab.loadEnvironment({ 'PORT': 'port' }),
  confab.defaults({ port: 3000 }),
  confab.freeze()
]); // { "port": 3000 }

Custom Transformations

We can also easily define new tranformations; if we want all applications to report certain configuration values at startup, for instance, we can whitelist and log the interesting parts:

function logConfigValue (whiteList) {
  return function (config) {
    var output = {};
    whitelist.forEach(function (key) {
      if (config.hasOwnProperty(key)) {
        output[key] = config[key];
      }
    });
    console.log(output);
  };
}

The custom transformation can now be used just like the built-ins:

var config = confab([
  confab.loadEnvironment({ 'PORT': 'port' }),
  confab.defaults({ port: 3000 }),
  confab.freeze()
  logConfigValue('port') // log { "port": "3000" }
]);

Portability

Besides encouraging simple, declarative configuration, confab also provides a platform for applying consistent rules across multiple projects. Once our transformation list is settled, we can package the configuration logic up and publish it to a (private) package repository. Maybe we want to require all projects to respect JSON config files stored in a “global” directory:

module.exports = function myConfig (projectName) {

  var env = process.env('NODE_ENV') || 'default';
  var homeDir = process.env('USERPROFILE') || process.env('HOME');

  return confab([
    confab.loadJSON([
      path.resolve(__dirname, 'config.json'),
      path.resolve(homeDir, '.org-config', projectName + '.' + env + '.json'
    ]),
    confab.loadEnvironment({ 'PORT': 'port' }),
    confab.defaults({ port: 3000 }),
    confab.freeze(),
    logConfigValue(['port'])
  ]);
};

To reuse this configuration, we can simply install it:

$ npm install my-config

…and pass it project-specific options as needed.

// app-config.js
module.exports = require('my-config')('my-app');

Conclusion

That’s about it: simple, reusable configuration logic. It’s been invaluable to me for simplifying configuration management and runtime visibility across projects, scheduled tasks, and services, and now it’s open-sourced for your development convenience.

Happy Hacking!



But wait! There's more—

View all posts