If you’re shipping software, sooner or later you’ll want to know what’s happening inside it.
Logs shine light on the dark corners of an application, revealing smoke before production can burst into flames and lending credence to otherwise-unsubstantiated business claims. In fact, if a business process isn’t drawing insight from the log, it probably isn’t paying attention. Users use, hackers hack, leads are generated, bills run up, and systems chatter back and forth. From incident response to business intelligence, it’s all there in the log.
Yet these systems rarely get their due. Unless compliance dictates otherwise, logs tend to be touched only as a knee-jerk response to intrusion or brokenness in production. There’s no time like now to change that, starting with the two hard problems in logging (like computer science, there are only two):
- Collecting the right stuff
- Playing it back
First, “the right stuff.” Details that are useful for local debugging balloon when exposed to production volumes; unauthorized access is likely more important than validation errors; and so on. Striking the right balance between signal to noise is more art than science, but here are some resources to get started:
If you haven’t given much thought to logs before, Master Zen’s 10 commandments are a good place to start
Once you’re ready to dive deeper, check out Loggly’s ultimate guide to logging. These articles extend the high-level commandments with practical, hands-on recipes for many popular logging frameworks.
If you’re already logging up a storm, it may be time for a quick audit of everything going by. The OWasp project maintains a convenient cheat sheet of what should and shouldn’t turn up in the logs.
Next up, playback. Logfile legibility is enormously simplified by records written in machine-readable form at an appropriate level and with adequate context. But then we need somewhere to store them.
In containerized environments, providing a log collector like fluentd in a container sidecar can provide similar ease-of-use to application developers.
And the ELK stack combines a battle-tested datastore (Elasticsearch) with a flexible interface for querying them. While raw logs deter everyone but the hardened sysadmins, Kibana dashboards can make them available to the entire organization.
Finally, one article for the abstraction itself. Jay Kreps helped develop Apache Kafka and knows a thing or two about transaction logs. While this post isn’t strictly about application logging, it’s a great way of thinking about what an even more general use for the log.
Effective logging takes structure and discipline, but the gains in operational transparency, business intelligence, and auditability are a handsome reward for the effort. If you’re looking for a good logging platform to use with your application, shoot me a note! I can’t promise recommendations for every language or system, but I’m always happy to share answers where I have them.
And next month, if a tree falls in production, you’ll be able to tell. From the log, of course!
Ahem. Let’s regroup in a month to hear how it’s gone. And in the meantime I’ll just show myself out…