Log file analysis is critical in modern IT operations, development and security. Yet it’s difficult, manually intensive, and unlike many other cognitive aspects of working on software addressed by everything from IDEs to coding guides to APIs to libraries, the human searching, reading and inferential thinking required by log file analysis is difficult to automate. Perhaps most importantly, the information it produces is difficult for non-technical decision-makers to use or even appreciate.
From discussions with system administrators and data scientists in the field, we’ve heard a range of common issues arising in modern businesses facing the difficulty and the necessity of log file analysis. The most common of them that crystallize into two issues:
- Prospect-based risk: Management is reluctant to invest in log file analysis until the prospect of some exigent circumstance forces their hand — usually a site crash, data breach or IT crisis of some kind.
- Data accessibility for decision-makers: Business intelligence is fundamentally useless in the long run if it’s not made accessible and comprehensible to non-technical decision-makers.
The clearest examples of prospect-based risk assessment and decision-maker data accessibility occur in IT security.