Structure in Unstructured Logs

There is growing talk about the need for structured logs. Proponents promote benefits such as ease of querying. Today, there are many sources of unstructured data that are a wealth of valuable information to network operations teams. In this blog we discuss some of the ways Augtera’s Network AI finds and uses structure, in unstructured logs. 

Unstructured Logging Challenges 

In the canonical programming meme, the first thing programmers do is output an unstructured message “Hello World”. As their journey continues, they often lean on logging, whether to a console/terminal window or a logging subsystem. Sometimes for debugging, sometimes to record an error, and other times as an audit trail. While there is debate about when and how logging should be used, the reality is, it is used often by programmers across all areas of IT. 

The benefit of the humble “print” command is it creates human readable messages, and this is the ultimate power – humans can read the message and often quickly get a sense of what is going on, or at least where to start exploring what is going on. However, machines, not commonly having the same language skills as humans, find unstructured logs more difficult to parse. Many believe that unstructured logs present so many problems that there needs to be shift to structured logging. The argument is that structured logging makes querying, indexing, and integration into standard visualization tools easier. A quick Google Search on “structured logs” delivers many results. Vendors talking about their structured logging capabilities, Google itself providing JSON structured logging in Cloud Logging, structured logging in Kubernetes, and more. 

Rare log messages and Metric extraction are two examples of finding structure in unstructured logs

While there are many benefits of structured data, unstructured logging is pervasive today. In networking, Syslog is a common example. Equipment software often writes to Syslog. Messages for similar events differ from equipment supplier to equipment supplier, and even programmers within the same equipment supplier, thus complicating Syslog analysis by machines. 

Extracting Metrics from Unstructured Logs 

Augtera Network AI comes with a sophisticated metric analysis and visualization capability. At the heart of our metric analysis is purpose-built networking algorithms that differentiate our anomaly detection from threshold-only systems. We detect complete failures, and gray failures, without the false positives & false negatives of other approaches. As a result, significant value can be realized by customers when metrics embedded in log messages are transformed to metrics and routed to metric processing. 

There are numerous solutions on the market that define rules for transforming log messages to metrics, so this blog will not spend much time discussing rules-based approaches. However, Augtera’s Network AI has this capability. 

Natural Language Processing of Unstructured Log Messages 

On April 27th, Augtera announced Zero Day Anomalies for Syslog. The important elements of that announcement were: 

  • Real-Time natural language processing (NLP) for billions of log messages per day, without queuing or dropping messages 
  • The ability to detect rare / not seen before log messages and flag them as potential anomalies, the first time they are seen 

The ability to detect truly new messages is not based on simple text matching, because the same message can vary in nuanced ways. Detection is based on understanding semantics in a message and knowing whether the semantics are new. This capability was developed because network failures are sometimes preceded by Syslog messages that have never been seen before. In addition, many long-standing anomalies simply go unnoticed. 

This approach is essentially creating, or rather revealing, structure that is typically not understood by machines who just see log messages as a sequence of ASCII codes. Sure, it is not the same kind of structure as a standard with a well-defined and adhered to fields, but it is structure that has already led to one compelling capability, detection of rare/new messages, and will have other applications in the future. It is seeing structure without defining and maintaining rules. 

Conclusion 

There are many approaches to finding structure in unstructured logs. Augtera’s Network AI supports rules-based approaches and NLP-based approaches. The former can have great precision but comes with the overhead of maintenance. The latter requires less maintenance and is yielding new capabilities that rules-based systems do not, for example, the ability to identify rare/never seen before messages. In a world where unstructured logs are still pervasive. Both approaches are being used by Augtera customers.  

Related