Purpose-Built for Network Operations
LogAI was developed to address Network Operations uses cases. To process in real-time, streaming log data, from any log, without latency or message drops. Specifically, to address the challenges of producing actionable insights from unstructured text data at such high volumes, where noise levels are so high, and the unknown unknowns are prevalent and undetected.
Most log tools collect, store, and search /query as many log messages, from as many sources, as possible. This historical record of log messages is good for compliance, and historical searches for incident context and/or root cause. However, these tools are not what is needed by operations teams to detect and take real-time action on existing and emerging incidents.
LogAI changes the log experience through high-performance, high-efficiency real-time AI/ML processing of streaming log messages from Syslog, or other non-standard format Logs such as Cloud Logs encoded as JSON and ingested using Kafka or Augtera APIs. The result is actionable and automated AI “needles” from the Log data haystack.
Network operations teams today either have no ability to process Logs in real-time or can only create limited rules to identify a set of known Logsignatures. This leaves several gaps:
- Anomaly symptoms that have never been seen before i.e., the unknown unknowns
- Anomalies that are never acted on because they are lost in the noise
- Immediate implementation of new known Log signatures due to cumbersome rules-based approaches and/or lag time for internal software development and/or other inter-organization / process issues
- Extraction of metrics for anomaly detection & visualization
- Message rate-based anomaly detection with high fidelity and low noise
- Elimination of noise prior to signaling consoles, trouble ticket systems, and automation systems
Augtera Networks LogAI is a vastly different approach.
LogAI has flexible Ingestion
Syslog has long been the standard of network equipment vendors, however with the rise of cloud-based systems and message streaming technologies new logs formats, such as encoding logs in JSON are becoming more common. However, logs are generated by Cloud based systems as well as equipment vendors in many other non-standard formats, typically encoded as JSON. Further logs are collected, normalized, and distributed by operations teams in many other formats, typically also encoded as JSON. Kafka is an emerging message bus for many types of data within Network Operations environments.
LogAI supports all these scenarios today and at its core is agnostic to the format of the log messages when they are ingested, as all logs are normalized to a common internal format.
LogAI Lets You See the Unknown Unknowns
Log messages across the industry are increasing in volume, velocity, and variety. No two equipment vendors generate the same log messages and log messages change and evolve even within the same equipment vendor, over time. When the symptoms of emerging incidents arise, they are easily missed. In addition, many persistent issues remain, lost in the noise.
Zero Day Anomalies is a Natural Language Processing (NLP) based capability that detects new, rare, and unique log messages the first time they appear. So, if they are the symptom of a current or future incident, action can be taken immediately. The capability focuses on rare messages that have not been seen within a customer-defined period.
Determining what is a new or rare log message is not trivial though. There are so many nuances in log messages that simple text processing will lead to false results. To realize this capability, LogAI uses a high-performance, high-efficiency Natural Language Processing implementation, purpose-built by Augtera Networks. This is needed because semantic understanding is necessary to determine what is a new or rare log message.
This industry-first capability is the result of purpose-built Natural Language Processing (NLP) technology, illustrated in the image above. For more details request that an Augtera Networks engineer contact you.
LogAI Lets You See a Larger Spectrum of Known Knowns
There will always be a place for high-performance detection of known Log signatures that are operationally relevant or symptoms of incidents. LogAI “classifiers” are for this purpose. LogAI classifiers are regex-based rules that can be applied at run-time on streaming Logs at scale. However, the number of classifiers each individual network operations team can determine and create is smaller than what a whole community can. Augtera provides collective learning and classifiers from the entire customer base. The full collection of classifiers is made available to everyone in the customer base. This accelerates the detection of “known knowns” for all Augtera customers.
In response to the detection of log signatures through “classifiers”, the customer can specify numerous actions, including:
- Sending the message to the Network AI/ML pipeline
- Formatting some aspect of the log message – for example, changing the severity, adding meta data, adding IP information
- Publishing the message to a Kafka topic
LogAI Lets You See Log Message Rate Changes
A meaningful up or down shift in the rate of even a known log message is often the symptom of a problem. However, algorithms designed to simply track any abnormal rate are noisy. Augtera Networks developed a purpose-built algorithm for identifying meaningful bursts in messages.
In the graphic above, BGP state changes are detected. Normally, generating an alarm for every BGP state change would be noisy. Here though, Augtera Network AI detects a significant change in rate, indicating a BGP issue in the network.
Metric Extraction from Logs
Log messages often contain metrics. However, embedded in an unstructured log message, these metrics are hard to extract, analyze and visualize with network operations tools.
Augtera Networks enables the extraction of metrics from log messages. Metrics can then be used by the rest of the Augtera Network AI platform for anomaly detection, gray failure detection, incident root identification, and notification (collaboration, trouble ticketing, automation systems,…). Metric extraction can be used for any log message, regardless of where in the network the message pertains to. The extraction process is not specific to a set of messages.
The above graphic is an example of metric extraction and the resultant visualization. On this occasion, metrics have been extracted from syslog messages like the below:
Hostname=“AutoRegressionXXX.YYY.ZZZ.com“,SlotId=“0“,errdefs_msgno=”999“,BIGIQ_templateResourceName=“ltm:virtual:aaa“,appName=“bbb“,Entity=“TCP“,Module=“tcp“,AggrInterval=“30“,EOCTimestamp=“999“,HitCount=“1“,ApplicationName=“dosHttpApp2“,RequestSide=“Client-side“,VSName=“Common/dosHttpApp2/dosHttpApp2vs1“,RemoteHostIP=“m.n.o.p“,SubnetName=“a.b.c.d“,NextHop=“aa:bb:cc:dd“,UserKey=“Aggregated“,SubscriberName=“Aggregated\“,SubscriberIdType=“Unknown“,TowerName=“Aggregated“,RadioType=“N/A“,Continent=“-“,CountryRegion=“Aggregated“,CityPostcode=“Aggregated“,RTTCount=“120“,RTTSum=“186“,RTTMax=“2“,RTTMin=“1“,RTTVarSum=“0“,PktsRecv=“118“,PktsLost=“0“,PktsSent=“118“,GoodPutSnd=“51795“,GoodPutRcv=”24481…
One of Many Correlated Data Types
Whether metric extraction is used or not, log messages can be correlated with other data types / sources. To Augtera’s Network AI multi-layer, topology-aware correlation technology, log messages are just another data source, normalized, and used for actionable insights like any other data.
LogAI Search
LogAI is like typical log tools that are designed as a historical search capability for the many exabytes of data stored. However, for the customer-defined period data is stored, LogAI search has some compelling capabilities.
After automated normalization and enrichment, structured search can be used, including searching by tags. Traditional regex searches can also be done.
Conclusion
Logs contain a wealth of information that operators increasingly want integrated into real-time workflows. In addition, Network Operations teams can no longer be simply reactive to incidents, they must proactively detect, see potential incidents before they occur, and prevent those incidents from ever happening.
The Augtera Network AI platform, including LogAI, was developed to not only enable network operations teams to react faster, but to reduce the need to react by eliminating noise and preventing future incidents.
To request more information on LogAI or to request a demonstration:
Related Links
- LogAI Press Release
- LogAI Frequently Asked Questions (FAQ)
- Tech Field Day/Networking Field Day 28 video presentation: Augtera Real-Time
