CloudLens – Intro, How to make use of it.

The most challenging problem developers today face is analyzing logs. Logs contain a mix of text and semi-structured meta-data such as timestamps and error numbers. Logs are often aggregated from multiple sources with under-specified, heterogeneous formats. Parsing techniques based on schemas or grammars are not practical.

So, have you spent sleepless nights looking through log files of cloud-native applications trying to find the root-cause of a software bug? Well, look no further, CloudLens is here. Introducing CloudLens for Swift and JavaScript!

CloudLens is a Swift and JavaScript library for processing machine-generated text such as log streams. It relies on pattern matching for efficient, targeted data extraction. Matches can augment the raw input with structured attributes and trigger actions. These actions can compute metrics, emit alerts, or build a model of the system of interest.

For example, the single line of CloudLens code shown below can detect error messages in a log stream, extract the error code for subsequent analyses, and count errors:

stream.process(onPattern: “error (?<error:Number>\\d+)”) { _ in errorCount += 1 }

What’s the use of CloudLens :

CloudLens understands and supports the incremental nature of log processing. Processing starts with the very first log entry and reacts to new input as soon as it becomes available. Log structures can be discovered and improved over time.

CloudLens allows encapsulation and reuse of scripts via the definition and invocation of lenses. It is designed to handle both finite and infinite streams as well as nested streams, making it possible for instance to analyze session logs embedded in server logs, including multi-pass analyses on finite logs and slices of infinite logs.

In simpler words, CloudLens is about reducing the pain and effort of analyzing machine-generated text such as log streams, be it for postmortem root cause analysis or on-the-fly anomaly detection.

Key features of CloudLens:

• Pattern matching with structure attributes and trigger actions
• Action chaining and stream element mutation
• Support for incremental log processing
• Support for finite and infinite streams

It is easy to try CloudLens online using IBM’s Swift Sandbox. Simply press Play to run the example. The code editor is fully functional but the sandbox cannot access the network, so testing is limited to the supplied log.txt file originally produced by Travis CI for Apache OpenWhisk.

CloudLens will shorten the time to identify application problems by automating log analysis. It brings the power and convenience of a modern programming language ecosystem to log analysis. It implements the stream processing engine and the core log processing capabilities of other tools in a compact kernel and offers unparalleled extensibility.

CloudLens is available both as a swift and javascript library and it can be deployed on macOS and Linux. Read this documentation to setup CloudLens for your project.

The CloudLens project is maintained by IBM. It is an open source project, you can contribute to this open source project if you’re interested in building, improving the development process. The CloudLens open source project is present on GitHub.