Overview#

This guide describes the following:

CLP’s system requirements
How to choose a CLP flavor
How to use CLP.

System Requirements#

To run a CLP release, you’ll need:

Docker
Python

Docker#

To check whether Docker is installed on your system, run:

docker version

If Docker isn’t installed, follow these instructions to install it.

NOTE:

If you’re not running as root, ensure Docker can be run without superuser privileges.
If you’re using Docker Desktop, ensure version 4.34 or higher is installed, and host networking is enabled.

Python#

To check whether Python is installed on your system, run:

python3 --version

CLP requires Python 3.8 or higher. If Python isn’t installed, or if the version isn’t high enough, install or upgrade it by following the instructions for your OS.

Choosing a flavor#

There are two flavors of CLP:

clp-json for compressing and searching JSON logs.
clp-text for compressing and searching unstructured text logs.

Note

Both flavors contain the same binaries but are configured with different values for the package.storage_engine key in the package’s config file (etc/clp-config.yml).

clp-json#

The JSON flavor of CLP is appropriate for JSON logs, where each log event is an independent JSON object. For example:

{
  "t": {
    "$date": "2023-03-21T23:46:37.392"
  },
  "ctx": "conn11",
  "msg": "Waiting for write concern."
}
{
  "t": {
    "$date": "2023-03-21T23:46:37.392"
  },
  "msg": "Set last op to system time"
}

The log file above contains two log events represented by two JSON objects printed one after the other. Whitespace is ignored, so the log events could also appear with no newlines and indentation.

If you’re using JSON logs, download and extract the clp-json release from the Releases page, then proceed to the clp-json quick-start guide.

clp-text#

The text flavor of CLP is appropriate for unstructured text logs, where each log event contains a timestamp and may span one or more lines.

Note

If your logs don’t contain timestamps or CLP can’t automatically parse the timestamps in your logs, it will treat each line as an independent log event.

For example:

2015-03-23T15:50:17.926Z INFO container_1 Transitioned from ALLOCATED to ACQUIRED
2015-03-23T15:50:17.927Z ERROR Scheduler: Error trying to assign container token
java.lang.IllegalArgumentException: java.net.UnknownHostException: i-e5d112ea
    at org.apache.hadoop.security.buildTokenService(SecurityUtil.java:374)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Caused by: java.net.UnknownHostException: i-e5d112ea
    ... 17 more

The log file above contains two log events, both beginning with a timestamp. The first is a single line, while the second contains multiple lines.

If you’re using unstructured text logs, download and extract the clp-text release from the Releases page, then proceed to the clp-text quick-start guide.

Using CLP#

Once you’ve installed CLP’s requirements and downloaded a CLP release, proceed to the quick-start guide for your chosen flavor by clicking the corresponding link below.

CLP for JSON logs

How to compress and search JSON logs.

clp-json

CLP for unstructured text logs

How to compress and search unstructured text logs.

clp-text