Overview#
This guide describes the following:
System Requirements#
To run a CLP release, you’ll need:
Docker#
To check whether Docker is installed on your system, run:
docker version
If Docker isn’t installed, follow these instructions to install it.
NOTE:
If you’re not running as root, ensure Docker can be run without superuser privileges.
If you’re using Docker Desktop, ensure version 4.34 or higher is installed, and host networking is enabled.
Python#
To check whether Python is installed on your system, run:
python3 --version
CLP requires Python 3.8 or higher. If Python isn’t installed, or if the version isn’t high enough, install or upgrade it by following the instructions for your OS.
Choosing a flavor#
There are two flavors of CLP:
clp-json for compressing and searching JSON logs.
clp-text for compressing and searching unstructured text logs.
Note
Both flavors contain the same binaries but are configured with different values for the
package.storage_engine
key in the package’s config file (etc/clp-config.yml
).
clp-json#
The JSON flavor of CLP is appropriate for JSON logs, where each log event is an independent JSON object. For example:
{
"t": {
"$date": "2023-03-21T23:46:37.392"
},
"ctx": "conn11",
"msg": "Waiting for write concern."
}
{
"t": {
"$date": "2023-03-21T23:46:37.392"
},
"msg": "Set last op to system time"
}
The log file above contains two log events represented by two JSON objects printed one after the other. Whitespace is ignored, so the log events could also appear with no newlines and indentation.
If you’re using JSON logs, download and extract the clp-json
release from the
Releases page, then proceed to the clp-json quick-start guide.
clp-text#
The text flavor of CLP is appropriate for unstructured text logs, where each log event contains a timestamp and may span one or more lines.
Note
If your logs don’t contain timestamps or CLP can’t automatically parse the timestamps in your logs, it will treat each line as an independent log event.
For example:
2015-03-23T15:50:17.926Z INFO container_1 Transitioned from ALLOCATED to ACQUIRED
2015-03-23T15:50:17.927Z ERROR Scheduler: Error trying to assign container token
java.lang.IllegalArgumentException: java.net.UnknownHostException: i-e5d112ea
at org.apache.hadoop.security.buildTokenService(SecurityUtil.java:374)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Caused by: java.net.UnknownHostException: i-e5d112ea
... 17 more
The log file above contains two log events, both beginning with a timestamp. The first is a single line, while the second contains multiple lines.
If you’re using unstructured text logs, download and extract the clp-text
release from the
Releases page, then proceed to the clp-text quick-start guide.
Using CLP#
Once you’ve installed CLP’s requirements and downloaded a CLP release, proceed to the quick-start guide for your chosen flavor by clicking the corresponding link below.
CLP for JSON logs
How to compress and search JSON logs.
CLP for unstructured text logs
How to compress and search unstructured text logs.