clp-json quick-start#
This page will walk you through how to start CLP and use it to compress and search JSON logs.
Caution
If you’re using a clp-json
release, you can only compress and search JSON logs. This limitation
will be addressed in a future version of CLP.
Starting CLP#
To start CLP, run:
sbin/start-clp.sh
Note
If CLP fails to start (e.g., due to a port conflict), try adjusting the settings in
etc/clp-config.yml
and then run the start command again.
Compressing JSON logs#
To compress some JSON logs, run:
sbin/compress.sh --timestamp-key '<timestamp-key>' <path1> [<path2> ...]
<timestamp-key>
is the field path of the kv-pair that contains the timestamp in each log event.E.g., if your log events look like
{"timestamp": {"iso8601": "2024-01-01 00:01:02.345", ...}}
, you should entertimestamp.iso8601
as the timestamp key.
Caution
Log events without the specified timestamp key will not have an assigned timestamp. Currently, these events can only be searched from the command line (when you don’t specify a timestamp filter).
<path...>
are paths to JSON log files or directories containing such files.Each JSON log file should contain each log event as a separate JSON object, i.e., not as an array.
The compression script will output the compression ratio of each dataset you compress, or you can use the UI to view overall statistics.
Compressed logs will be stored in the directory specified by the archive_output.storage.directory
config option in etc/clp-config.yml
(archive_output.storage.directory
defaults to
var/data/archives
).
Tip
To compress logs from object storage, see Using object storage.
Sample logs#
For some sample logs, check out the open-source datasets.
Searching JSON logs#
You can search your compressed logs from CLP’s UI or the command line.
In clp-json, queries are written as a set of conditions (predicates) on key-value pairs (kv-pairs). For example, Figure 1 shows a query that matches the first log event in Figure 2.
ctx: "conn11" AND msg: "*write concern*"
{
"t": {
"$date": "2023-03-21T23:46:37.392"
},
"ctx": "conn11",
"msg": "Waiting for write concern."
}
{
"t": {
"$date": "2023-03-21T23:46:37.392"
},
"msg": "Set last op to system time"
}
The query in Figure 1 will match log events that contain the kv-pair "ctx": "conn11"
as well as a kv-pair with key "msg"
and a value that matches the wildcard query
"*write concern*"
.
A complete reference for clp-json’s query syntax is available on the syntax reference page.
Searching from the UI#
To search your compressed logs from CLP’s UI, open http://localhost:4000 in
your browser (if you changed webui.host
or webui.port
in etc/clp-config.yml
, use the new
values).
Figure 3 shows the search page after running a query.

The numbered circles in Figure 3 correspond to the following elements:
The query input box. The format of your query should conform to CLP’s JSON search syntax.
The query case-sensitivity toggle. When turned on, CLP will search for log events that match the case of your query.
The time range selector. CLP will search for log events that are in the specified time range. You can select a preset filter (e.g.,
Last 15 minutes
;Yesterday
) from the dropdown, or chooseCustom
and set the start time and end time directly.The dataset selector. CLP will search for log events that belong to the selected dataset.
The search results timeline. After a query, the timeline will show the number of results across the time range of your query.
You can click and drag to zoom into a time range.
When you mouse over a bar in the timeline, a popup will display the range and the number of search results in that range.
The search results.
You can sort by
Ascending
orDescending
timestamp by clicking theTimestamp
header in the table.Each search result includes a link to the original file in which the log event was found.
Note
By default, the UI will only return 1,000 of the latest search results. To perform searches which return more results, use the command line.
Searching from the command line#
To search your compressed logs from the command line, run:
sbin/search.sh '<query>'
To narrow your search to a specific time range:
Add
--begin-time <epoch-timestamp-millis>
to filter for log events after a certain time.<epoch-timestamp-millis>
is the timestamp as milliseconds since the UNIX epoch.
Add
--end-time <epoch-timestamp-millis>
to filter for log events before a certain time.
To perform case-insensitive searches, add the --ignore-case
flag.
Caution
To match the convention of other tools, by default, searches are case-insensitive in the UI and searches are case-sensitive on the command line.
Stopping CLP#
If you need to stop CLP, run:
sbin/stop-clp.sh