Using object storage#

CLP can:

  • compress logs from object storage (e.g., S3);

  • store archives on object storage; and

  • cache stream files (used for viewing compressed logs) on object storage.

This guide explains how to configure and use CLP for all three use cases. Note that you can choose to use object storage for any combination of the three use cases (e.g., compress logs from S3 and cache the stream files on S3, but store archives on the local filesystem).

Note

Currently, only the clp-json release supports object storage. Support for clp-text will be added in a future release.

Note

Currently, CLP only supports using S3 as object storage. Support for other object storage services will be added in a future release.

Prerequisites#

  1. This guide assumes you’re able to configure, start, stop, and use a CLP cluster as described in the quick-start guide.

  2. An S3 bucket and key prefix containing the logs you wish to compress.

  3. An S3 bucket and key prefix where you wish to store compressed archives.

  4. An S3 bucket and key prefix where you wish to cache stream files.

  5. A supported AWS authentication method configured with the necessary permissions to access the S3 buckets and prefixes mentioned above.

    Note

    You may use a single authentication method for all the use cases above, or a separate one for each.

Supported AWS authentication methods#

clp-json currently supports the AWS authentication methods described below.

Caution

Short-term STS credentials (which include a Session Token) are not supported directly. Instead, use named profiles (with IAM Identity Center authentication or IAM role assumption) which provide the required permissions and don’t require specifying credentials directly.

Long-term IAM user credentials#

clp-json can authenticate using long-term credentials for an IAM user.

  • To create a user, follow this guide.

    • You don’t need to assign any groups or policies to the user at this stage since we will attach policies in later steps, depending on which object storage use cases you require.

  • To generate the credentials, follow this guide.

    • Choose the “Other” use case to generate long-term credentials.

Named profiles#

clp-json can authenticate using AWS CLI named profiles. Named profiles can themselves make use of a variety of AWS authentication mechanisms, including:

Follow this guide for more information on configuring profiles with the AWS CLI.

Note

Profile configurations are stored in your AWS config directory (typically ~/.aws).

Environment variables for long-term credentials#

clp-json can authenticate using long-term IAM user credentials specified through the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

EC2 instance IAM roles#

clp-json can authenticate using IAM roles attached to an EC2 instance (that CLP is hosted on).

Follow this guide to attach an IAM role to an instance.

Configuration#

The subsections below explain how to configure your object storage bucket and CLP for each use case:

Configuring object storage

Configuring your object storage bucket and IAM permissions for each use case.

object-storage-config

Configuring CLP

Configuring CLP to use object storage for each use case.

clp-config

Using CLP with object storage#

The subsection below explains how to use CLP with object storage for each use case:

Using CLP with object storage

Using CLP to compress, search, and view log files from object storage.

clp-usage