AWS S3#

This guide explains how to configure and use CLP with AWS S3.

Note

Currently, only clp-json supports object storage. Support for clp-text will be added in a future release.

Note

This guide focuses on AWS S3. For S3-compatible storage services (e.g., MinIO, Ceph), see the S3-compatible storage guide.

Tip

If you’re using object storage because the host(s) on which you’re running CLP are ephemeral, consider also using external databases for metadata storage (to ensure data persistence in case of host replacements).

Supported uses#

Table 1 shows the supported uses of AWS S3 in CLP.

Use case

Supported

Compress logs from AWS S3

Store archives on AWS S3

Cache stream files on AWS S3

Table 1: The supported uses of AWS S3 in CLP.

Note

You can choose to use AWS S3 for any combination of the three use cases above (e.g., compress logs from AWS S3 and cache the stream files on AWS S3, but store archives on the local filesystem).

Prerequisites#

  1. This guide assumes you’re able to configure, start, stop, and use CLP as described in the clp-json quick-start guide.

  2. Depending on which use cases you require:

    • An S3 bucket and key prefix containing the logs you wish to compress.

    • An S3 bucket and key prefix where you wish to store compressed archives.

    • An S3 bucket and key prefix where you wish to cache stream files.

  3. A supported AWS authentication method configured with the necessary permissions to access the S3 buckets and prefixes mentioned above.

    Note

    You may use a single authentication method for all the use cases above, or a separate one for each.

Supported AWS authentication methods#

clp-json currently supports the AWS authentication methods described below.

Caution

Short-term STS credentials (which include a Session Token) are not supported directly. Instead, use named profiles (with IAM Identity Center authentication or IAM role assumption) which provide the required permissions and don’t require specifying credentials directly.

Long-term IAM user credentials#

clp-json can authenticate using long-term credentials for an IAM user.

  • To create a user, follow this guide.

    • You don’t need to assign any groups or policies to the user at this stage since we will attach policies in later steps, depending on which object storage use cases you require.

  • To generate the credentials, follow this guide.

    • Choose the “Other” use case to generate long-term credentials.

Named profiles#

clp-json can authenticate using AWS CLI named profiles. Named profiles can themselves make use of a variety of AWS authentication mechanisms, including:

Follow this guide for more information on configuring profiles with the AWS CLI.

Note

Profile configurations are stored in your AWS config directory (typically ~/.aws).

Environment variables for long-term credentials#

clp-json can authenticate using long-term IAM user credentials specified through the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

EC2 instance IAM roles#

clp-json can authenticate using IAM roles attached to an EC2 instance (that CLP is hosted on).

Follow this guide to attach an IAM role to an instance.

Configuration#

The subsections below explain how to configure your AWS S3 bucket and CLP for each use case:

Configuring AWS S3

Configuring your AWS S3 bucket and IAM permissions for each use case.

aws-s3-config

Configuring CLP

Configuring CLP to use AWS S3 for each use case.

clp-config

Using CLP with AWS S3#

The subsection below explains how to use CLP with AWS S3 for each use case:

Using CLP with AWS S3

Using CLP to compress, search, and view log files from AWS S3.

using-clp-with-aws-s3