Skip to main content

Export metrics to AWS Kinesis Data Streams

Prerequisites

To use AWS Kinesis for metric collecting and processing, you should first install AWS SDK for C++. libcrypto, libssl, and libcurl are also required to compile Netdata with Kinesis support enabled. Next, Netdata should be re-installed from the source. The installer will detect that the required libraries are now available.

If the AWS SDK for C++ is being installed from source, it is useful to set -DBUILD_ONLY=kinesis. Otherwise, the build process could take a very long time. Note, that the default installation path for the libraries is /usr/local/lib64. Many Linux distributions don't include this path as the default one for a library search, so it is advisable to use the following options to cmake while building the AWS SDK:

sudo cmake -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_ONLY=kinesis <aws-sdk-cpp sources>

The -DCMAKE_INSTALL_PREFIX=/usr option also ensures that third party dependencies are installed in your system during the SDK build process.

Configuration

To enable data sending to the Kinesis service, run ./edit-config exporting.conf in the Netdata configuration directory and set the following options:

[kinesis:my_instance]
enabled = yes
destination = us-east-1

Set the destination option to an AWS region.

Set AWS credentials and stream name:

    # AWS credentials
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
# destination stream
stream name = your_stream_name

Alternatively, you can set AWS credentials for the netdata user using AWS SDK for C++ standard methods.

Netdata automatically computes a partition key for every record with the purpose to distribute records across available shards evenly.

Was this page helpful?

Need further help?

Search for an answer in our community forum.

Contribute