S3 Bucket

Overview

One of the simplest implementations for reporting is to send your logs from Kinesis Firehose to an S3 bucket, where they'll be stored in date-ordered directories.

The following is an example configuration which creates a new Kinesis Firehose in an AWS account, set up to pipe logs to an S3 bucket.

You will likely want to make some changes to the below, such as:

  • Naming your S3 bucket to include some reference to Sym
  • Using environment and tag values that are appropriate to your organization and use case.

πŸ“˜

Make sure you've enabled the aws/kinesis-firehose add-on in your Runtime Connector, as described on the main AWS Kinesis Firehose page.

Declare the Kinesis Firehose Connector

The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.

module "kinesis_firehose_connector" {
  source  = "symopsio/kinesis-firehose-connector/aws"
  version = ">= 3.0.0"
  environment = "main"
}

Create a Delivery Stream

Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to extended_s3. This will declare a Kinesis Firehose that is connected to an S3 bucket.

🚧

SymEnv is a required tag!

Your Kinesis Firehose Delivery Stream must have a tag SymEnv that matches the environment specified in your runtime-connector module!

resource "aws_kinesis_firehose_delivery_stream" "sym_logs" {
  name        = "SymS3ReportingLogsMain"
  destination = "extended_s3"

  extended_s3_configuration {
    # The IAM Role and S3 Bucket are declared by the kinesis-firehose-connector module
    role_arn   = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
  }

  tags = {
    # This SymEnv tag is required and MUST match the `environment` in your `runtime-connector` module
    # because the aws/kinesis-firerhose add-on only grants access to Firehoses tagged with a matching SymEnv value
    SymEnv = "main"
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_firehose.

  • integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions created by the aws/kinesis-firehose add-on.
  • stream_name: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "s3_firehose" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions from the aws/kinesis-firehose add-on
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.sym_logs.name
  }
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.s3_firehose.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Full Example

You can find the complete code for this example in our S3 Log Destination Example.