Overview

One of the simplest implementations for reporting is to send your logs from Kinesis Firehose to an S3 bucket, where they'll be stored in date-ordered directories.

The following is an example configuration which creates a new Kinesis Firehose in an AWS account, set up to pipe logs to an S3 bucket.

You will likely want to make some changes to the below, such as:

• Naming your S3 bucket to include some reference to Sym
• Using environment and tag values that are appropriate to your organization and use case.

📘

Prerequisites

1. An environment.tf file generated by symflow init
   a. If you have not run symflow init, please follow the instructions in Installing Sym
2. A Runtime Connector Role defined in runtime.tf
   a. If you do not have a runtime.tf, please follow the instructions in Connecting Sym to AWS
3. A firehose.tf file created during the tutorial on the main AWS Kinesis Firehose

Declare the Kinesis Firehose Connector

The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.

Add this module to your firehose.tf file:

module "kinesis_firehose_connector" {
  source      = "symopsio/kinesis-firehose-connector/aws"
  version     = ">= 3.0.0"
  environment = local.environment_name
}

Create a Delivery Stream

Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to extended_s3. This will declare a Kinesis Firehose that is connected to an S3 bucket.

🚧

SymEnv is a required tag!

Your Kinesis Firehose Delivery Stream must have a tag SymEnv that matches the environment specified in the aws_iam_policy.aws_kinesis_firehose policy you defined!

resource "aws_kinesis_firehose_delivery_stream" "sym_logs" {
  name        = "SymS3ReportingLogsMain"
  destination = "extended_s3"

  extended_s3_configuration {
    # The IAM Role and S3 Bucket are declared by the kinesis-firehose-connector module
    role_arn   = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
  }

  tags = {
    # This SymEnv tag is required and MUST match the SymEnv specified in the aws_iam_policy.aws_kinesis_firehose policy
    SymEnv = local.environment_name
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_firehose.

• integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions created by the aws/kinesis-firehose add-on.
• stream_name: The name of the Kinesis Firehose Delivery Stream

resource "sym_log_destination" "s3_firehose" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions defined by aws_iam_policy.aws_kinesis_firehose
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.sym_logs.name
  }
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

# ... other resources omitted

resource "sym_environment" "this" {
  # ... other attributes omitted
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.s3_firehose.id]

  # ... other attributes omitted
}

Full Example

You can find the complete code for this example in our S3 Log Destination Example.