Overview

The following is an example configuration which creates a new Kinesis Firehose in an AWS account, set up to pipe logs to an S3 bucket.

You will likely want to make some changes to the below, such as:

• Naming your S3 bucket to include some reference to Sym
• Using environment and tag values that are appropriate to your organization and use case.

📘

Prerequisites

1. An environment.tf file generated by symflow init
   a. If you have not run symflow init, please follow the instructions in Installing Sym
2. A runtime_connector module defined in connectors.tf
   a. If you do not have a connectors.tf, please follow the instructions in AWS Runtime Setup
3. A kinesis_firehose_access module defined in connectors.tf during the tutorial on the main AWS Kinesis Firehose

Declare the Kinesis Firehose Connector

The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.

Create a firehose.tf file with the following contents:

module "kinesis_firehose_connector" {
  source      = "symopsio/kinesis-firehose-connector/aws"
  version     = ">= 3.0.0"
  environment = local.environment_name
}

Create a Delivery Stream

Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to extended_s3. This will declare a Kinesis Firehose that is connected to an S3 bucket.

🚧

SymEnv is a required tag!

Your Kinesis Firehose Delivery Stream must have a tag SymEnv that matches the environment input of your kinesis_firehose_access module!

resource "aws_kinesis_firehose_delivery_stream" "sym_logs" {
  name        = "SymS3ReportingLogsMain"
  destination = "extended_s3"

  extended_s3_configuration {
    # The IAM Role and S3 Bucket are declared by the kinesis-firehose-connector module
    role_arn   = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
  }

  tags = {
    # This SymEnv tag is required and MUST match the SymEnv specified in your kinesis_firehose_access module
    SymEnv = local.environment_name
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_firehose.

• integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to module.runtime_connector.sym_integration.id, which has the permissions created by the kinesis_firehose_access module.
• stream_name: The name of the Kinesis Firehose Delivery Stream

resource "sym_log_destination" "s3_firehose" {
  type = "kinesis_firehose"

  # The Runtime Connector sym_integration has Kinesis Firehose permissions defined by the kinesis_firehose_access module
  integration_id = module.runtime_connector.sym_integration.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.sym_logs.name
  }
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

# ... other resources omitted

resource "sym_environment" "this" {
  # ... other attributes omitted
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.s3_firehose.id]

  # ... other attributes omitted
}

Full Example

You can find the complete code for this example in our S3 Log Destination Example.