Overview
The following is an example configuration which creates a new Kinesis Firehose in an AWS account, set up to pipe logs to an S3 bucket.
You will likely want to make some changes to the below, such as:
- Naming your S3 bucket to include some reference to Sym
- Using
environment
andtag
values that are appropriate to your organization and use case.
Prerequisites
- An
environment.tf
file generated bysymflow init
a. If you have not runsymflow init
, please follow the instructions in Installing Sym- A
runtime_connector
module defined inconnectors.tf
a. If you do not have aconnectors.tf
, please follow the instructions in AWS Runtime Setup- A
kinesis_firehose_access
module defined inconnectors.tf
during the tutorial on the main AWS Kinesis Firehose
Declare the Kinesis Firehose Connector
The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.
Create a firehose.tf
file with the following contents:
module "kinesis_firehose_connector" {
source = "symopsio/kinesis-firehose-connector/aws"
version = ">= 3.0.0"
environment = local.environment_name
}
Create a Delivery Stream
Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to extended_s3
. This will declare a Kinesis Firehose that is connected to an S3 bucket.
SymEnv is a required tag!
Your Kinesis Firehose Delivery Stream must have a tag
SymEnv
that matches the environment input of yourkinesis_firehose_access
module!
resource "aws_kinesis_firehose_delivery_stream" "sym_logs" {
name = "SymS3ReportingLogsMain"
destination = "extended_s3"
extended_s3_configuration {
# The IAM Role and S3 Bucket are declared by the kinesis-firehose-connector module
role_arn = module.kinesis_firehose_connector.firehose_role_arn
bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
}
tags = {
# This SymEnv tag is required and MUST match the SymEnv specified in your kinesis_firehose_access module
SymEnv = local.environment_name
}
}
Add a Log Destination
Define a sym_log_destination
resource with type = kinesis_firehose
.
integration_id
: The integration containing the permissions to push to Kinesis Firehose. This should be set tomodule.runtime_connector.sym_integration.id
, which has the permissions created by thekinesis_firehose_access
module.stream_name
: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "s3_firehose" {
type = "kinesis_firehose"
# The Runtime Connector sym_integration has Kinesis Firehose permissions defined by the kinesis_firehose_access module
integration_id = module.runtime_connector.sym_integration.id
settings = {
stream_name = aws_kinesis_firehose_delivery_stream.sym_logs.name
}
}
Add the Log Destination to your Environment
Each sym_environment
accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids
list.
# ... other resources omitted
resource "sym_environment" "this" {
# ... other attributes omitted
# Add your log destinations here
log_destination_ids = [sym_log_destination.s3_firehose.id]
# ... other attributes omitted
}
Full Example
You can find the complete code for this example in our S3 Log Destination Example.
Updated 4 months ago