S3 Bucket
Overview
One of the simplest implementations for reporting is to send your logs from Kinesis Firehose to an S3 bucket, where they'll be stored in date-ordered directories.
The following is an example configuration which creates a new Kinesis Firehose in an AWS account, set up to pipe logs to an S3 bucket.
You will likely want to make some changes to the below, such as:
- Naming your S3 bucket to include some reference to Sym
- Using
environment
andtag
values that are appropriate to your organization and use case.
Prerequisites
- An
environment.tf
file generated bysymflow init
a. If you have not runsymflow init
, please follow the instructions in Installing Sym- A Runtime Connector Role defined in
runtime.tf
a. If you do not have aruntime.tf
, please follow the instructions in Connecting Sym to AWS- A
firehose.tf
file created during the tutorial on the main AWS Kinesis Firehose
Declare the Kinesis Firehose Connector
The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.
Add this module to your firehose.tf
file:
module "kinesis_firehose_connector" {
source = "symopsio/kinesis-firehose-connector/aws"
version = ">= 3.0.0"
environment = local.environment_name
}
Create a Delivery Stream
Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to extended_s3
. This will declare a Kinesis Firehose that is connected to an S3 bucket.
SymEnv is a required tag!
Your Kinesis Firehose Delivery Stream must have a tag
SymEnv
that matches the environment specified in theaws_iam_policy.aws_kinesis_firehose
policy you defined!
resource "aws_kinesis_firehose_delivery_stream" "sym_logs" {
name = "SymS3ReportingLogsMain"
destination = "extended_s3"
extended_s3_configuration {
# The IAM Role and S3 Bucket are declared by the kinesis-firehose-connector module
role_arn = module.kinesis_firehose_connector.firehose_role_arn
bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
}
tags = {
# This SymEnv tag is required and MUST match the SymEnv specified in the aws_iam_policy.aws_kinesis_firehose policy
SymEnv = local.environment_name
}
}
Add a Log Destination
Define a sym_log_destination
resource with type = kinesis_firehose
.
integration_id
: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions created by theaws/kinesis-firehose
add-on.stream_name
: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "s3_firehose" {
type = "kinesis_firehose"
# The Runtime Permission Context has Kinesis Firehose permissions defined by aws_iam_policy.aws_kinesis_firehose
integration_id = sym_integration.runtime_context.id
settings = {
stream_name = aws_kinesis_firehose_delivery_stream.sym_logs.name
}
}
Add the Log Destination to your Environment
Each sym_environment
accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids
list.
# ... other resources omitted
resource "sym_environment" "this" {
# ... other attributes omitted
# Add your log destinations here
log_destination_ids = [sym_log_destination.s3_firehose.id]
# ... other attributes omitted
}
Full Example
You can find the complete code for this example in our S3 Log Destination Example.
Updated 11 days ago