AWS Kinesis Data Stream
Send Sym logs to Kinesis Data Stream, and from there, anywhere else!
Overview
With the Kinesis Data Stream Log Destination, you can send the full stream of Reporting events from Sym to be processed by a Data Stream.
Simply create a Log Destination of type kinesis_data_stream
, and add it to your Environment.
Prerequisites
- An
environment.tf
file generated bysymflow init
a. If you have not runsymflow init
, please follow the instructions in Installing Sym- A Runtime Connector Role defined in
runtime.tf
a. If you do not have aruntime.tf
, please follow the instructions in Connecting Sym to AWS
Configure the Kinesis Data Stream Log Destination
The aws/kinesis-data-stream-connector
module creates a Kinesis Data Stream and the outputs are used to configure an AWS IAM Policy that grants the Runtime Connector Role permissions to publish to the Data Stream.
The Data Stream can be further configured with inputs to the module. See the Terraform Registry documentation for the Kinesis Data Stream Connector for more details.
Create a kinesis_data_stream.tf
file with the following contents:
# A module that provisions a single Kinesis Data Stream
module "kinesis_data_stream_connector" {
source = "symopsio/kinesis-data-stream-connector/aws"
version = ">= 1.0.0"
environment = local.environment_name
# .. Other optional inputs. See Terraform registry docs.
}
# An AWS IAM Policy that grants the permission to publish to the specified Kinesis Data Streams
# and the perimssion to list streams.
resource "aws_iam_policy" "kinesis_data_stream" {
name = "SymKinesisDataStreamProd"
path = "/sym/"
description = "AWS IAM Policy granting access to Kinesis Data Stream"
policy = <<EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:PutRecord",
"kinesis:PutRecords"
],
"Resource": [ module.kinesis_data_stream_connector.data_stream_arn ]
},
{
"Effect": "Allow",
"Action": [
"kinesis:ListStreams"
],
"Resource": "*"
}
]
}
EOT
}
# Attach the IAM policy declared above to the Runtime Connector Role.
resource "aws_iam_role_policy_attachment" "aws_kinesis_data_stream_attach" {
policy_arn = aws_iam_policy.kinesis_data_stream.arn
role = aws_iam_role.sym_runtime_connector_role.name
}
# A sym_log_destination that tells the Sym Runtime where to send audit logs.
resource "sym_log_destination" "data_stream" {
type = "kinesis_data_stream"
# The Runtime Permission Context has Kinesis Data Stream permissions from aws_iam_policy.kinesis_data_stream policy
integration_id = sym_integration.runtime_context.id
settings = {
stream_name = module.kinesis_data_stream_connector.data_stream_name
}
}
Add the Log Destination to your Environment
Each sym_environment
accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids
list.
# ... other resources omitted
resource "sym_environment" "this" {
# ... other attributes omitted
# Add your log destinations here
log_destination_ids = [sym_log_destination.data_stream.id]
# ... other attributes omitted
}
Example Configuration Snippet
# Note: Other unrelated resources have been omitted
# A module that provisions a single Kinesis Data Stream
module "kinesis_data_stream_connector" {
source = "symopsio/kinesis-data-stream-connector/aws"
version = ">= 1.0.0"
environment = local.environment_name
# .. Other optional inputs. See Terraform registry docs.
}
# An AWS IAM Policy that grants the permission to publish to the specified Kinesis Data Streams
# and the perimssion to list streams.
resource "aws_iam_policy" "kinesis_data_stream" {
name = "SymKinesisDataStreamProd"
path = "/sym/"
description = "AWS IAM Policy granting access to Kinesis Data Stream"
policy = <<EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:PutRecord",
"kinesis:PutRecords"
],
"Resource": [ module.kinesis_data_stream_connector.data_stream_arn ]
},
{
"Effect": "Allow",
"Action": [
"kinesis:ListStreams"
],
"Resource": "*"
}
]
}
EOT
}
# Attach the IAM policy declared above to the Runtime Connector Role.
resource "aws_iam_role_policy_attachment" "aws_kinesis_data_stream_attach" {
policy_arn = aws_iam_policy.kinesis_data_stream.arn
role = aws_iam_role.sym_runtime_connector_role.name
}
# A sym_log_destination that tells the Sym Runtime where to send audit logs.
resource "sym_log_destination" "data_stream" {
type = "kinesis_data_stream"
# The Runtime Permission Context has Kinesis Data Stream permissions from aws_iam_policy.kinesis_data_stream policy
integration_id = sym_integration.runtime_context.id
settings = {
stream_name = module.kinesis_data_stream_connector.data_stream_name
}
}
Updated 11 days ago