AWS Kinesis Data Stream

Send Sym logs to Kinesis Data Stream, and from there, anywhere else!

Suggest Edits

Overview

With the Kinesis Data Stream Log Destination, you can send the full stream of Reporting events from Sym to be processed by a Data Stream.

Simply create a Log Destination of type kinesis_data_stream, and add it to your Environment.

📘

Prerequisites

  1. An environment.tf file generated by symflow init
    a. If you have not run symflow init, please follow the instructions in Installing Sym
  2. A Runtime Connector Role defined in runtime.tf
    a. If you do not have a runtime.tf, please follow the instructions in Connecting Sym to AWS

Configure the Kinesis Data Stream Log Destination

The aws/kinesis-data-stream-connector module creates a Kinesis Data Stream and the outputs are used to configure an AWS IAM Policy that grants the Runtime Connector Role permissions to publish to the Data Stream.

The Data Stream can be further configured with inputs to the module. See the Terraform Registry documentation for the Kinesis Data Stream Connector for more details.

Create a kinesis_data_stream.tf file with the following contents:

# A module that provisions a single Kinesis Data Stream
module "kinesis_data_stream_connector" {
  source  = "symopsio/kinesis-data-stream-connector/aws"
  version = ">= 1.0.0"

  environment = local.environment_name

  # .. Other optional inputs. See Terraform registry docs.
}

# An AWS IAM Policy that grants the permission to publish to the specified Kinesis Data Streams
# and the perimssion to list streams.
resource "aws_iam_policy" "kinesis_data_stream" {
  name = "SymKinesisDataStreamProd"
  path = "/sym/"

  description = "AWS IAM Policy granting access to Kinesis Data Stream"
  policy      = <<EOT
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [ module.kinesis_data_stream_connector.data_stream_arn ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams"
      ],
      "Resource": "*"
    }
  ]
}
EOT
}

# Attach the IAM policy declared above to the Runtime Connector Role.
resource "aws_iam_role_policy_attachment" "aws_kinesis_data_stream_attach" {
  policy_arn = aws_iam_policy.kinesis_data_stream.arn
  role       = aws_iam_role.sym_runtime_connector_role.name
}

# A sym_log_destination that tells the Sym Runtime where to send audit logs.
resource "sym_log_destination" "data_stream" {
  type = "kinesis_data_stream"
  
  # The Runtime Permission Context has Kinesis Data Stream permissions from aws_iam_policy.kinesis_data_stream policy
  integration_id = sym_integration.runtime_context.id
  
  settings = {
    stream_name = module.kinesis_data_stream_connector.data_stream_name
  }  
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

# ... other resources omitted

resource "sym_environment" "this" {
  # ... other attributes omitted
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.data_stream.id]

  # ... other attributes omitted
}

Example Configuration Snippet

# Note: Other unrelated resources have been omitted

# A module that provisions a single Kinesis Data Stream
module "kinesis_data_stream_connector" {
  source  = "symopsio/kinesis-data-stream-connector/aws"
  version = ">= 1.0.0"

  environment = local.environment_name

  # .. Other optional inputs. See Terraform registry docs.
}

# An AWS IAM Policy that grants the permission to publish to the specified Kinesis Data Streams
# and the perimssion to list streams.
resource "aws_iam_policy" "kinesis_data_stream" {
  name = "SymKinesisDataStreamProd"
  path = "/sym/"

  description = "AWS IAM Policy granting access to Kinesis Data Stream"
  policy      = <<EOT
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [ module.kinesis_data_stream_connector.data_stream_arn ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams"
      ],
      "Resource": "*"
    }
  ]
}
EOT
}

# Attach the IAM policy declared above to the Runtime Connector Role.
resource "aws_iam_role_policy_attachment" "aws_kinesis_data_stream_attach" {
  policy_arn = aws_iam_policy.kinesis_data_stream.arn
  role       = aws_iam_role.sym_runtime_connector_role.name
}

# A sym_log_destination that tells the Sym Runtime where to send audit logs.
resource "sym_log_destination" "data_stream" {
  type = "kinesis_data_stream"
  
  # The Runtime Permission Context has Kinesis Data Stream permissions from aws_iam_policy.kinesis_data_stream policy
  integration_id = sym_integration.runtime_context.id
  
  settings = {
    stream_name = module.kinesis_data_stream_connector.data_stream_name
  }  
}

Updated 11 days ago