AWS Kinesis Data Stream

Send Sym logs to Kinesis Data Stream, and from there, anywhere else!

Suggest Edits

Overview

With the Kinesis Data Stream Log Destination, you can send the full stream of Reporting events from Sym to be processed by a Data Stream.

Simply create a Log Destination of type kinesis_data_stream, and add it to your Environment.

🚧

Did you connect the Sym Runtime with your AWS Account?

If you have not yet, follow the Connect the Sym Runtime with your AWS Account tutorial on the main AWS page to set up your Runtime Connector module and Runtime Permission Context before continuing.

Configure the Kinesis Data Stream Connector

The aws/kinesis-data-stream-connector module creates a Kinesis Data Stream and the outputs will be used to configure the Kinesis Data Stream Add-on in the next step.

The Data Stream can be further configured with inputs to the module. See the Terraform Registry documentation for the Kinesis Data Stream Connector for more details.

module "kinesis_data_stream_connector" {
  source  = "symopsio/kinesis-data-stream-connector/aws"
  version = "1.0.0"

  environment = "main"

  # .. Other optional inputs. See Terraform registry docs.
}

Add the Kinesis Data Stream Add-on to your Runtime Connector

The aws/kinesis-data-stream add-on gives your Sym Runtime the permissions to push to AWS Kinesis Data Streams.

  • addon_params: To configure the AWS Kinesis Data Stream add-on, you will need to add a list of Data Stream ARNs that you wish to allow Sym to publish to. Include the Data stream created by the Kinesis Data Stream Connector.
module "runtime_connector" {
  source  = "symopsio/runtime-connector/aws"
  version = ">= 1.1.0"

  environment = "main"

    # This add-on is required for Reporting via Kinesis Data Stream
  addons = ["aws/kinesis-data-stream"]

  addon_params = {
    # The streams that you wish to Sym to publish to.
    "aws/kinesis-data-stream" = {
       "stream_arns" = [module.kinesis_data_stream_connector.data_stream_arn]
    }
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_data_stream.

  • integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions created by the aws/kinesis-data-stream add-on.
  • stream_name: The name of the Kinesis Data Stream created by the Kinesis Data Stream Connector.
resource "sym_log_destination" "data_stream" {
  type = "kinesis_data_stream"
  
  # The Runtime Permission Context has Kinesis Data Stream permissions from the aws/kinesis-data-stream add-on
  integration_id = sym_integration.runtime_context.id
  
  settings = {
    stream_name = module.kinesis_data_stream_connector.data_stream_name
  }  
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.data_stream.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Example Configuration Snippet

# Note: This snippet is truncated. Other unrelated resources have been omitted

# Kinesis Data Stream Connector to create a Data Stream
module "kinesis_data_stream_connector" {
  source  = "symopsio/kinesis-data-stream-connector/aws"
  version = "1.0.0"

  environment = "main"

  # .. Other optional inputs. See Terraform registry docs.
}

# Runtime Connector with Kinesis Data Stream add-on
module "runtime_connector" {
  source  = "symopsio/runtime-connector/aws"
  version = ">= 1.0.0"

  environment = "main"

    # This add-on is required for Reporting via Kinesis Data Stream
  addons = ["aws/kinesis-data-stream"]

  addon_params = {
    # The streams that you wish to Sym to publish to.
    "aws/kinesis-data-stream" = {
       "stream_arns" = [module.kinesis_data_stream_connector.data_stream_arn]
    }
  }
}

# The Sym Log Destination pointing to the Data Stream
resource "sym_log_destination" "data_stream" {
  type = "kinesis_data_stream"
  
  # The Runtime Permission Context has Kinesis Data Stream permissions from the aws/kinesis-data-stream add-on
  integration_id = sym_integration.runtime_context.id
  
  settings = {
    stream_name = module.kinesis_data_stream_connector.data_stream_name
  }  
}

# Sym Environment with Kinesis Data Stream as a log destination
resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.data_stream.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Updated about 1 month ago