AWS Kinesis Firehose

Send Sym logs to Kinesis Firehose, and from there, anywhere else!

Overview

With the Kinesis Firehose Log Destination, you can send the full stream of Reporting events from Sym to any destination supported by Kinesis Firehose.

This is a powerful integration that can sit upstream of any number of logging destinations, including:

  • AWS S3
  • DataDog
  • New Relic
  • Redshift
  • Splunk

📘

Prerequisites

If you have not yet, follow the AWS Runtime Setup tutorial to set up your runtime_connector module before continuing.

Give the Runtime Connector Role Permissions to Publish to Kinesis Firehose

In connectors.tf, add a module named kinesis_firehose_access:

# The kinesis_firehose_access module generates an AWS IAM Policy that grants permissions to publish to the given AWS Kinesis Firehose.
# Those permissions will be granted to the Runtime Connector IAM Role so that the Sym Runtime can publish to the Kinesis Firehose.
module "kinesis_firehose_access" {
  source  = "symopsio/kinesis-firehose-addon/aws"
  version = ">= 1.1.0"

  environment = local.environment_name
  iam_role_name = module.runtime_connector.sym_runtime_connector_role.name
}

The kinesis_firehose_access modules defines an AWS IAM Policy that grants Put Record permissions to Kinesis Firehose destinations that are tagged with SymEnv = environment_name and grants the permission to list delivery streams.

Downstream logging destinations

Once you've set up a basic Kinesis Firehose, you will be able to pipe your Sym logs downstream to a number of destinations. The simplest approach is to send logs to an S3 bucket, but the combination of Kinesis + Terraform's mutual support for sending logs to HTTP endpoints makes most downstream integrations a breeze.

Downstream configurations for AWS Kinesis Firehose will usually take one of three types: Kinesis firehose built-in support, support via HTTP endpoint, or abstracted support via Sym module.

Further reading

For more information on sending logs downstream from AWS Kinesis Firehose, see the HashiCorp AWS Provider docs here.