Downstream Logging Examples

Logs are only useful if you can read them somewhere.

Overview

Once you've set up a basic Kinesis Firehose or Data Stream integration, you can pipe your Sym logs downstream to a number of destinations. The simplest approach is to send logs to an S3 bucket, but the combination of Kinesis + Terraform's mutual support for sending logs to HTTP endpoints makes most downstream integrations a breeze.

In this section, we'll show you a number of partial implementations for ensuring your logs can be shipped end to end into your reporting destination of choice.

Firehose downstream configurations

Downstream configurations for AWS Kinesis Firehose will usually take one of three types: Kinesis firehose built-in support, support via HTTP endpoint, or abstracted support via Sym module.

Sym module

Sym has a growing list of native support that makes implementation as easy as writing a single module.

module "datadog_connector" {
  source  = "terraform.symops.com/symopsio/datadog-connector/sym"
  version = ">= 1.0.0"

  environment = "prod"
  datadog_access_key = var.sensitive_access_key
}

Kinesis supported destination

For many common logging destinations, you can natively declare your downstream system as a destination inside of your delivery stream resource.

resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
  name        = "name-of-the-stream"
  destination = "destination-service-name" # e.g. "splunk" or "redshift"
  
  # Configuration details
  # .
  # .
  # .
}

HTTP endpoint destination

Others (e.g. New Relic) can be set up via http_endpoint_configuration. In this case, the resource will require both the configuration for the HTTP endpoint, and an additional backup configuration for S3.

resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
  name        = "name-of-the-stream"
  destination = "http_endpoint"

  s3_configuration {
    role_arn           = aws_iam_role.firehose.arn
    bucket_arn         = aws_s3_bucket.bucket.arn
  }

  http_endpoint_configuration {
    url                = "destination-url"
    name               = "destination-name"
    # Configuration details
    # .
    # . 
    # .
  }
}

Further reading

For more information on sending logs downstream from AWS Kinesis Firehose, see the HashiCorp AWS Provider docs here.


Did this page help you?