New Relic

Suggest Edits

Overview

Once you've set up your Firehose, connecting it to New Relic is as easy as configuring a downstream http_endpoint destination and backup s3_configuration in Terraform.

The following Kinesis Firehose setup is a Terraform version of the New Relic tutorial.

📘

Make sure you've enabled the aws/kinesis-firehose add-on in your Runtime Connector, as described on the main AWS Kinesis Firehose page.

Create a New Relic License Key

You will need a New Relic License Key. Create one in the New Relic API Key UI.

Save the New Relic License Key in AWS Secrets Manager

For Kinesis Firehose to stream to New Relic, it will need an API key. Saving it in AWS Secrets Manager will make it easily and securely accessible when configuring the delivery stream later.

resource "aws_secretsmanager_secret" "new_relic_api_key" {
  name        = "main/new-relic-api-key"
  description = "New Relic API Key for Kinesis Firehose"
}

Terraform apply the secret and set its value

aws secretsmanager put-secret-value --secret-id "main/new-relic-api-key" --secret-string "YOUR-NEW-RELIC-API-KEY"

Declare the Kinesis Firehose Connector

The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.

module "kinesis_firehose_connector" {
  source  = "symopsio/kinesis-firehose-connector/aws"
  version = ">= 3.0.0"
  environment = "main"
}

Create a Delivery Stream

Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to http_endpoint. This will declare a Kinesis Firehose that will forward to the configured HTTP endpoint.

📘

Configuring the Delivery Stream Buffer Size

New Relic strongly advises to use 1 MiB as the Buffer size and to use GZIP body compression.

🚧

SymEnv is a required tag!

Your Kinesis Firehose Delivery Stream must have a tag SymEnv that matches the environment specified in your runtime_connector module!

resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
  name = "SymNewRelicReportingLogsMain"
  destination = "http_endpoint"

  # New Relic destination via HTTP
  http_endpoint_configuration {
    url                = "https://aws-api.newrelic.com/firehose/v1"
    name               = "New Relic"

    # Your New Relic API Key stored in AWS
    access_key         = aws_secretsmanager_secret.new_relic.arn

    # New Relic's recommended buffering size is 1 MiB
    buffering_size     = 1 # MiB
    buffering_interval = 600 # Seconds

    # The IAM Role is declared by the kinesis_firehose_connector module
    role_arn           = module.kinesis_firehose_connector.firehose_role_arn
    s3_backup_mode     = "FailedDataOnly"

    request_configuration {
      content_encoding = "GZIP"
    }
  }

  # Backup S3 configuration
  s3_configuration {
    role_arn = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
    buffer_size        = 10
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  tags = {
    # This SymEnv tag is required and MUST match the `environment` in your `runtime_connector` module
    # because the aws/kinesis-firerhose add-on only grants access to Firehoses tagged with a matching SymEnv value
    SymEnv = "main"
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_firehose.

  • integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions created by the aws/kinesis-firehose add-on.
  • stream_name: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "new_relic" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions from the aws/kinesis-firehose add-on
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
  }
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.new_relic.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Example Configuration Snippet

# Note: This snippet is truncated. Other unrelated resources have been omitted

# Runtime Connector with Kinesis Firehose add-on
module "runtime_connector" {
  source  = "symopsio/runtime-connector/aws"
  version = ">= 1.0.0"

  environment = "main"

    # This add-on is required for Reporting via Kinesis Firehose
  addons = ["aws/kinesis-firehose"]
}

# New Relic API Key
resource "aws_secretsmanager_secret" "new_relic_api_key" {
  name        = "main/new-relic-api-key"
  description = "New Relic API Key for Kinesis Firehose"
}

# Kinesis Firehose Dependencies
module "kinesis_firehose_connector" {
  source  = "symopsio/kinesis-firehose-connector/aws"
  version = ">= 3.0.0"
  environment = "main"
}

# Kinesis Firehose Delivery Stream to New Relic
resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
  name = "SymNewRelicReportingLogsMain"
  destination = "http_endpoint"

  # New Relic destination via HTTP
  http_endpoint_configuration {
    url                = "https://aws-api.newrelic.com/firehose/v1"
    name               = "New Relic"

    # Your New Relic API Key stored in AWS
    access_key         = aws_secretsmanager_secret.new_relic.arn

    # New Relic's recommended buffering size is 1 MiB
    buffering_size     = 1 # MiB
    buffering_interval = 600 # Seconds

    # The IAM Role is declared by the kinesis_firehose_connector module
    role_arn           = module.kinesis_firehose_connector.firehose_role_arn
    s3_backup_mode     = "FailedDataOnly"

    request_configuration {
      content_encoding = "GZIP"
    }
  }

  # Backup S3 configuration
  s3_configuration {
    role_arn = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
    buffer_size        = 10
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  tags = {
    # This SymEnv tag is required and MUST match the `environment` in your `runtime_connector` module
    # because the aws/kinesis-firerhose add-on only grants access to Firehoses tagged with a matching SymEnv value
    SymEnv = "main"
  }
}

# The Sym Log Destination pointing to the New Relic Firehose
resource "sym_log_destination" "new_relic" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions from the aws/kinesis-firehose add-on
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
  }
}

# Sym Environment with New Relic as a Log Destination  
resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.new_relic.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Updated 2 months ago