New Relic

Suggest Edits

Overview

Once you've set up your Firehose, connecting it to New Relic is as easy as configuring a downstream http_endpoint destination and backup s3_configuration in Terraform.

The following Kinesis Firehose setup is a Terraform version of the New Relic tutorial.

📘

Prerequisites

  1. An environment.tf file generated by symflow init
    a. If you have not run symflow init, please follow the instructions in Installing Sym
  2. A Runtime Connector Role defined in runtime.tf
    a. If you do not have a runtime.tf, please follow the instructions in Connecting Sym to AWS
  3. A firehose.tf file created during the tutorial on the main AWS Kinesis Firehose

Create a New Relic License Key

You will need a New Relic License Key. Create one in the New Relic API Key UI.

Save the New Relic License Key in AWS Secrets Manager

For Kinesis Firehose to stream to New Relic, it will need an API key. Saving it in AWS Secrets Manager will make it easily and securely accessible when configuring the delivery stream later.

resource "aws_secretsmanager_secret" "new_relic_api_key" {
  name        = "sym/main/new-relic-api-key"
  description = "New Relic API Key for Kinesis Firehose"
}

Terraform apply the secret and set its value

aws secretsmanager put-secret-value --secret-id "sym/main/new-relic-api-key" --secret-string "YOUR-NEW-RELIC-API-KEY"

Declare the Kinesis Firehose Connector

The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.

module "kinesis_firehose_connector" {
  source      = "symopsio/kinesis-firehose-connector/aws"
  version     = ">= 3.0.0"
  environment = "main"
}

Create a Delivery Stream

Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to http_endpoint. This will declare a Kinesis Firehose that will forward to the configured HTTP endpoint.

📘

Configuring the Delivery Stream Buffer Size

New Relic strongly advises to use 1 MiB as the Buffer size and to use GZIP body compression.

🚧

SymEnv is a required tag!

Your Kinesis Firehose Delivery Stream must have a tag SymEnv that matches the environment specified in the aws_iam_policy.aws_kinesis_firehose policy you defined!

resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
  name = "SymNewRelicReportingLogsMain"
  destination = "http_endpoint"

  # New Relic destination via HTTP
  http_endpoint_configuration {
    url                = "https://aws-api.newrelic.com/firehose/v1"
    name               = "New Relic"

    # Your New Relic API Key stored in AWS
    access_key         = aws_secretsmanager_secret.new_relic.arn

    # New Relic's recommended buffering size is 1 MiB
    buffering_size     = 1 # MiB
    buffering_interval = 600 # Seconds

    # The IAM Role is declared by the kinesis_firehose_connector module
    role_arn           = module.kinesis_firehose_connector.firehose_role_arn
    s3_backup_mode     = "FailedDataOnly"

    request_configuration {
      content_encoding = "GZIP"
    }
  }

  # Backup S3 configuration
  s3_configuration {
    role_arn = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
    buffer_size        = 10
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  tags = {
    # This SymEnv tag is required and MUST match the SymEnv tag specified in the aws_iam_policy.aws_kinesis_firehose policy
    SymEnv = local.environment_name
  }
}

Add a Log Destination

Define a sym_log_destination resource with type = kinesis_firehose.

  • integration_id: The integration containing the permissions to push to Kinesis Firehose. This should be set to your Runtime Permission Context Integration, which has the permissions defined by the aws_iam_policy.aws_kinesis_firehose resource.
  • stream_name: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "new_relic" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions from aws_iam_policy.aws_kinesis_firehose policy
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
  }
}

Add the Log Destination to your Environment

Each sym_environment accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids list.

resource "sym_environment" "this" {
  name            = "main"
  runtime_id      = sym_runtime.this.id
  error_logger_id = sym_error_logger.slack.id
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.new_relic.id]

  integrations = {
    slack_id = sym_integration.slack.id
  }
}

Example Configuration Snippet

# Note: This snippet is truncated. Other unrelated resources have been omitted

# An AWS IAM Policy that grants the permission to publish to Kinesis Firehose Delivery Streams tagged with SymEnv
# and the perimssion to list Delivery Streams.
resource "aws_iam_policy" "aws_kinesis_firehose" {
  name = "SymKinesisFirehose${title(local.environment_name)}"
  path = "/sym/"

  description = "Addon policy granting access to Kinesis Firehose"
  policy      = <<EOT
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "*",
      "Condition": { "StringEquals": { "firehose:ResourceTag/SymEnv": "${local.environment_name}" } }
    },
    {
      "Effect": "Allow",
      "Action": [
        "firehose:ListDeliveryStreams"
      ],
      "Resource": "*"
    }
  ]
}
EOT
}

# Attach the IAM policy declared above to the Runtime Connector Role defined in runtime.tf
resource "aws_iam_role_policy_attachment" "aws_kinesis_firehose_attach" {
  policy_arn = aws_iam_policy.aws_kinesis_firehose.arn
  role       = aws_iam_role.sym_runtime_connector_role.name
}

# New Relic API Key
resource "aws_secretsmanager_secret" "new_relic_api_key" {
  name        = "main/new-relic-api-key"
  description = "New Relic API Key for Kinesis Firehose"
}

# Kinesis Firehose Dependencies
module "kinesis_firehose_connector" {
  source  = "symopsio/kinesis-firehose-connector/aws"
  version = ">= 3.0.0"
  environment = local.environment_name
}

# Kinesis Firehose Delivery Stream to New Relic
resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
  name = "SymNewRelicReportingLogsMain"
  destination = "http_endpoint"

  # New Relic destination via HTTP
  http_endpoint_configuration {
    url                = "https://aws-api.newrelic.com/firehose/v1"
    name               = "New Relic"

    # Your New Relic API Key stored in AWS
    access_key         = aws_secretsmanager_secret.new_relic.arn

    # New Relic's recommended buffering size is 1 MiB
    buffering_size     = 1 # MiB
    buffering_interval = 600 # Seconds

    # The IAM Role is declared by the kinesis_firehose_connector module
    role_arn           = module.kinesis_firehose_connector.firehose_role_arn
    s3_backup_mode     = "FailedDataOnly"

    request_configuration {
      content_encoding = "GZIP"
    }
  }

  # Backup S3 configuration
  s3_configuration {
    role_arn = module.kinesis_firehose_connector.firehose_role_arn
    bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
    buffer_size        = 10
    buffer_interval    = 60
    compression_format = "GZIP"
  }

  tags = {
    # This SymEnv tag is required and MUST match the SymEnv specified in the aws_iam_policy.aws_kinesis_firehose policy
    SymEnv = local.environment_name
  }
}

# The Sym Log Destination pointing to the New Relic Firehose
resource "sym_log_destination" "new_relic" {
  type = "kinesis_firehose"

  # The Runtime Permission Context has Kinesis Firehose permissions from the aws_iam_policy.aws_kinesis_firehose policy
  integration_id = sym_integration.runtime_context.id

  settings = {
    stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
  }
}

# ... other resources omitted

# Sym Environment with New Relic as a Log Destination  
resource "sym_environment" "this" {
  # ... other attributes omitted
  
  # Add your log destinations here
  log_destination_ids = [sym_log_destination.new_relic.id]

  # ... other attributes omitted
}

Updated 11 days ago