New Relic
Overview
Once you've set up your Firehose, connecting it to New Relic is as easy as configuring a downstream http_endpoint
destination and backup s3_configuration
in Terraform.
The following Kinesis Firehose setup is a Terraform version of the New Relic tutorial.
Prerequisites
- An
environment.tf
file generated bysymflow init
a. If you have not runsymflow init
, please follow the instructions in Installing Sym- A
runtime_connector
module defined inconnectors.tf
a. If you do not have aconnectors.tf
, please follow the instructions in AWS Runtime Setup- A
kinesis_firehose_access
module defined inconnectors.tf
during the tutorial on the main AWS Kinesis Firehose
Create a New Relic License Key
You will need a New Relic License Key. Create one in the New Relic API Key UI.
Save the New Relic License Key in AWS Secrets Manager
For Kinesis Firehose to stream to New Relic, it will need an API key. Saving it in AWS Secrets Manager will make it easily and securely accessible when configuring the delivery stream later.
resource "aws_secretsmanager_secret" "new_relic_api_key" {
name = "sym/main/new-relic-api-key"
description = "New Relic API Key for Kinesis Firehose"
}
Terraform apply the secret and set its value
aws secretsmanager put-secret-value --secret-id "sym/main/new-relic-api-key" --secret-string "YOUR-NEW-RELIC-API-KEY"
Declare the Kinesis Firehose Connector
The Kinesis Firehose Connector module declares the AWS dependencies required to declare a Kinesis Firehose, such as the IAM role the Firehose will assume and the backup S3 bucket.
module "kinesis_firehose_connector" {
source = "symopsio/kinesis-firehose-connector/aws"
version = ">= 3.0.0"
environment = "main"
}
Create a Delivery Stream
Declare a aws_kinesis_firehose_delivery_stream resource, and set the destination to http_endpoint
. This will declare a Kinesis Firehose that will forward to the configured HTTP endpoint.
Configuring the Delivery Stream Buffer Size
New Relic strongly advises to use 1 MiB as the Buffer size and to use GZIP body compression.
SymEnv is a required tag!
Your Kinesis Firehose Delivery Stream must have a tag
SymEnv
that matches the environment input of yourkinesis_firehose_access
module!
resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
name = "SymNewRelicReportingLogsMain"
destination = "http_endpoint"
# New Relic destination via HTTP
http_endpoint_configuration {
url = "https://aws-api.newrelic.com/firehose/v1"
name = "New Relic"
# Your New Relic API Key stored in AWS
access_key = aws_secretsmanager_secret.new_relic.arn
# New Relic's recommended buffering size is 1 MiB
buffering_size = 1 # MiB
buffering_interval = 600 # Seconds
# The IAM Role is declared by the kinesis_firehose_connector module
role_arn = module.kinesis_firehose_connector.firehose_role_arn
s3_backup_mode = "FailedDataOnly"
request_configuration {
content_encoding = "GZIP"
}
}
# Backup S3 configuration
s3_configuration {
role_arn = module.kinesis_firehose_connector.firehose_role_arn
bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
buffer_size = 10
buffer_interval = 60
compression_format = "GZIP"
}
tags = {
# This SymEnv tag is required and MUST match the SymEnv specified in your kinesis_firehose_access module
SymEnv = local.environment_name
}
}
Add a Log Destination
Define a sym_log_destination
resource with type = kinesis_firehose
.
integration_id
: The integration containing the permissions to push to Kinesis Firehose. This should be set tomodule.runtime_connector.sym_integration.id
, which has the permissions created by thekinesis_firehose_access
module.stream_name
: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "new_relic" {
type = "kinesis_firehose"
# The Runtime Connector sym_integration has Kinesis Firehose permissions defined by the kinesis_firehose_access module
integration_id = module.runtime_connector.sym_integration.id
settings = {
stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
}
}
Add the Log Destination to your Environment
Each sym_environment
accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids
list.
resource "sym_environment" "this" {
name = "main"
runtime_id = sym_runtime.this.id
error_logger_id = sym_error_logger.slack.id
# Add your log destinations here
log_destination_ids = [sym_log_destination.new_relic.id]
integrations = {
slack_id = sym_integration.slack.id
}
}
Example Configuration Snippet
# Note: This snippet is truncated. Other unrelated resources have been omitted
# The runtime_connector module creates an IAM Role that the Sym Runtime can assume to execute operations in your AWS account.
module "runtime_connector" {
source = "symopsio/runtime-connector/aws"
version = "~> 2.0"
environment = local.environment_name
}
# The kinesis_firehose_access module generates an AWS IAM Policy that grants permissions to publish to the given AWS Kinesis Firehose.
# Those permissions will be granted to the Runtime Connector IAM Role so that the Sym Runtime can publish to the Kinesis Firehose.
module "kinesis_firehose_access" {
source = "symopsio/kinesis-firehose-addon/aws"
version = ">= 1.1.0"
environment = local.environment_name
iam_role_name = module.runtime_connector.sym_runtime_connector_role.name
}
# New Relic API Key
resource "aws_secretsmanager_secret" "new_relic_api_key" {
name = "main/new-relic-api-key"
description = "New Relic API Key for Kinesis Firehose"
}
# Kinesis Firehose Dependencies
module "kinesis_firehose_connector" {
source = "symopsio/kinesis-firehose-connector/aws"
version = ">= 3.0.0"
environment = local.environment_name
}
# Kinesis Firehose Delivery Stream to New Relic
resource "aws_kinesis_firehose_delivery_stream" "new_relic" {
name = "SymNewRelicReportingLogsMain"
destination = "http_endpoint"
# New Relic destination via HTTP
http_endpoint_configuration {
url = "https://aws-api.newrelic.com/firehose/v1"
name = "New Relic"
# Your New Relic API Key stored in AWS
access_key = aws_secretsmanager_secret.new_relic.arn
# New Relic's recommended buffering size is 1 MiB
buffering_size = 1 # MiB
buffering_interval = 600 # Seconds
# The IAM Role is declared by the kinesis_firehose_connector module
role_arn = module.kinesis_firehose_connector.firehose_role_arn
s3_backup_mode = "FailedDataOnly"
request_configuration {
content_encoding = "GZIP"
}
}
# Backup S3 configuration
s3_configuration {
role_arn = module.kinesis_firehose_connector.firehose_role_arn
bucket_arn = module.kinesis_firehose_connector.firehose_bucket_arn
buffer_size = 10
buffer_interval = 60
compression_format = "GZIP"
}
tags = {
# This SymEnv tag is required and MUST match the SymEnv specified in your kinesis_firehose_access module
SymEnv = local.environment_name
}
}
# The Sym Log Destination pointing to the New Relic Firehose
resource "sym_log_destination" "new_relic" {
type = "kinesis_firehose"
# The Runtime Connector sym_integration has Kinesis Firehose permissions defined by the kinesis_firehose_access module
integration_id = module.runtime_connector.sym_integration.id
settings = {
stream_name = aws_kinesis_firehose_delivery_stream.new_relic.name
}
}
# ... other resources omitted
# Sym Environment with New Relic as a Log Destination
resource "sym_environment" "this" {
# ... other attributes omitted
# Add your log destinations here
log_destination_ids = [sym_log_destination.new_relic.id]
# ... other attributes omitted
}
Updated 5 months ago