Datadog via Kinesis Firehose
Overview
Use the datadog-connector
module to provision a Kinesis Firehose that to pipes logs to Datadog. For more information about streaming logs from Kinesis Firehose to Datadog, checkout the Datadog Docs.
Prerequisites
- An
environment.tf
file generated bysymflow init
a. If you have not runsymflow init
, please follow the instructions in Installing Sym- A
runtime_connector
module defined inconnectors.tf
a. If you do not have aconnectors.tf
, please follow the instructions in AWS Runtime Setup- A
kinesis_firehose_access
module defined inconnectors.tf
during the tutorial on the main AWS Kinesis Firehose
Create a DataDog API key
On the Datadog Configuration page, create an API Key. Save the key, so that you can use it to configure the Datadog Connector in the next section.
Configure the Datadog Connector
Declare a datadog_access_key
variable
datadog_access_key
variableWe will be utilizing a sensitive variable to pass your Datadog access key to the datadog-connector
module.
Security Note
The
datadog_access_key
variable is sensitive. Be sure to manage this value either:
- In AWS Secrets Manager, where the value can be retrieved using
aws_secretsmanager_secret_version
- In an environment variable or a
tfvars
file that is excluded from version control. For example,export TF_VAR_datadog_access_key="my-access-key"
.The Access Key will also still be stored in Terraform state, so ensure that your Terraform state file is securely stored. See Terraform's recommendations for storing sensitive state.
Declare a variable datadog_access_key
in a variables.tf
file.
variable "datadog_access_key" {
description = "Secret used by the Firehose to send logs to Datadog. DO NOT check this into version control."
type = string
sensitive = true
}
Add the Datadog Connector Module
The Datadog Connector is a extended version of the Kinesis Firehose Connector, and will create a Kinesis Firehose Delivery Stream that will pipe reporting logs to Datadog, given an API key. The output of this module will be used to configure a Sym Log Destination next.
Check your environments!
Make sure that the
environment
in yourdatadog_connector
module matches theenvironment
input of yourkinesis_firehose_access
module.
module "datadog_connector" {
source = "symopsio/datadog-connector/aws"
# Note: To use this module with AWS Provider 4.x, use version "~> 3.0"
version = "~> 4.0"
# This environment value MUST match environment value passed into the kinesis_firehose_access module
environment = local.environment_name
# This variable should NOT be checked into source control!
datadog_access_key = var.datadog_access_key
# If you are in the EU, you will need to specify the EU intake URL!
# datadog_intake_url = "https://aws-kinesis-http-intake.logs.datadoghq.eu/v1/input"
# Any tags passed into the connector will be used to tag the Kinesis Firehose Deliery Stream,
# as well as each log entry sent to Datadog. The SymEnv = environment_name tag will always
# be included by default.
tags = {
TagName = "TagValue"
}
}
Add a Log Destination
Define a sym_log_destination
resource with type = kinesis_firehose
.
integration_id
: The integration containing the permissions to push to Kinesis Firehose. This should be set to module.runtime_connector.sym_integration.id, which has the permissions created by thekinesis_firehose_access
module.stream_name
: The name of the Kinesis Firehose Delivery Stream
resource "sym_log_destination" "datadog" {
type = "kinesis_firehose"
# The Runtime Connector sym_integration has Kinesis Firehose permissions defined by the kinesis_firehose_access module
integration_id = module.runtime_connector.sym_integration.id
settings = {
# The firehose stream name is outputted by the datadog_connector module
stream_name = module.datadog_connector.firehose_name
}
}
Add the Log Destination to your Environment
Each sym_environment
accepts a list of Log Destinations to send reporting logs to. Add the ID of the Log Destination you just defined to the log_destination_ids
list.
resource "sym_environment" "this" {
name = "main"
runtime_id = sym_runtime.this.id
error_logger_id = sym_error_logger.slack.id
# Add your log destinations here
log_destination_ids = [sym_log_destination.datadog.id]
integrations = {
slack_id = sym_integration.slack.id
}
}
Full Example
You can find the complete code for this example in our Datadog Log Destination Example.
Updated 2 months ago