One of the nice things about running your container workloads on AWS ECS is the ability to use AWS Systems Manager Parameters to store sensitive values and inject them into your containers as Docker secrets.

This is a more secure option because your sensitive variables do not need to be accessible from your deployment tools, they only have to be available from the ECS/EC2 instance using an IAM role. For our example, we are going to use Terraform to create the SSM Parameter, and the service that consumes it. Also, injecting the wrong endpoint configuration into the wrong environment becomes a thing of the past.

We will need to make sure that the EC2 instance is configured to talk to the Systems Manager and has permission to pull parameter values.

Step 1: Configure SSM Agent

You’ll want to configure the SSM Agent on your ECS nodes, preferably in the user data file. Example:

# install pip curl -O https://bootstrap.pypa.io/get-pip.py python get-pip.py --user export PATH=~/.local/bin:$PATH # install AWS CLI pip install --upgrade --user awscli export PATH=/home/ec2-user/.local/bin:$PATH # install SSM Agent and dependencies sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm sudo yum install -y polkit # install cloudwatch agent curl -o amazon-cloudwatch-agent.rpm https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm sudo rpm -U ./amazon-cloudwatch-agent.rpm

Step 2: IAM Role Permissions

Whatever role your EC2 instances launch with will need to have access to retrieve SSM Parameter Store params. Example IAM policy:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:DescribeParameters" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ssm:GetParameters" ], "Resource": "arn:aws:ssm:us-east-1:<accountid>:parameter/*" } ] }

Now your EC2 instances have permission to call SSM, and they also have the AWS CLI and SSM Agent installed. Now we need to create some parameters. You can use the Amazon console for this, but for this example we are using Terraform.

Step 3: Create a SSM Parameter

SSM Parameters lend well to a nested path naming structure. In this example, we use parameters separated by environment and application.

locals { db_host="db1.mydomain.local" } resource "aws_ssm_parameter" "db_host" { name = "/production/myapp/db-host" type = "String" value = "${local.db_host}" }

Don’t forget to run:

>terraform plan

and

>terraform apply

Step 4: Inject SSM Parameter into ECS

In your task definition, add each value in a secrets block, using valueFrom instead of value, with an ARN reference to your parameter.

, "secrets": [ { "name": "DB_HOST", "valueFrom": "arn:aws:ssm:${var.aws_region}:<account_id>:parameter/production/myapp/db-host" } ]

Using this technique, access to sensitive information is much more restrictive than if you were keeping these values within your code. Developers only need environment variables for local development, and container orchestration can handle injecting the right value for the right app in the right environment for hosted applications. Secrets are now on a “need to know” basis. Fantastic!

May 21 20
derrickatalto9

 

Datadog is an incredibly powerful APM and infrastructure monitoring and alerting tool. Terraform is an incredibly powerful infrastructure automation tool. If you are scripting your infrastructure with Terraform, you’ll want to make sure that your monitors and alerts are scripted as well.

You’ll need to make sure you have set up a Datadog API Key and App Key before scripting with Terraform. It is common to use variables for your api key and app key when you include your provider.

In a file called variables.tf, declare the following variables:

variable "datadog_api_key" { default = "" } variable "datadog_app_key" { default = "" }

In a file called terraform.tfvars, place your DataDog API and app keys (do not commit this file to source control if you can help it):

datadog_api_key = "************" datadog_app_key = "************"

Now, in a file called main.tf, place your provider and pass your key variables to it:

provider "datadog" { api_key = "${var.datadog_api_key}" app_key = "${var.datadog_app_key}" }

Initialize the Datadog provider from the command line

> terraform init

Now you are ready to create DataDog monitors from Terraform. Let’s look at how to setup a simple drive space alert. First, let’s go back to variables.tf and add one more variable for our drive space alert thresholds.

variable "c_disk_thresholds" { type = "map" default = { critical = 90 warning = 85 ok = 80 } }

The variable above is a map containing default values for ‘ok’, ‘warning’, and ‘critical’ thresholds, and our monitor will be measured by percentage, so we use 80, 85, and 90 respectively.

We are going to add one more variable for the alert footer text. This is where you are going to want to put your recipients.

variable "datadog_alert_footer" { default = <<EOF @your-dd-slack-user @you@yourdomain.com EOF }

Now that we have all our variables in place, we can create our alert. Our goal with this alert is to monitor all Windows agents C drive, warning at 85% capacity, and going critical at 90% capacity. Either create a new .tf file, or just add this right in main.tf.

resource "datadog_monitor" "c_disk_free" { name = "{{host.name}} C Low Free Space" query = "avg(last_5m):avg:system.disk.in_use{device:c:} by {host} * 100 > ${var.c_disk_thresholds.critical}" type = "metric alert" notify_no_data = false include_tags = true thresholds = "${var.c_disk_thresholds}" message = <<EOM {{#is_alert}} C Drive Usage is {{value}} percent. {{/is_alert}} {{#is_recovery}} C Drive Usage returned to a safe state, {{value}} percent. {{/is_recovery}} ${var.datadog_alert_footer} EOM }

Run terraform plan and apply if there are no issues

> terraform apply

Once your changes are applied, you will see your new alert monitoring all Windows server C drives!

DataDog Alert Profile
May 21 20
derrickatalto9