AWS instances can fill their disks with logs files, Docker volumes, old versions of libraries - and once the disk is full, the server is likely to fall over, and you may not even be able to SSH into it.

So it would be great to have a way of being alerted if the disk space is getting low..

One way to do that is to install the CloudWatch agent, configure it to report on disk space, and set up an alert if the disk space usage exceeds a certain limit.

The steps are as follows:

  1. create an IAM role to allow the agent to report metrics - the role will be associated with the EC2 instance
  2. install the CloudWatch agent on the server and start it - it'll need a config file
  3. configure the alarm in CloudWatch with a threshold value and a notification action

Create IAM Role for CloudWatch Agent

The role looks like this:

Make sure your EC2 instance is has this role:

Install the CloudWatch agent on the server

For Ubuntu, the easiest thing is to download the deb file and install from the command line.

I use this script - invoke it with the IP address of the server as the parameter (it expects that you'll have a key file called "aws.pem" in your .ssh directory)

#!/bin/bash
set -e

# Public IP of the EC2 instance
ip="$1"

# set up access commands and variables
PEM="${HOME}/.ssh/aws.pem"
SSH="ssh -i ${PEM} ubuntu@${ip}"

$SSH wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$SSH sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

cat cloudwatch-agent-config.json | $SSH "cat > cloudwatch-agent-config.json"

$SSH sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:cloudwatch-agent-config.json

You'll need the JSON config file in your directory before you run that - something like this:

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "statsd": {
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      }
    }
  }
}

You can create your own config file using the wizard - see links below for details. Don't select the collectd options - you end up having to create dummy config files for that on your server.

Set up the CloudWatch alarm

The metrics should now be being sent to the CloudWatch service - check in the CloudWatch console

Then create a new alarm from that metric

and make sure that you set the threshold value to something sensible like 80%

You'll want a SNS topic that you can send alerts to, with an email subscription:

Finally, give it a name

You should now get an email whenever the disk space usage metric goes above the threshold value.

There are plenty more aspects of the metrics and the alarms that you can configure, once you're happy that the basics are working.

Full details:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html