Security Considerations When Enabling Systems Manager by Default for EC2

Introduction

A couple days ago Amazon announced the release of a new feature called Default Host Management Configuration. Despite the mundane sounding name, this is actually exciting news for those that use AWS Systems Manager to manage fleets of EC2 instances. The functionality of this latest release is that users now have the ability to enable AWS Systems Manager access by default across all existing and new EC2 instances in an account.

In this article we’ll take a deeper dive into this latest feature to explain the benefits and drawbacks of enabling Default Host Management Configuration. We’ll also share some of our learnings on how to get this feature up and running in your own account as we encountered a few hiccups when playing in the console.

Default Host Management Configuration (DHMC): An Overview

DHMC allows AWS users to enable AWS Systems Manager by default across all existing and new EC2 instances in an account. Note that this feature is region-specific, so to use this feature ensure it is enabled in all applicable regions you require.

If you’re not familiar with AWS Systems Manager, it allows users to organise, monitor and automate management tasks on their AWS resources.

For EC2 instances, Systems manager has some useful features:

• Fleet Manager - monitor and manage instances
• Sessions Manager - create an SSH-like session to an instance
• Patch Manager - automates patching of instances

What are the benefits

No Need to Open Ports on EC2 Security Groups for SSH Access

In order to access an EC2 instance directly via SSH, we’d normally need to open up port 22 for inbound requests on the associated security group. When using SSM’s Session Manager, we gain the ability to close off port 22 access and allow AWS to handle SSH sessions via the SSM Agent.

Avoid Giving EC2 Instances an SSH Key Pair

In the same vein, rather than giving your instance a dedicated keypair for access via SSH (either directly, or through a bastion/jump host) we can hand off responsibility to Session Manager and the SSM Agent.

A Simple Way to Access to SSH Audit Trails for all Managed Instances

Any time a session is created via the SSM Session Manager, an audit log is created. This is a free, managed way to audit your SSH access logs. As it stands, this doesn’t provide detailed information on the contents of a session, but it paints a picture of who’s accessing instances and for how long. This can (and should) still be used in conjunction with an instance-level ssh.log audit trail.

One Click SSH Session from the Console/CLI

I’m not entirely convinced this should just be in the ‘benefits’ pile, but it is pretty neat. By using Session Manager we now have a very convenient way to create SSH-like sessions via the console or CLI.

There’s obvious dangers that come bundled with this ease of access. One compromised laptop or AWS account now has a very simple route into EC2 instances without having to authenticate with a Bastion or get around a WAF I.P. rule.

What are the Drawbacks

SSM Agent Must be on the Latest Version

We’ll discuss this more thoroughly in the section below, but it’s worth mentioning that the Linux 2 AMI does not ship with a version of SSM Agent that is compatible with DHCM. It’s possible that a future version of the AMI will, but for now you’d need to tag an update script into the user data or SSH onto the box and manually update it.

Enabling DHMC Will Opt in All EC2 Instances in a Region

This is a biggie. With the recent push for cloud services to include one click configurations, it’s a no-brainer SSM to enable fleet management for existing/new instances while bypassing the manual role configuration that’s normally required.

Having said that, this is still an exceptionally open ended policy. All existing instances. All new instances. This has the potential to be a nasty security hole unless you’re extremely confident in your user access management and understanding of all existing/future infrastructure that could be hosted in your EC2 fleet.

Fire and Forget Approach to SSH Access

I think the key takeaway here is that DHMC can easily be enabled and forgotten about. If you have a personal account or a sandbox account that doesn’t host any production infrastructure then it’s easy to imagine DHMC being quite useful. But for most people using AWS that’s not going to be the case.

Allowing SSH access to all future EC2 instances has the potential to be dangerous. It’s easy to imagine a scenario where an engineer has enabled DHMC then left a team without imparting the knowledge that all future instances can now be directly connected to through Systems Manager.

Implementation Watch-Outs

We spent some time testing out Default Host Management Configuration in a sandbox account, and although the process of enabling this feature was straightforward we did encounter some issues. If you want to start using DHMC in your own account, make sure you take into account the following:

Instances Must be Using the Latest SSM Agent

In order for Default Host Management Configuration to work, your instances need to be running the SSM Agent version 3.2.582.0 or later.

However, when creating a new instance we found that not all AMIs have the latest version of SSM agent installed. The AWS docs on checking your SSM agent version number instruct you to SSH into your instance, but this would mean opening up port 22 on your EC2 instance and negates some of the benefits of using DHMC.

To combat this issue when launching a new EC2 instance, you could include the following script in your instance's user data to install the latest version of SSM agent when the instance first boots.

#!/bin/bash 

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

Instances Must be Using Metadata Version 2

Another potential watch-out is that instances must be using Instance Metadata Service Version 2 for Default Host Management Configuration to work.

For all newly created instances this shouldn’t be an issue as both Version 1 and Version 2 are enabled by default. If you want to double-check this when launching an instance, the option for this can be found under “Advanced Details → Metadata Version.”

If your instance does not use Metadata Version 2, then check out the AWS doc on transitioning your instances.

Security Considerations - Should we be Using This?

If you’ve read this far and are unsure whether to enable DHMC, then no, don’t enable it. When fiddling with configuration that can affect your SSH access routes it’s always wiser to enforce the Principle of Least Privilege and lean toward the cautious approach.

The bottom line here is that DHCM is an account-wide (albeit in one region) setting that is going to impact both existing and new infrastructure when enabled with its default settings.

Is DHCM more secure than opening up port 22 and assigning all instances a key pair? Yes.

Is DHCM more secure than having a single bastion/jump host? No.

This configuration option is intended to be used in conjunction with sensible AWS user roles. If anyone with access to Session Manager can now open an SSH connection to every instance in an account then it’s more important than before to ensure that Session Manager access is restricted to the smallest possible subset of users.

For us, the potential security drawbacks outweigh the value gained from using DHMC. There is perhaps a use-case here for non-production personal accounts or sandbox accounts for users that are really confident in their account governance policies and are willing to accept some risk in exchange for ease of fleet management.

About Clear Horizon Digital - Our Cloud Offering

We’re a Leeds based Software Consultancy that offers a broad range of AWS services, from architecture and design to maintenance and support. Read more about our Cloud Services.