Risk in AWS SSM Port Forwarding

Aug 03, 23

I recently stumbled on a suprising AWS Systems Manager Session Manager (SSM) default that can introduce risk, especially for customers using SSM’s Port Forwarding features.

tl;dr make sure you’re familiar with ssm:SessionDocumentAccessCheck if you’re using SSM for Port Forwarding.

Setting the Stage

Here’s the minimal policy for an IAM principal to connect to a given EC2 instance.

Policy A:

  statement {
    actions = [
      "ssm:TerminateSession",
      "ssm:StartSession",
    ]
    resources = [
      "arn:aws:ec2:*:*:instance/i-0000000000000",
      "arn:aws:ssm:us-east-1:*:session/*"
      ]
  }

A more realistic (explicit, least privileged, and usable) policy might look like this:

Policy B:

  statement {
    actions = [
      "ssm:StartSession",
    ]
    resources = [
      "arn:aws:ec2:us-east-1:*:instance/i-0000000000000",
      "arn:aws:ssm:us-east-1:*:document/SSM-SessionManagerRunShell",
    ]
  }
  statement {
    actions = [
      "ssm:DescribeSessions",
      "ssm:GetConnectionStatus",
      "ssm:DescribeInstanceInformation",
      "ssm:DescribeInstanceProperties",
      "ec2:DescribeInstances",
    ]
    resources = ["*"]
  }
  statement {
    actions = [
      "ssm:TerminateSession",
      "ssm:ResumeSession",
    ]
    resources = ["*"]
    condition {
      test     = "StringLike"
      variable = "ssm:resourceTag/aws:ssmmessages:session-id"
      values   = ["${aws:userid}"]
    }
  }

What’s the difference?

  • Policy B limits you to only terminating your own session
  • Policy B allows session resumption
  • Policy B adds convenience read access to ssm and ec2 data
  • Policy B explicitly spells out the SSM-SessionManagerRunShell document, that otherwise is the default

Test driving Policy B

Let’s test getting a shell using SSM. Assuming everything else is set up, it’ll look something like:

$ aws ssm start-session --target i-0000000000000   
Starting session with SessionId: EX-SESS-07a16060613c408b5

We can also make the default document explicit:

$ aws ssm start-session --target i-0000000000000 --document-name SSM-SessionManagerRunShell   
Starting session with SessionId: EX-SESS-08b26060613c408b5

Now, let’s say I want to port forward instead - as in Shipping RDS IAM Authentication (with a bastion host & SSM). We can check out the documentation, and see “Starting a session (port forwarding to remote host)” uses the document AWS-StartPortForwardingSessionToRemoteHost.

Let’s start by trying with Policy B:

$ aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["mydb.example.us-east-2.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["3306"]}'  
An error occurred (AccessDeniedException) when calling the StartSession operation: User: arn:aws:sts::{accountId}:assumed-role/{role_name}/{session_name} is not authorized to perform: ssm:StartSession on resource: arn:aws:ec2:us-west-2:{accountId}:instance/i-0000000000000 because no identity-based policy allows the ssm:StartSession action

Okay - so we get an AccessDenied error, which makes sense - Policy B doesn’t grant access to the document we’re attempting to use. Let’s make a small tweak, swapping in the AWS-StartPortForwardingSessionToRemoteHost document:

Policy C:

  statement {
    actions = [
      "ssm:StartSession",
    ]
    resources = [
      "arn:aws:ec2:us-east-1:*:instance/i-0000000000000",
      "arn:aws:ssm:us-east-1:*:document/AWS-StartPortForwardingSessionToRemoteHost",
    ]
  }
  statement {
    actions = [
      "ssm:DescribeSessions",
      "ssm:GetConnectionStatus",
      "ssm:DescribeInstanceInformation",
      "ssm:DescribeInstanceProperties",
      "ec2:DescribeInstances",
    ]
    resources = ["*"]
  }
  statement {
    actions = [
      "ssm:TerminateSession",
      "ssm:ResumeSession",
    ]
    resources = ["*"]
    condition {
      test     = "StringLike"
      variable = "ssm:resourceTag/aws:ssmmessages:session-id"
      values   = ["${aws:userid}"]
    }
  }

We can see the port forwarding is successful after this change:

$ aws ssm start-session
–target instance-id
–document-name AWS-StartPortForwardingSessionToRemoteHost
–parameters ‘{“host”:[“mydb.example.us-east-2.rds.amazonaws.com”],”portNumber”:[“3306”], “localPortNumber”:[“3306”]}’ Starting session with SessionId: EX-SESS-08b26060613c408b5 Port 8443 opened for sessionId EX-SESS-08b26060613c408b5. Waiting for connections…

Now, let’s try the inverse, and call the original SSM-SessionManagerRunShell document with our new Policy C:

aws ssm start-session –target i-0000000000000 –document-name SSM-SessionManagerRunShell
Starting session with SessionId: EX-SESS-08b26060613c408b5

Wait, why did that work?

The magic SSM-SessionManagerRunShell document

The basic premise of AWS IAM is:

  • By default, all requests are implicitly denied with the exception of the AWS account root user, which has full access.
  • An explicit allow in an identity-based or resource-based policy overrides this default.
  • If a permissions boundary, Organizations SCP, or session policy is present, it might override the allow with an implicit deny.
  • An explicit deny in any policy overrides any allows.

But, if you go back to Policy A you can see that despite never granting access to SSM-SessionManagerRunShell, start-session still worked.

By default, when a user in your account has been granted permission to start sessions by their AWS Identity and Access Management (IAM) policy, they are also granted access to the SSM-SessionManagerRunShell SSM document

This makes sense in some ways, without this default start-session wouldn’t be a functional permission in isolation. But this special case, where access is not implicitly denied, is easily missed when using SSM for more narrow purposes using scoped documents - prominently for Port Forwarding.

What should you do?

This consideration is documented by AWS under “Enforce a session document permission check for the AWS CLI”.

In short, if you want to use SSM for Port Forwarding without implicitly granting shell access, you need to add the following condition:

"Condition": {
    "BoolIfExists": {
        "ssm:SessionDocumentAccessCheck": "true"
    }
}

If the SessionDocumentAccessCheck condition element is set to true, and you specify a document name in the Resource, you must provide the specified document name when starting a session. If you provide a different document name when starting a session, the request fails.

It’s hard to know whether customers are consistently using this (documented!) condition when setting up Port Forwarding with SSM - a quick Github search finds a few instances of the “vulnerable” pattern. In the end, this led to an advisory: GHSA-q4pp-j36h-3gqg

Third party guides are hit-or-miss, with many eliding the implict access, and others (like Sym) calling out:

Note that by default, users can always use the SSM-SessionManagerRunShell document even if you don’t give that permission explicitly. You can turn this behavior off if you want to manage all document access explicitly.