AWS Systems Manager

Systems Manager is a collection of loosely related tools for operating servers: running commands across a fleet, patching, storing configuration, maintaining desired state, and connecting to instances without SSH.

Everything depends on one thing working first, and it is where almost all Systems Manager time is spent.

Getting instances to appear

aws ssm describe-instance-information \
  --query 'InstanceInformationList[].[InstanceId,PingStatus,AgentVersion,PlatformName]' \
  --output table

An empty list means the prerequisites are not met. There are exactly three, and they are the answer to nearly every "SSM does not see my instance" question:

1. The agent must be running. Amazon Linux 2023, Amazon Linux 2, Ubuntu 16.04+ and Windows Server AMIs include it. Other images need it installed.

2. The instance profile needs AmazonSSMManagedInstanceCore:

aws iam attach-role-policy --role-name EC2InstanceRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

This is the most common cause. An instance with no instance profile at all, or one missing this policy, never registers and reports nothing.

3. Network access to the SSM endpoints. The agent makes outbound connections. It never receives inbound traffic, so no security group ingress rule is needed. A private subnet therefore needs either a NAT gateway or VPC endpoints:

for svc in ssm ssmmessages ec2messages; do
  aws ec2 create-vpc-endpoint --vpc-endpoint-type Interface \
    --vpc-id vpc-0abc123 --service-name "com.amazonaws.eu-west-1.$svc" \
    --subnet-ids subnet-0a1b2c3d --security-group-ids sg-0123456789abcdef0 \
    --private-dns-enabled
done

All three endpoints are required. ssmmessages handles Session Manager and ec2messages handles Run Command, so omitting one produces a partially working setup. The instance appears in the console but sessions fail, which is a confusing place to end up. The endpoint security group must allow inbound 443 from the instances.

If an instance still does not appear, read the agent log:

sudo tail -50 /var/log/amazon/ssm/amazon-ssm-agent.log

Run Command

Execute a command across a fleet without SSH:

aws ssm send-command \
  --document-name AWS-RunShellScript \
  --targets 'Key=tag:Environment,Values=production' \
  --parameters 'commands=["systemctl restart nginx","systemctl is-active nginx"]' \
  --max-concurrency 10% --max-errors 2 \
  --comment "restart nginx after config change" \
  --output-s3-bucket-name my-ssm-output

aws ssm list-command-invocations --command-id "$CMD_ID" --details \
  --query 'CommandInvocations[].[InstanceId,Status,CommandPlugins[0].Output]' --output table

Two parameters make this safe on a real fleet. --max-concurrency 10% runs on a tenth of the targets at a time rather than all at once. --max-errors 2 halts the whole operation after two failures, so a broken command does not roll across production.

Targeting by tag rather than instance ID means the command automatically covers instances that did not exist when you wrote it.

Every invocation is recorded in CloudTrail with the calling identity, which is a real advantage over SSH: there is an audit trail of who ran what, without needing to collect shell history.

Parameter Store

Configuration and secrets, with a hierarchy:

aws ssm put-parameter --name /prod/orders/db-host \
  --value "orders.abc123.eu-west-1.rds.amazonaws.com" --type String

aws ssm put-parameter --name /prod/orders/db-password \
  --value 'correct-horse-battery-staple' --type SecureString --key-id alias/orders

aws ssm get-parameters-by-path --path /prod/orders --recursive --with-decryption \
  --query 'Parameters[].[Name,Value]' --output table

--with-decryption is required for SecureString values, and the caller needs kms:Decrypt on the key. A missing KMS permission here is a frequent and confusing failure, because the parameter read itself is authorised.

Use a consistent hierarchy (/env/app/key), since IAM policies can then scope to a path prefix:

{
  "Effect": "Allow",
  "Action": ["ssm:GetParameter", "ssm:GetParametersByPath"],
  "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/prod/orders/*"
}

Parameter Store or Secrets Manager

Parameter Store for configuration and for secrets you rotate manually. Secrets Manager when you want automatic rotation. Particularly for RDS, where the integration handles the database side too. Paying per secret for values that never rotate is waste; managing manual rotation of a database password is worse.

Patch Manager

aws ssm create-patch-baseline --name prod-linux \
  --operating-system AMAZON_LINUX_2023 \
  --approval-rules 'PatchRules=[{
    PatchFilterGroup={PatchFilters=[{Key=CLASSIFICATION,Values=[Security,Bugfix]},{Key=SEVERITY,Values=[Critical,Important]}]},
    ApproveAfterDays=7,
    ComplianceLevel=CRITICAL
  }]'

ApproveAfterDays=7 is the setting worth understanding: patches are approved automatically a week after release, which gives time for a bad patch to be withdrawn while keeping you current. Zero days means installing everything immediately; never approving means an unpatched fleet.

Patching runs in a maintenance window:

aws ssm create-maintenance-window --name prod-patching \
  --schedule "cron(0 3 ? * SUN *)" --duration 4 --cutoff 1 \
  --schedule-timezone "Europe/London" --allow-unassociated-targets

--cutoff 1 stops new tasks starting in the final hour, so long-running work does not get cut off at the window's close.

aws ssm describe-instance-patch-states-for-patch-group --patch-group prod \
  --query 'InstancePatchStates[].[InstanceId,InstalledCount,MissingCount,FailedCount]' \
  --output table

Scan-only mode (Operation=Scan) reports compliance without installing anything, which is the right way to start.

State Manager

Where Run Command is a one-off, State Manager enforces continuously:

aws ssm create-association --name AWS-RunShellScript \
  --targets 'Key=tag:Environment,Values=production' \
  --parameters 'commands=["systemctl enable --now amazon-cloudwatch-agent"]' \
  --schedule-expression "rate(30 minutes)" \
  --association-name ensure-cloudwatch-agent \
  --compliance-severity MEDIUM

That reapplies every thirty minutes, so an instance that drifts, or a new one that launches, converges without anyone noticing. It is lightweight configuration management for teams not running Ansible or Puppet, and it does not need an agent beyond the one already there.

Automation

Multi-step runbooks combining AWS API calls, commands and approvals:

aws ssm start-automation-execution \
  --document-name AWS-RestartEC2Instance \
  --parameters 'InstanceId=i-0123456789abcdef0'

AWS provides many pre-built documents, and Automation is what Config auto-remediation invokes. Custom runbooks can include an aws:approve step that pauses for a human, which turns a risky manual procedure into a repeatable one with an audit trail.

Inventory and Compliance

aws ssm create-association --name AWS-GatherSoftwareInventory \
  --targets 'Key=InstanceIds,Values=*' \
  --schedule-expression "rate(1 day)"

Inventory collects installed applications, network configuration, Windows updates and custom attributes across the fleet, queryable through Resource Data Sync into S3 and Athena. "Which instances have this vulnerable package" becomes a query rather than a broadcast command.

Common errors

An error occurred (InvalidInstanceId) when calling the SendCommand operation

The instance is not registered with SSM. Check the three prerequisites.

Parameter /prod/orders/db-password not found

Wrong region, wrong path, or no ssm:GetParameter permission on that path. A denied read looks identical to a missing parameter.

AccessDeniedException: ... is not authorized to perform: kms:Decrypt

Reading a SecureString needs KMS permission as well as SSM permission.

Instance in Fleet Manager but sessions fail. The ssmmessages endpoint is missing.

Quick reference

---

Key takeaways

• Three prerequisites: agent running, AmazonSSMManagedInstanceCore on the instance profile, network to the endpoints.
• All three VPC endpoints: ssm, ssmmessages, ec2messages, or you get a half-working setup.
• The agent connects outbound. No inbound security group rule is needed.
• Target by tag, not instance ID, so new instances are covered automatically.
• Use --max-concurrency and --max-errors on any fleet-wide command.
• Parameter Store for configuration, Secrets Manager for rotation.
• Reading a SecureString needs kms:Decrypt as well as SSM permissions.
• ApproveAfterDays balances staying current against installing a bad patch.
• State Manager reapplies configuration, so drift and new instances self-correct.

---

Check your understanding

---

Frequently Asked Questions

Why don't my instances appear in Systems Manager?

Three prerequisites, and one of them is missing. The SSM agent must be running. It is preinstalled on Amazon Linux, recent Ubuntu and Windows AMIs but not everything. The instance profile must include AmazonSSMManagedInstanceCore, which is the most common omission. And the instance needs outbound network access to the ssm, ssmmessages and ec2messages endpoints, via NAT gateway, internet gateway or VPC endpoints. Check /var/log/amazon/ssm/amazon-ssm-agent.log on the instance; it names the failure directly.

Which VPC endpoints does Systems Manager need?

Three interface endpoints: com.amazonaws.<region>.ssm, .ssmmessages and .ec2messages. Creating only the first gives a partially working setup where instances register and appear in the console while Session Manager and Run Command fail. A confusing state to debug. ssmmessages carries Session Manager traffic and ec2messages carries Run Command. The endpoint security group must allow inbound 443 from the instance security group, and private DNS should be enabled.

What is the difference between Parameter Store and Secrets Manager?

Parameter Store is free at the standard tier, stores configuration and encrypted SecureString values, and has no rotation capability. Secrets Manager costs roughly 40 cents per secret per month and adds automatic rotation through Lambda, cross-account access via resource policies, random password generation and a larger size limit. Use Parameter Store for configuration and for secrets you rotate manually, and Secrets Manager where automatic rotation matters. Particularly for RDS, where the built-in integration updates the database and the stored secret together.

How do I run a command on many instances safely?

Target by tag rather than instance ID, so new instances are included automatically, and set --max-concurrency to a percentage such as 10% so the command rolls out gradually rather than hitting the whole fleet at once. Set --max-errors to a small number so the operation stops after a few failures instead of propagating a broken command everywhere. Send output to S3 or CloudWatch Logs, since inline output is truncated. Every invocation is recorded in CloudTrail with the calling identity.

What does State Manager do that Run Command does not?

Run Command executes once; State Manager enforces continuously. An association reapplies its document on a schedule, every thirty minutes, daily, whatever you set, so an instance whose configuration drifts is corrected, and a newly launched instance matching the target tags converges without anyone doing anything. It is lightweight configuration management using the agent you already have, which is often enough for teams not running Ansible or Puppet.

How should I configure Patch Manager?

Create a patch baseline filtered to the classifications and severities you care about, with ApproveAfterDays set to around seven. Patches are then adopted automatically a week after release, which leaves time for a bad one to be withdrawn while keeping the fleet current. Run patching inside a maintenance window with a cutoff so long tasks are not truncated at the end. Start in scan-only mode to see compliance without installing anything, and check describe-instance-patch-states-for-patch-group for the resulting picture.