AWS CodeDeploy

CodeDeploy orchestrates deployments to EC2, on-premises servers, Lambda and ECS. What it adds over copying files or updating a service is controlled traffic shifting with automatic rollback. A canary that watches an alarm and reverts itself if the alarm fires.

The three compute platforms behave quite differently, which is worth knowing before reading further: EC2 deployments run scripts on the instances, while Lambda and ECS deployments shift traffic between versions and run no scripts on any host.

The appspec file

For EC2, appspec.yml at the root of your bundle describes what to copy and which scripts to run:

version: 0.0
os: linux
files:
  - source: /
    destination: /var/www/app
    overwrite: true
permissions:
  - object: /var/www/app
    owner: app
    group: app
    mode: 755
hooks:
  ApplicationStop:
    - location: scripts/stop.sh
      timeout: 60
      runas: root
  BeforeInstall:
    - location: scripts/backup.sh
      timeout: 120
  AfterInstall:
    - location: scripts/install-deps.sh
      timeout: 300
  ApplicationStart:
    - location: scripts/start.sh
      timeout: 60
  ValidateService:
    - location: scripts/healthcheck.sh
      timeout: 120

For Lambda and ECS it is JSON or YAML naming versions rather than scripts:

version: 0.0
Resources:
  - orders-service:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: "arn:aws:ecs:eu-west-1:123456789012:task-definition/orders:42"
        LoadBalancerInfo:
          ContainerName: "app"
          ContainerPort: 8080
Hooks:
  - BeforeAllowTraffic: "arn:aws:lambda:eu-west-1:123456789012:function:smoke-test"
  - AfterAllowTraffic: "arn:aws:lambda:eu-west-1:123456789012:function:verify"

ApplicationStop runs the old revision's script

The lifecycle detail that causes the most confusion. ApplicationStop executes from the previously deployed revision, not the one you are deploying, because the old version is what is running and needs stopping.

So a broken ApplicationStop script blocks every future deployment, and fixing it in the new revision does not help, since the old one still runs. Recovery means either deploying with --ignore-application-stop-failures or removing the agent's stored revision on the instance:

aws deploy create-deployment --application-name orders \
  --deployment-group-name prod \
  --ignore-application-stop-failures \
  --s3-location bucket=my-artifacts,key=orders-v43.zip,bundleType=zip

Keep ApplicationStop scripts trivial and idempotent, systemctl stop app || true rather than anything that can fail.

Deployment configurations

EC2 in-place updates instances in batches:

EC2 blue-green provisions a replacement fleet, shifts the load balancer, then terminates the original. Safer, and it needs double capacity briefly.

Lambda and ECS shift traffic between versions:

A canary is only useful if something is watching, which is the next section.

Automatic rollback is the point

aws deploy create-deployment-group \
  --application-name orders --deployment-group-name prod \
  --service-role-arn "$CODEDEPLOY_ROLE_ARN" \
  --deployment-config-name CodeDeployDefault.ECSCanary10Percent5Minutes \
  --auto-rollback-configuration 'enabled=true,events=DEPLOYMENT_FAILURE,DEPLOYMENT_STOP_ON_ALARM' \
  --alarm-configuration 'enabled=true,alarms=[{name=orders-5xx-rate},{name=orders-p99-latency}]' \
  --blue-green-deployment-configuration '{
    "terminateBlueInstancesOnDeploymentSuccess": {"action":"TERMINATE","terminationWaitTimeInMinutes":30},
    "deploymentReadyOption": {"actionOnTimeout":"CONTINUE_DEPLOYMENT"}
  }'

DEPLOYMENT_STOP_ON_ALARM with a real alarm is what makes this worth using. Without it, a canary sends 10% of traffic to a broken version for five minutes and then confidently promotes it to 100%.

Pick alarms that reflect user experience, 5xx rate, p99 latency, a business metric, not CPU. A deployment that raises the error rate should roll back; one that raises CPU by 10% probably should not.

terminationWaitTimeInMinutes: 30 keeps the old fleet alive after the switch, so a manual rollback is immediate rather than a redeploy.

Lifecycle hooks for validation

Hooks are where a deployment becomes more than a file copy. On Lambda and ECS they are Lambda functions that must report back:

import boto3
cd = boto3.client('codedeploy')

def handler(event, context):
    deployment_id = event['DeploymentId']
    hook_id = event['LifecycleEventHookExecutionId']
    try:
        assert smoke_test_passes()
        status = 'Succeeded'
    except Exception:
        status = 'Failed'
    cd.put_lifecycle_event_hook_execution_status(
        deploymentId=deployment_id,
        lifecycleEventHookExecutionId=hook_id,
        status=status,
    )

The function must call put_lifecycle_event_hook_execution_status. A hook that returns without reporting leaves the deployment waiting until it times out. An hour by default, which looks like a hang.

BeforeAllowTraffic runs against the new version before any real traffic reaches it, which is the right place for a smoke test. AfterAllowTraffic runs once traffic is shifted.

The two EC2 prerequisites

Almost every EC2 CodeDeploy failure is one of these.

The agent must be installed and running:

sudo systemctl status codedeploy-agent
sudo tail -100 /var/log/aws/codedeploy-agent/codedeploy-agent.log

An instance without a running agent is simply skipped or fails with no useful message. Install it via user data, bake it into the AMI, or deploy it with State Manager.

The instance profile needs S3 read access to the artefact bucket. The agent pulls the bundle itself using the instance role, so a missing s3:GetObject produces a download failure that reads like a CodeDeploy problem.

Deployment groups target instances by tag or Auto Scaling group:

aws deploy update-deployment-group \
  --application-name orders --current-deployment-group-name prod \
  --ec2-tag-filters 'Key=Environment,Value=production,Type=KEY_AND_VALUE' \
  --auto-scaling-groups web-asg

Targeting an Auto Scaling group means instances launched later receive the current revision automatically, which is what you want.

Diagnosing a failure

aws deploy get-deployment --deployment-id d-ABCDEF123 \
  --query 'deploymentInfo.[status,errorInformation]'

aws deploy list-deployment-targets --deployment-id d-ABCDEF123

aws deploy get-deployment-target --deployment-id d-ABCDEF123 \
  --target-id "i-0123456789abcdef0" \
  --query 'deploymentTarget.instanceTarget.lifecycleEvents[?status==`Failed`]'

That last query names the failing lifecycle event and its diagnostic output, including the script's exit code and last lines of stdout, which is the fastest route to the cause. For EC2, follow up in the agent log on the instance.

Do you need CodeDeploy?

Worth asking honestly.

For ECS, the built-in rolling update handles most deployments without CodeDeploy. Add CodeDeploy when you want genuine blue-green with traffic shifting, canary percentages and automatic rollback on alarms.

For Lambda, SAM's DeploymentPreference configures CodeDeploy for you in five lines, which is much easier than assembling it directly.

For EC2, CodeDeploy is a reasonable choice, though an immutable approach using an Auto Scaling group instance refresh with a new AMI is often simpler and avoids maintaining deployment scripts on instances entirely.

Common errors

The deployment failed because a specified file already exists at this location:
/var/www/app/index.html

CodeDeploy will not overwrite files it did not place. Set overwrite: true in appspec.yml, or clean the directory in BeforeInstall.

Script at specified location: scripts/start.sh failed with exit code 1

Read the diagnostic output from get-deployment-target. Also check the script is executable and has LF line endings. A CRLF script fails with a confusing "not found".

The overall deployment failed because too many individual instances failed

Exceeded the deployment group's minimum healthy hosts. Look at one failing instance's lifecycle events.

Deployment hangs, then times out. A Lambda hook that never reported its status, or an ApplicationStop script waiting on something.

Quick reference

---

Key takeaways

• ApplicationStop runs from the previous revision, so a broken one blocks all future deployments.
• Keep stop scripts trivial and idempotent.
• A canary without an alarm is theatre. Set DEPLOYMENT_STOP_ON_ALARM with real alarms.
• Alarm on user-facing metrics: error rate and latency, not CPU.
• Lifecycle hook Lambdas must call put_lifecycle_event_hook_execution_status or the deployment hangs.
• EC2 needs the agent running and S3 read on the instance profile.
• Target Auto Scaling groups so new instances get the current revision.
• get-deployment-target gives the actual script output: start there.
• terminationWaitTimeInMinutes keeps rollback instant after a blue-green switch.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between in-place and blue-green deployment in CodeDeploy?

In-place updates the application on existing instances, in batches you configure, no extra capacity needed, but instances are briefly out of service and rollback means deploying the previous revision again. Blue-green provisions a replacement set of instances, shifts the load balancer to them once healthy, and terminates the original after a wait period, so rollback is switching back, which is near-instant. Blue-green costs double capacity briefly and is the safer choice for anything user-facing.

Why does my deployment keep failing at ApplicationStop?

Because ApplicationStop runs from the previously deployed revision, not the one being deployed. Its job is to stop what is currently running. A broken stop script therefore blocks every subsequent deployment, and fixing it in the new revision changes nothing until you get past it once. Deploy with --ignore-application-stop-failures to break the cycle. Keep these scripts trivial and idempotent, such as systemctl stop app || true, precisely because of this property.

How do I set up automatic rollback?

Two parts, and both are required. Set --auto-rollback-configuration with the events that should trigger it, including DEPLOYMENT_STOP_ON_ALARM, and set --alarm-configuration listing CloudWatch alarms to watch. Without the alarms, a canary shifts traffic, waits its interval and promotes regardless of what happened. Choose alarms reflecting user experience, error rate, p99 latency, a conversion metric, rather than CPU, which moves for reasons unrelated to whether the deployment is good.

Why is my CodeDeploy deployment stuck and eventually timing out?

On Lambda or ECS, almost always a lifecycle hook function that never reported its result. A hook Lambda must call put_lifecycle_event_hook_execution_status with Succeeded or Failed; returning without doing so leaves CodeDeploy waiting until the hook times out, which defaults to an hour and looks like a hang. Make sure the call happens in the failure path too. On EC2, the equivalent cause is a script waiting on something that never completes.

What does the CodeDeploy agent need on EC2?

Two things, and they account for most EC2 failures. The agent itself must be installed and running, check systemctl status codedeploy-agent and its log under /var/log/aws/codedeploy-agent/, and the instance profile must allow s3:GetObject on the artefact bucket, because the agent downloads the bundle itself using the instance role. Install the agent through user data, a baked AMI, or Systems Manager State Manager so newly launched instances always have it.

Do I need CodeDeploy for ECS?

Not for ordinary deployments. ECS's built-in rolling update replaces tasks gradually and handles most cases. CodeDeploy earns its place when you want genuine blue-green with two target groups, canary or linear traffic shifting by percentage, lifecycle hooks running validation before traffic arrives, and automatic rollback driven by CloudWatch alarms. For Lambda, SAM's DeploymentPreference configures the same machinery in a few lines and is much easier than wiring it up directly.