The ECS scheduler evaluated every registered instance and none satisfied the task's requirements. It helpfully tells you what the closest one was short of.
aws ecs describe-services --cluster prod --services api \
--query 'services[0].events[0:5].message' --output text
| Closest-match reason | Cause |
|---|---|
insufficient memory available | No instance has enough free memory |
insufficient CPU units available | Same, for CPU |
is already using a port required by your task | Static host port conflict |
has insufficient ENI capacity | awsvpc mode, ENI limit reached |
| No closest match named at all | No instances registered, or constraints exclude all |
Memory and CPU
The scheduler reserves against declared values, not actual usage. An instance with 4 GB free memory in reality but 4 GB already reserved by task definitions cannot take another task.
aws ecs list-container-instances --cluster prod --query 'containerInstanceArns' --output text \
| xargs aws ecs describe-container-instances --cluster prod --container-instances \
--query 'containerInstances[].{id:ec2InstanceId,mem:remainingResources[?name==`MEMORY`].integerValue|[0],cpu:remainingResources[?name==`CPU`].integerValue|[0],tasks:runningTasksCount}' \
--output table
| id | mem | cpu | tasks |
| i-0abc123 | 412 | 256 | 3 |
412 MB remaining against a task asking for 512 MB. That is the whole problem.
Two levers in the task definition:
{
"memory": 512, // hard limit: the container is killed above this
"memoryReservation": 256, // soft: what the scheduler reserves
"cpu": 256
}
memoryReservation is what placement uses. Setting only memory means the scheduler reserves the hard limit, which is usually far more than the container needs and wastes capacity. Setting both, with a realistic reservation and a headroom limit, packs instances much better.
Note the instance also reserves memory for the ECS agent and the operating system, so a 2 GB instance does not offer 2048 MB.
Port conflicts
The closest matching container-instance 8f2a1c is already using a port required by your task.
In bridge mode with a fixed hostPort, only one task per instance can bind it.
"portMappings": [{ "containerPort": 8080, "hostPort": 8080 }]
Use dynamic port mapping:
"portMappings": [{ "containerPort": 8080, "hostPort": 0 }]
hostPort: 0 assigns an ephemeral port, and an ALB target group registers whichever port each task received. Several tasks per instance then work.
Better, use awsvpc:
{
"networkMode": "awsvpc",
"portMappings": [{ "containerPort": 8080 }]
}
Each task gets its own ENI and its own IP, so there is no port sharing at all. It is also required for Fargate and for security groups per task.
ENI limits in awsvpc mode
awsvpc trades the port problem for an ENI one. Each task consumes an ENI, and every instance type has a limit:
aws ec2 describe-instance-types --instance-types t3.medium \
--query 'InstanceTypes[].NetworkInfo.{max:MaximumNetworkInterfaces,ipv4:Ipv4AddressesPerInterface}'
A t3.medium allows 3 ENIs, one of which is the primary, so two tasks regardless of how much CPU and memory are free. That surprises people badly.
ENI trunking raises it substantially:
aws ecs put-account-setting --name awsvpcTrunking --value enabled
Instances launched after enabling it get far higher limits. Existing instances must be replaced.
No closest match at all
When the message names no instance, nothing is registered or every instance is excluded:
aws ecs describe-clusters --clusters prod \
--query 'clusters[].{active:registeredContainerInstancesCount,running:runningTasksCount}'
Zero registered instances means the ECS agent is not running or the instances lack the cluster configuration. On an ECS-optimised AMI:
echo "ECS_CLUSTER=prod" >> /etc/ecs/ecs.config
Placement constraints can also exclude everything:
"placementConstraints": [
{ "type": "memberOf", "expression": "attribute:ecs.instance-type =~ m5.*" }
]
aws ecs list-attributes --cluster prod --target-type container-instance
Let a capacity provider scale for you
The durable fix is not to add instances by hand:
aws ecs create-capacity-provider \
--name prod-asg \
--auto-scaling-group-provider "autoScalingGroupArn=$ASG_ARN,managedScaling={status=ENABLED,targetCapacity=80},managedTerminationProtection=ENABLED"
With managed scaling, ECS scales the ASG when tasks cannot be placed, so this error becomes a brief delay rather than a stuck deployment. managedTerminationProtection stops the ASG terminating an instance that still has tasks on it.
Fargate removes the whole class of problem, at a higher per-task cost.
Deployments need headroom
A rolling deployment with maximumPercent: 200 needs capacity for twice the tasks briefly. On a tightly packed cluster that fails to place the new tasks and the deployment stalls.
aws ecs update-service --cluster prod --service api \
--deployment-configuration "maximumPercent=150,minimumHealthyPercent=100"
Lowering maximumPercent reduces the headroom needed, at the cost of a slower rollout.
A checklist
describe-services --query 'services[0].events[0:5].message'and read the closest match.- Memory or CPU → compare
remainingResourcesagainst the task's reservation. - Use
memoryReservationfor placement andmemoryas the hard limit. - Port conflict →
hostPort: 0, or move toawsvpc. awsvpc→ check the instance type's ENI limit; enableawsvpcTrunking.- No closest match → check
registeredContainerInstancesCountand placement constraints. - Add a capacity provider with managed scaling so this self-resolves.
- Deployment stalls only → lower
maximumPercent.
Frequently Asked Questions
Why does ECS say there is no memory when the instance looks idle?
Because the scheduler reserves against declared values in the task definition, not measured usage. An instance running three tasks that each declared 1 GB has 3 GB reserved even if all three are using 100 MB. remainingResources in describe-container-instances is what the scheduler sees, and it is the number to compare against your task's reservation. Using memoryReservation for the soft reservation and memory for the hard limit lets you pack instances much more densely.
What is the difference between memory and memoryReservation?
memory is a hard limit: the container is killed if it exceeds it. memoryReservation is a soft reservation and is what the scheduler uses for placement, with the container allowed to burst above it when the instance has spare memory. Setting only memory means the scheduler reserves the full hard limit, which usually wastes a great deal of capacity. Set memoryReservation to realistic usage and memory to a safe ceiling.
How do I run several copies of the same task on one instance?
Either set hostPort: 0 in bridge mode, which assigns an ephemeral port per task and works with an ALB's dynamic port mapping, or use awsvpc network mode, which gives each task its own elastic network interface and IP so there is no port sharing at all. awsvpc is generally preferable because it also enables per-task security groups, with the caveat that it introduces an ENI limit per instance.
Why can my instance only run two tasks in awsvpc mode?
Because each task consumes an elastic network interface, and every instance type has a maximum. A t3.medium supports three ENIs, one of which is the primary, leaving two for tasks regardless of free CPU and memory. Enable ENI trunking with aws ecs put-account-setting --name awsvpcTrunking --value enabled, which raises the limit considerably, and note that only instances launched after enabling it get the higher limit.
How do I stop this happening during deployments?
A rolling deployment with the default maximumPercent of 200 briefly needs capacity for twice the running tasks, which a tightly packed cluster cannot provide. Either add headroom, lower maximumPercent to something like 150 to trade rollout speed for capacity, or attach a capacity provider with managed scaling so ECS scales the Auto Scaling group automatically when tasks cannot be placed. The last option turns the error into a short delay rather than a stalled deployment.