Amazon Linux 2 vs AL2023: The /tmp tmpfs Default That's Killing Your Containers

We upgraded our EC2 instances from Amazon Linux 2 to Amazon Linux 2023. Deployment went smoothly. Then the OOM kills started.

Containers on the upgraded hosts began dying with out-of-memory errors. Memory usage on the hosts looked higher than expected. The applications hadn't changed. The instance types hadn't changed. Only the OS had changed — and somewhere in that change, a default had shifted that nobody had documented in our migration checklist.

After digging through /proc/mounts and comparing the two AMIs, the cause was clear: Amazon Linux 2023 mounts /tmp as tmpfs by default. Amazon Linux 2 does not.

What changed between AL2 and AL2023

On Amazon Linux 2, /tmp is a regular directory on the root filesystem. Writes to /tmp consume disk space. Memory is unaffected.

On Amazon Linux 2023, /tmp is mounted as tmpfs — an in-memory filesystem, via the systemd tmp.mount unit. The tmpfs size is capped at 50% of the host's physical RAM by default.

Verify on an AL2023 host:

$ mount | grep tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=50%)

$ df -h /tmp
Filesystem  Size  Used Avail Use%
tmpfs       7.7G  4.2G  3.5G  55%

On AL2, /tmp is on the root EBS volume — disk space, not RAM.

Why this caused OOM kills

Our application writes heavily to /tmp — temporary processing files, intermediate artifacts, cache files cleaned up after each job. On AL2, these writes went to EBS. On AL2023, they went straight into RAM.

With enough concurrent workload, /tmp fills up to 50% of host RAM. The kernel OOM killer then terminates processes to reclaim memory. On a container host, that means containers die.

The failure mode is insidious because it doesn't look like a /tmp problem. You see OOM kills. You check application heap — looks normal. You check container limits — not exceeded. The last thing you think to check is what filesystem /tmp is backed by.

Other AL2 vs AL2023 differences worth knowing

Package manager — AL2 uses yum. AL2023 uses dnf. Scripts and Dockerfiles that call yum install need updating.

IMDSv2 required — AL2023 requires IMDSv2 by default. Applications using the old IMDSv1 endpoint without a session token will fail silently.

OpenSSL 3.x — AL2023 ships OpenSSL 3. Applications depending on deprecated ciphers or older OpenSSL 1.x APIs may break.

Python — python (unversioned) is not available. Scripts with #!/usr/bin/env python shebangs will fail. Use python3 explicitly.

NetworkManager — AL2023 uses NetworkManager exclusively. Legacy /etc/sysconfig/network-scripts/ configs don't work.

SELinux — AL2023 ships with SELinux in permissive mode (AL2 had it disabled). Unlikely to cause immediate breakage but worth monitoring.

Fixing the /tmp tmpfs issue

Option 1: Disable the tmpfs mount

# Mask the systemd unit to prevent /tmp tmpfs from activating
sudo systemctl mask tmp.mount
sudo reboot

# Verify — should show nothing for /tmp
mount | grep tmp

Option 2: Resize the tmpfs

# Create a drop-in to override the size
sudo mkdir -p /etc/systemd/system/tmp.mount.d/
cat <

Option 3: Redirect application temp writes to EBS

# Create a dedicated temp dir on a mounted EBS volume
mkdir -p /data/tmp && chmod 1777 /data/tmp

# Set per application
export TMPDIR=/data/tmp
export JAVA_OPTS="-Djava.io.tmpdir=/data/tmp"

For container workloads, Option 3 is often the cleanest: mount an EBS volume for temp files and set TMPDIR in the container environment. This decouples containers from host /tmp behavior entirely.

Check before upgrading

# Watch /tmp usage over a typical workload cycle on AL2
watch -n 5 'du -sh /tmp/* 2>/dev/null | sort -hr | head -20'

# See what processes are writing to /tmp
inotifywait -m -r /tmp --format '%T %w%f %e' --timefmt '%H:%M:%S' 2>/dev/null | head -100

If /tmp usage regularly exceeds what you'd be comfortable holding in RAM, plan to disable or resize tmpfs before rolling out AL2023.

Container-specific notes

The tmpfs /tmp is a host-level resource shared across all containers. One container writing large temp files can OOM-kill unrelated containers on the same host.

• Account for /tmp tmpfs usage when setting container memory limits — it counts against host RAM, not EBS
• Use container-level tmpfs mounts with explicit size limits: --tmpfs /tmp:rw,size=512m
• Override TMPDIR per container to a volume-backed path
• Monitor host tmpfs usage separately from container memory in your observability stack

My take

The AL2023 /tmp tmpfs default is a reasonable choice in isolation — faster I/O, aligns with modern systemd defaults, matches other distributions. The problem is it's a silent behavior change with a failure mode that's easy to misattribute to application memory leaks or container resource limits.

When migrating Linux distributions, always diff mount output between old and new before touching production. A two-line change in /proc/mounts can cascade into OOM kills weeks after the upgrade.