LinuxLinux

Read-only file system

The kernel remounted the filesystem read-only to protect it, or it was mounted that way. How to tell a deliberate mount from an I/O error, and recover safely.

hard fix5 min read

the linux error
touch: cannot touch '/var/log/app.log': Read-only file system

OSError: [Errno 30] Read-only file system

EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:83: Detected aborted journal
EXT4-fs (nvme0n1p1): Remounting filesystem read-only

Do this first3 steps

Run these in order. Each one tells you what its output means before you change anything.

  1. 1

    Find out whether the kernel remounted it or it was mounted that way

    sudo dmesg -T | grep -iE "remount|I/O error|journal|EXT4-fs error|XFS" | tail -20

    An EXT4-fs error or I/O error before the remount means the kernel protected the filesystem from a failing device. No such message means it was simply mounted read-only, which is a configuration question and much less alarming.

  2. 2

    Check how it is currently mounted

    findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /var

    The options list shows ro or rw. Compare against /etc/fstab: a filesystem mounted ro that fstab says should be rw points at a boot-time failure rather than an intentional setting.

  3. 3

    Do not remount rw until you know why it went read-only

    sudo smartctl -H /dev/nvme0n1 2>/dev/null || echo "smartctl not installed"

    Remounting a filesystem that went read-only because of I/O errors invites further corruption. Check device health first, and plan for fsck on an unmounted filesystem if the journal aborted.

All 7 sections

Two entirely different situations produce this, and treating them the same is how data gets lost.

The kernel remounted it. Something went wrong, usually I/O errors, and the kernel switched to read-only to prevent further damage. This is a hardware or corruption event.

It was mounted read-only. A container's root filesystem, an ro entry in /etc/fstab, a read-only NFS export, or a failed rw mount at boot. This is configuration.

Which one is it?

sudo dmesg -T | grep -iE "remount|I/O error|journal|EXT4-fs error|XFS" | tail -20
[Mon Sep 21 09:14:02 2026] EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:83: Detected aborted journal
[Mon Sep 21 09:14:02 2026] EXT4-fs (nvme0n1p1): Remounting filesystem read-only

That is the kernel protecting you. Do not just remount read-write.

No such message, and you are looking at configuration:

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /var
/dev/nvme0n1p1 /var ext4 ro,relatime

Case 1: The kernel remounted it

Work out whether the device is failing before touching anything.

sudo smartctl -H /dev/nvme0n1
sudo smartctl -a /dev/nvme0n1 | grep -iE "reallocated|pending|media_errors|percentage_used"
sudo journalctl -k --since "2 hours ago" | grep -i "i/o error"

On a cloud instance, also check the provider's console. An EBS volume with an impaired status or an instance with a degraded host shows up there and not in SMART.

If the device is healthy, the filesystem still needs checking, and fsck must run on an unmounted filesystem. Running it on a mounted one causes corruption.

For a non-root filesystem:

sudo umount /var
sudo fsck -y /dev/nvme0n1p1
sudo mount /var

For the root filesystem, you cannot unmount it. Either boot from rescue media, or schedule a check at the next boot:

sudo touch /forcefsck
sudo reboot

On systemd, fsck.mode=force as a kernel parameter does the same.

Take an image first if the data matters and you have somewhere to put it:

sudo dd if=/dev/nvme0n1p1 of=/mnt/backup/p1.img bs=4M status=progress

fsck -y answers yes to everything, which can discard data it cannot repair. On anything irreplaceable, run it without -y and read each question.

Once you are confident:

sudo mount -o remount,rw /var

Remounting without investigating is the mistake. If the underlying device is failing, writes resume against failing hardware and the corruption gets worse.

Case 2: Mounted read-only by configuration

grep -v '^#' /etc/fstab | column -t

An ro where you expected rw is straightforward. Fix the entry:

UUID=1a2b3c4d-...  /var  ext4  defaults  0  2
sudo mount -o remount,rw /var

If fstab says rw and it is mounted ro, the mount failed at boot and systemd fell back. journalctl -b | grep -i mount shows why.

errors=remount-ro in the mount options is the default for ext4 and is what produced Case 1. It is correct behaviour and should stay. errors=continue writes through errors and makes corruption worse; errors=panic halts the machine.

Containers

docker run --read-only myapp:1.0

A read-only root filesystem is a good hardening default. Give the application writable space where it genuinely needs it:

docker run --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  -v app-data:/data \
  myapp:1.0

In Kubernetes:

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp
volumes:
  - name: tmp
    emptyDir: {}

Many images write to /tmp, /var/run or a cache directory, so enabling this usually means adding one or two emptyDir mounts. That is worth doing: it turns an arbitrary-write vulnerability into a much less useful one.

Cloud instances

An EBS volume detached and reattached, or an instance stopped and started while a volume was busy, commonly comes back read-only.

lsblk -f
sudo dmesg -T | grep -i nvme

AWS Nitro instances present EBS volumes as NVMe devices with names that do not match the attachment name, so /dev/sdf in the console can be /dev/nvme1n1 on the instance. Use lsblk -f and UUIDs in fstab rather than device names, which change across reboots.

A checklist

  1. dmesg -T | grep -i remount. A kernel remount is not a configuration problem.
  2. Kernel remount → check SMART and the cloud console before anything else.
  3. Never fsck a mounted filesystem.
  4. Root filesystem → touch /forcefsck and reboot, or use rescue media.
  5. Image the device first if the data matters.
  6. No kernel error → findmnt and /etc/fstab.
  7. Keep errors=remount-ro. It is doing its job.
  8. Containers → read-only root is correct; add tmpfs or emptyDir for writable paths.

Frequently Asked Questions

Why did my filesystem become read-only by itself?

The kernel remounted it to prevent further damage after detecting a problem, almost always I/O errors from the underlying device or a corrupted journal. ext4 mounts with errors=remount-ro by default precisely so that a failing disk causes a visible, safe failure rather than silent corruption spreading through your data. dmesg -T shows the triggering error immediately before the remount line, and that message is the thing to investigate, not the read-only state itself.

Can I just remount it read-write?

Not before you know why it went read-only. If the cause was I/O errors from a failing device, remounting read-write resumes writing to failing hardware and makes the corruption worse. Check device health with smartctl -H, look at the cloud provider's console for a volume or host problem, and run fsck on the unmounted filesystem. Once the device is confirmed healthy and the filesystem is clean, mount -o remount,rw is safe.

How do I run fsck on the root filesystem?

You cannot unmount root while the system is running, and running fsck on a mounted filesystem causes corruption. The two options are to boot from rescue media and check the device from there, or to schedule a check for the next boot with sudo touch /forcefsck followed by a reboot. On systemd, adding fsck.mode=force to the kernel command line achieves the same thing. Either way, take a dd image first if the data matters.

Should I use errors=continue to keep the system running?

No. errors=continue tells the kernel to carry on writing after detecting filesystem errors, which turns a contained problem into progressive corruption across your data. The read-only remount is a safety mechanism, and an application failing to write is a much better outcome than a filesystem quietly becoming inconsistent. If availability during a disk fault matters, the answer is redundancy at the storage layer, not disabling the kernel's protection.

Why is my container's filesystem read-only?

Because it was started with --read-only, or in Kubernetes with readOnlyRootFilesystem: true. This is a deliberate hardening measure and a good default, since it prevents an attacker who achieves code execution from modifying binaries or dropping a persistent payload. Most applications need somewhere writable, typically /tmp, /var/run or a cache directory, so mount a tmpfs or an emptyDir at those specific paths rather than disabling the setting entirely.

Reference and practice

Learn the underlying concept

Other Linux errors