[Cialug] Spontaneous Outbreak of Read-Only-ness

[Sean Flattery]

I've had this happen when the VM couldn't talk to the SAN hosting the file
system and would go read only due to IO wait. The high IO load was caused
by other noisy neighbors so it was tough to track down. If things aren't
virtualized, then I wholeheartedly endorse the advice to check your disks
and make sure you have good backups.

Thanks
Sean Flattery

Date: Wed, 24 Feb 2016 11:24:27 -0600
From: Todd Walton <tdwalton at gmail.com>
To: Central Iowa Linux Users Group <cialug at cialug.org>
Subject: [Cialug] Spontaneous Outbreak of Read-Only-ness
Message-ID:
        <CALm_Md9dSR1U8jkEWAKMi4qf+DtROLXg2XZzCozpp5pgm3WGiw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

I've had two weird read-only-izings happen in the past 24 hours.

First, a RHEL 6 box: Found I couldn't write files to /tmp, even as root.
Further poking revealed that I couldn't write to / either. 'cat
/proc/mounts' said / was rw. Changing SELinux to permissive didn't help.
Rebooted. All is well.

Then, a CentOS 6 box: Tried to update root password. It took the password
twice and then said "passwd: Authentication token manipulation error". I
tried to mv the shadow file, thinking maybe I'd re-shadow passwd, but it
wouldn't let me move it because... read-only filesystem. Again, 'cat
/proc/mounts' showed that that should not have been the case. I rebooted
and all is now well.

I can't troubleshoot these further right now because I made the problem go
away. But anybody seen this before?

--
Todd