Hello folks,

I have the following problem:

After powering on my fileserver after a power loss, I haven't got any
accessto my files anymore.
It is a SuSE linux 8.2 box with a 3ware Raid controller "Escalade 8506-8"
with 8 SATA harddisks connected to it forming a RAID level 5.

Taking a closer look shows, that the filesystem (raiser FS) get's mounted
read only, with the effect, that the whole boot-process isn't completed
correctly: almost no service starts correctly, reporting read-only
filesystem as the cause of error. After booting up, I can't even log in:
After I enter "root" as username the system answers "FATAL: cannot change
permissions to TTY: Read only file system"

I have made a screenshot of the boot-up, showing the crutial moment, where
the filesystem get's mounted read only. You will find it here:
http://lautenbacher.biz/downloads/DSCF1385.JPG
If needed I can provide further screenshots!

Can anybody please tell me
- why the filesystem get's mounted suddenly as read only? What could be the
possible reasons? The 3ware-controller shows all 8 hard disks to function
ok.
- how I can solve the problem, if I can't even log in?

Thanx for any advice!!!

Greetings
Tom

(f'up to alt.os.linux)

Peter Muller wrote:
> Hello folks,
>
> I have the following problem:
>
> After powering on my fileserver after a power loss, I haven't got any
> accessto my files anymore.
> It is a SuSE linux 8.2 box with a 3ware Raid controller "Escalade 8506-8"
> with 8 SATA harddisks connected to it forming a RAID level 5.
>
> Taking a closer look shows, that the filesystem (raiser FS) get's mounted
> read only, with the effect, that the whole boot-process isn't completed
> correctly: almost no service starts correctly, reporting read-only
> filesystem as the cause of error. After booting up, I can't even log in:
> After I enter "root" as username the system answers "FATAL: cannot change
> permissions to TTY: Read only file system"
>
> I have made a screenshot of the boot-up, showing the crutial moment, where
> the filesystem get's mounted read only. You will find it here:
> http://lautenbacher.biz/downloads/DSCF1385.JPG
> If needed I can provide further screenshots!
>
> Can anybody please tell me
> - why the filesystem get's mounted suddenly as read only? What could be the
> possible reasons? The 3ware-controller shows all 8 hard disks to function
> ok.
> - how I can solve the problem, if I can't even log in?

Just guessing here, the filesystem has some trouble that don't get displayed
on the bootup for some reason (the image is a bit difficult to read, so I
could have missed it), you would need to do a fsck check, reiserfs uses
reiserfsck, do it on the LVM.

I do suggest you add the '1' to your boot option, this usually will boot your
system into a single user environment, this should skip the login, otherwise
boot from a install CD with rescue mode.


--

//Aho

Hello Aho,

> I do suggest you add the '1' to your boot option, this usually will boot
> your system into a single user environment, this should skip the login,
> otherwise boot from a install CD with rescue mode.

how do I add the '1' and where? I am booting up with grub (0.93). what do I
have to set there to start up in single user mode?

Booting up from the SuSE-CD is not working out, ince the driver for the
3ware-controller isn't on it. So I haven't got access to the harddisks in
general.

Thanks a lot!

Tom Muller wrote:

>> I do suggest you add the '1' to your boot option, this usually will boot
>> your system into a single user environment, this should skip the login,
>> otherwise boot from a install CD with rescue mode.
>
> how do I add the '1' and where? I am booting up with grub (0.93). what do I
> have to set there to start up in single user mode?

When you get the grub boot menu, you select to edit (press e), then you select
the kernel row to edit (press e), then add last to the line [space][1][return]
and then select to boot (press b).


> Booting up from the SuSE-CD is not working out, ince the driver for the
> 3ware-controller isn't on it. So I haven't got access to the harddisks in
> general.

Thats tough luck then, I hope the single user mode will work for you.
Otherwise installing a minimal rescue instance could be something for you, of
course this require you have some media where you can put it on (usb memory,
if your motherboard supports booting from one).

--

//Aho

Hello again Aho!

> Just guessing here, the filesystem has some trouble that don't get
> displayed on the bootup for some reason (the image is a bit difficult to
> read, so I could have missed it), you would need to do a fsck check,
> reiserfs uses reiserfsck, do it on the LVM.

ok, thanks to your advices I have now finally managed to boot linux up and
to log in (*blush*)

I have booted the normal system as single user. With other words everything
started up, including the 3ware driver.

I had a look to my fstab. It shows correctly that my partitions sda6, 1, 7,
and 5 are getting mounted to their mounting points.
sda6 gets mounted to / with reiserfs and defaults
sda7 gets mounted to /data with reiserfs and defaults

"mount" also states that these partitions are mounted "rw"
but apparently they aren't.
when i try to unmount them with umount -a thy system says that the paritions
are not mounted.
raiserfschck reports no errors on / and on /data

what could i do now as a next step?

thanx for any advice!

Tom Muller wrote:

> I have booted the normal system as single user. With other words everything
> started up, including the 3ware driver.
>
> I had a look to my fstab. It shows correctly that my partitions sda6, 1, 7,
> and 5 are getting mounted to their mounting points.
> sda6 gets mounted to / with reiserfs and defaults
> sda7 gets mounted to /data with reiserfs and defaults
>
> "mount" also states that these partitions are mounted "rw"
> but apparently they aren't.
> when i try to unmount them with umount -a thy system says that the paritions
> are not mounted.

this most likely as / is mounted as ro, which makes it not possible to write
to /etc/mtab and therefore the system thinks all is unmounted, you can use the
"force" option for unmount, umount -na


> raiserfschck reports no errors on / and on /data

That seems to be okey.


> what could i do now as a next step?

Have you updated your kernel recently? There has happen a great deal in the
kernel the last couple of versions.

What does the dmesg output give you? Any hints there?

What if you remount /data as rw, do you get any error messages?


Sorry, but RAID ain't my strong side, did once in the 90's use SoftRAID, but
found LVM to be what I need.

--

//Aho

Aghaa I can see the problem already, a Daewoo monitor! <grin>

You normally get a RO filesystem if the startup fsck cant repair things
automagically. You do however get a fairly obvious warning about what it
has done. This in fact use to be the default around SuSE v6 and caused
me a few headaches at the time (ext2 FS)

There doesn't appear to be any other warnings in your screen dump. My
first thought was that /etc/fstab must have been mangled so it forces a
RO mount anyway. Weird!

To fix the problem I would boot of a rescue/knoppix CD and run
reiserfsck on the FS. I'd also have a cursory look over files like fstab
in /etc to see they weren't mangled in any way. I'd also check I can
write a file just for fun. If you get horrendous reiserfsck errors
(which seems stupid given your screen dump) and it wants a tree rebuild
make sure your backups are in place in case something goes drastically
wrong.

If you don't have a bootable CD source (and I am foggy in memory here)
you can start it up in single user or debug mode by appending an "s"
(without quotes) to the kernel boot line. I *think* this will get you in
without being asked for a root pwd in the version you are using. You can
then investigate the fault and remount the FS R/W from the command line
if need be.

Hope you find this helpful.

Cheers Bob

Peter Muller wrote:
> Hello folks,
>
> I have the following problem:
>
> After powering on my fileserver after a power loss, I haven't got any
> accessto my files anymore.
> It is a SuSE linux 8.2 box with a 3ware Raid controller "Escalade 8506-8"
> with 8 SATA harddisks connected to it forming a RAID level 5.

>> what could i do now as a next step?
>
> Have you updated your kernel recently? There has happen a great deal in
> the kernel the last couple of versions.
>
> What does the dmesg output give you? Any hints there?
>
> What if you remount /data as rw, do you get any error messages?

ok, I now have managed to remount all partitions in RW mode, and I can see
all my data.
But when i reboot, or change to runlevel 3 (mulituser), get the same RO
problem again.
after the last reboot my eyes caught briefly the message: "can't run fscheck
since root is not in RO" or something similar.
also a driver keeps reporting at bootup that there was an irregular system
shutdown.

As it seems there was somethig altered in the system because of the power
failiure, a kind of security feature, putting everything to readonly. But
where can I change that now?

Tom Muller wrote, On 05.02.2007 16:41:
> As it seems there was somethig altered in the system because of the power
> failiure, a kind of security feature, putting everything to readonly. But
> where can I change that now?

As I had the same error some months ago: you should check if your
/dev/null might be overwritten by a file. Because all the error messages
during boot refer to this device, this could be the problem.

Normally, /dev/null is not on a filesystems, as it's a device so you
shouldn't get such errors. But if it's overwritten by a file, the boot
scripts cannot write to this device anymore to suppress certain
messages, because the partitions are ro at this time.

You can create the device with the following command:
$ mknod /dev/null c 1 3

HTH,
-Philipp

> you should check if your /dev/null might be overwritten

The device should look like this:

| pluto:/dev# ls null
| crw-rw-rw- 1 root root 1, 3 2006-09-19 04:49 null

-Philipp

Tom Muller wrote:
>>> what could i do now as a next step?
>> Have you updated your kernel recently? There has happen a great deal in
>> the kernel the last couple of versions.
>>
>> What does the dmesg output give you? Any hints there?
>>
>> What if you remount /data as rw, do you get any error messages?
>
> ok, I now have managed to remount all partitions in RW mode, and I can see
> all my data.
> But when i reboot, or change to runlevel 3 (mulituser), get the same RO
> problem again.
> after the last reboot my eyes caught briefly the message: "can't run fscheck
> since root is not in RO" or something similar.
> also a driver keeps reporting at bootup that there was an irregular system
> shutdown.
>
> As it seems there was somethig altered in the system because of the power
> failiure, a kind of security feature, putting everything to readonly. But
> where can I change that now?

Okey, I would look closely at your /boot/grub/grub.conf and see if the root
file system is mounted as ro or rw, you want to mount it ro.

kernel /kernel-<version> ro root=/dev/sda6

If the ro is missing, then add it, or change to ro if you see rw.

--

//Aho

Hello Philipp,

> As I had the same error some months ago: you should check if your
> /dev/null might be overwritten by a file. Because all the error messages
> during boot refer to this device, this could be the problem.

BINGO! That was the problem!

Thank you Philip for this advice! And thanx also to J.O. Aho and Bob Bob for
your kind help!!!

Ciao,
Tom