तस्मादेवं विदित्वैनं नानुशोचितुमर्हसि ||
Bhagavat Gita 2:25
It is spoken that the soul is invisible, inconceivable and unchangeable. When this is so, one should not grieve for the body.
Recovering a root file system error in Amazon EC2 Ubuntu instance
Oct 01, 2014
First, do not play around with your home directory permissions
Recently, I was playing around with my Ubuntu free tier
t2.micro instance and used a
sudo command to change the access rights of the
/home/ubuntu directory to
777. I promptly logged out of my
putty session without realizing that EC2 does not take this easy. Result: My next
putty login failed ( with an error like "server returned public key") and
ftp stopped working as well. I had lots of unsaved work on that instance. I thought I was doomed. My first impulse was to let go, reinitialize the instance and restart from the ground up.
Second, All is not lost when you do. Don't panic
However, with some patience and Googling around, I found that I could solve the problem easily and recover my work. The solution was to
- create another Ubuntu
- detach the root volume from the broken instance after stopping it
- attach it to the newly created instance (after stopping it) as a second volume
- start the newly created instance
puttyinto the second instance. Your private key-pair will work for this instance
mountthe second volume in this instance to a newly created directory
- restore the home directory permissions to pristine conditions expected by EC2
unmountthe second volume
- detach the volume from this instance after stopping it
- reattach it as
rootvolume of the broken instance
- restart the broken instance
putty session started working again for me.
The detailed procedure is as follows:
Stop (do not terminate) the broken instance A, wait for it to come to a complete stop, detach the root EBS volume from the instance, wait for it to be detached, then attach the volume to instance B as
All the operations of starting/stopping instances and attaching/detaching volumes can be performed through your browser based EC2 Management Console.Normally, I found all root volumes are attached as
ssh (using your private key-pair) to instance B and mount the volume so that you can access its file system.
ssh [instance b] sudo mkdir -m 000 /vol-a sudo mount /dev/sdf1 /vol-a
At this point your entire root file system from instance A is available for viewing and editing under /vol-a on instance B. For example, you may want to:
Change directory to
Change permissions of directory back to original state using
sudo chmod 755 home/ubuntu
Check that the permissions do not allow write access to
If required copy all important files out of
Note: Please cross check and make sure that the owner of the
home/ubuntu directory is still
ubuntu. This normally should not be a problem as I found that both instances were identically configured with respect to user and group ids.
After you are done and you are happy with the files under
/vol-a, unmount the file system (still on instance-B):
sudo umount /vol-a sudo rmdir /vol-a
Go back to the EC2 Management Console and reattach the broken instance's root volume (that you fixed just now) back as
/dev/sda. Start this instance now. Your
putty login should work like a breeze.