Recovering a root file system error in Amazon EC2 Ubuntu instance

Udayakumar Nalinasekaren
Oct 01, 2014

First, do not play around with your home directory permissions

Amazon EC2

Recently, I was playing around with my Ubuntu free tier t2.micro instance and used a sudo command to change the access rights of the /home/ubuntu directory to 777. I promptly logged out of my putty session without realizing that EC2 does not take this easy. Result: My next putty login failed ( with an error like "server returned public key") and ftp stopped working as well. I had lots of unsaved work on that instance. I thought I was doomed. My first impulse was to let go, reinitialize the instance and restart from the ground up.

Second, All is not lost when you do. Don't panic

However, with some patience and Googling around, I found that I could solve the problem easily and recover my work. The solution was to

create another Ubuntu t2.micro instance
detach the root volume from the broken instance after stopping it
attach it to the newly created instance (after stopping it) as a second volume
start the newly created instance
putty into the second instance. Your private key-pair will work for this instance
mount the second volume in this instance to a newly created directory
restore the home directory permissions to pristine conditions expected by EC2
unmount the second volume
detach the volume from this instance after stopping it
reattach it as root volume of the broken instance
restart the broken instance

Voila! putty session started working again for me.

Detailed Procedure

The detailed procedure is as follows:

Stop (do not terminate) the broken instance A, wait for it to come to a complete stop, detach the root EBS volume from the instance, wait for it to be detached, then attach the volume to instance B as /dev/sdf.

All the operations of starting/stopping instances and attaching/detaching volumes can be performed through your browser based EC2 Management Console.Normally, I found all root volumes are attached as /dev/sda1

ssh (using your private key-pair) to instance B and mount the volume so that you can access its file system.

ssh [instance b]

sudo mkdir -m 000 /vol-a
sudo mount /dev/sdf1 /vol-a

Fix It

At this point your entire root file system from instance A is available for viewing and editing under /vol-a on instance B. For example, you may want to:

Change directory to /vol-a by doing cd /vol-a
Change permissions of directory back to original state using sudo chmod 755 home/ubuntu
Check that the permissions do not allow write access to group and others
If required copy all important files out of /vol-a/

Note: Please cross check and make sure that the owner of the home/ubuntu directory is still ubuntu. This normally should not be a problem as I found that both instances were identically configured with respect to user and group ids.

Finish up

After you are done and you are happy with the files under /vol-a, unmount the file system (still on instance-B):

sudo umount /vol-a
sudo rmdir /vol-a

Go back to the EC2 Management Console and reattach the broken instance's root volume (that you fixed just now) back as /dev/sda. Start this instance now. Your putty login should work like a breeze.