jump to navigation

How to Solve “SSH Permission Denied ” (public-key authentication) When Trying to Access Amazon’s EC2 January 6, 2013

Posted by Tournas Dimitrios in Amazon-aws.
trackback

This morning during a regular system-administration task  on an Amazon EC2 Linux CentOs box  , I came across a technical issue , and thought it would be a good subject for an article . While doing a customization on SSHd’s  configuration file (/etc/sshd/sshd_config) ,  suddenly I was locked out and could not have ssh access to the Linux Cloud-box . The message was short and clear “Access Denied” , it couldn’t be more descriptive 🙂  . Fortunately , the cloud-server was build “around” EC2 and EBS  , so the solution was only a few clicks away (to be honest with you , a few dozen clicks away ) .  An experienced sys-administrator can go straight to the “command-line workflow”  below , most likely the rest of this article is already known by him /her . For all newcomers to the world of Cloud-computing , a short reading prior this command-line workflow is absolutely necessary , otherwise  the whole concept will have no meaning  .

Prerequisites : The reader should already have a good grasp of Linux administration  and of course a basic knowledge of Amazon’s web-services . This article assumes that the reader already knows how to instantiate a new Linux machine (ami) via Amazon’s  web-interface  and how to remotely administrate  a Linux box via a terminal (no GUI’s) .

In Amazon’s parlance , an EC2 instance is an operating system running on a virtual machine (on the Cloud)  . There are dozen of instances (free or paid ) available , which have a funny name AMI . An Amazon Machine Image (AMI) is a  pre-configured operating system and  is used to create a virtual machine within Amazon’s Cloud-computing infrastructure . A special type of AMI is the ami-ebs type which actually uses an external persistent storage medium (in context of the virtual machine) to store it’s root file system . EBS provides persistent , high-performance , and high-availability block-level storage which you can attach to a running EC2 instance (in the same availability zone) in the form of volumes (1GB – 1TB) . Having the “root file-system” on it’s own storage device (EBS) , the system-administrator has a flexible tool-set in his / her hands (for instance , backups and mirroring ) . Other technical jargon used by Amazon is , start stop – reboot – terminate , of an EC2 instance . The first three are self-explanatory , the last (terminate) might confuse a newcomer to Amazon’s services . It designates a “throw it to the garbage bucket” , and that means no “restore” functionality . Here comes the good part , if the file system is stored into an EBS volume (which is a persistent storage) , then throwing the EC2 instance to the garbage ,  will not delete our data (configuration files , packages , compiled files ) . And that’s the key to solve the problem that was introduces at the beginning of this article . Simple solution , just mount the EBS volume on another working EC2 instance and edit it’s files . As of this writing , Amazon’s EBS volumes can’t be attached on multiple EC2 instances , before it can be attached to another EC2 instance , we have to detach it first  . Basic tasks  like : instantiation , starting , stopping , rebooting and terminating of an EC2 instance are done via a web-interface .  Here is the workflow :

  • Stop the EC2 instance .
  • Detach it’s EBS volume
  • Instantiate a new EC2 instance and start it up
  • Attach the EBS volume into the new EC2 instance (notice the name of the device its attached to /dev/sdf)
  • Login via SSH (public key authentication )
  • Run the following commands . Actually what you are doing is : mounting the EBS volume into the new file-system and editing the proper files .
# The old EBS volume is not jet attached into the new EC2 instance
[ec2-user@ip-10-243-113-228 ~]$ ls -l  /dev/  |grep xvd
lrwxrwxrwx 1 root root           5 Jan  5 17:46 root -> xvde1
lrwxrwxrwx 1 root root           4 Jan  5 17:46 sda -> xvde
lrwxrwxrwx 1 root root           5 Jan  5 17:46 sda1 -> xvde1
lrwxrwxrwx 1 root root           5 Jan  5 17:46 sda2 -> xvde2
lrwxrwxrwx 1 root root           4 Jan  5 17:46 sdf -> xvdj

# Manually attaching the EBS volume on the new EC2 instance (on /dev/sdf)

[ec2-user@ip-10-245-117-228 ~]$ ls -l  /dev/  |grep xvd
lrwxrwxrwx 1 root root           5 Jan  5 18:09 root -> xvdj1
lrwxrwxrwx 1 root root           4 Jan  5 18:04 sda -> xvde
lrwxrwxrwx 1 root root           5 Jan  5 18:04 sda1 -> xvde1
lrwxrwxrwx 1 root root           5 Jan  5 18:04 sda2 -> xvde2
lrwxrwxrwx 1 root root           4 Jan  5 18:09 sdf -> xvdj
lrwxrwxrwx 1 root root           5 Jan  5 18:09 sdf1 -> xvdj1
lrwxrwxrwx 1 root root           5 Jan  5 18:09 sdf2 -> xvdj2
brw-rw---- 1 root disk    202,  64 Jan  5 18:04 xvde
brw-rw---- 1 root disk    202,  65 Jan  5 18:04 xvde1
brw-rw---- 1 root disk    202,  66 Jan  5 18:04 xvde2
brw-rw---- 1 root disk    202, 144 Jan  5 18:09 xvdj
brw-rw---- 1 root disk    202, 145 Jan  5 18:09 xvdj1
brw-rw---- 1 root disk    202, 146 Jan  5 18:09 xvdj2
[ec2-user@ip-10-245-117-228 ~]$ mount
/dev/xvde1 on / type ext4 (rw,relatime)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[ec2-user@ip-10-245-117-228 ~]$ sudo mkdir /ebs
[ec2-user@ip-10-245-117-228 ~]$
[ec2-user@ip-10-245-117-228 ~]$ sudo mount -t ext4  /dev/xvdj1  /ebs
[ec2-user@ip-10-245-117-228 ~]$ mount
/dev/xvde1 on / type ext4 (rw,relatime)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdj1 on /ebs type ext4 (rw)
[ec2-user@ip-10-245-117-228 ~]$ sudo vi  /ebs/etc/ssh/sshd_config
[ec2-user@ip-10-245-117-228 ~]$ sudo umount  /dev/xvdj1
[ec2-user@ip-10-245-117-228 ~]$ mount
/dev/xvde1 on / type ext4 (rw,relatime)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[ec2-user@ip-10-245-117-228 ~]$
  • After the editing process , stop the EC2 instance
  • Detach the EBS volume from the new EC2 and attach it to the old EC2 instance
  • Start the old EC2 instance .

The same process can be applied to do some other form of tasks , for instance , changing (SSH) keys .

Advertisements

Comments»

1. Celsin - January 13, 2013

the contents are good, but if you improve the site structure, it would be better.

2. anke - January 28, 2013

Hello, I enjoy reading all of your post. I like to write a little comment to support you.

3. Lorena - April 23, 2013

I think this is among the most significant info for me.

And i am glad reading your article. But want to remark
on few general things, The website style is perfect,
the articles is really great : D. Good job, cheers

4. Mayo - May 13, 2013

Write more, thats all I have to say. Literally, it
seems as though you relied on the video to make your point.
You obviously know what youre talking about, why throw away your intelligence on just posting videos to your
weblog when you could be giving us something informative to read?

5. Larry Weya - September 26, 2013

Saved me a headache, thanks.

tournasdimitrios1 - September 26, 2013

@Larry
You are welcome , thanks for your comments


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s