Alestic.com

You Should Use EBS Boot Instances on Amazon EC2

2012-01-03T19:00:00Z

EBS boot vs. instance-store

If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store and accept my apology that I just mentioned it. Once you are completely comfortable with using EBS boot instances on EC2, you may (or may not) want to come back here and read why you made a good decision.

EC2 experts may find that there are specific cases, few and far between, where instance-store might make sense, but they don’t attempt to use instance-store without understanding and accounting for all the serious drawbacks and dangers that go with making this choice. For example, experts using instance-store don’t mind losing all of the data on the instance as they have designed the system so that the data is stored elsewhere and so that a new instance can easily and automatically be rebuilt from scratch.

One of the challenges for beginners is that many of the benefits of EBS boot don’t necessarily seem like something you’ll need to use right away. Then they get down the road and into situations where they realize that they would have been much better off if they had gone with EBS boot in the first place and may find it takes some work to make the transition.

Big benefits of EBS boot instances

Here are some of the reasons I use and recommend EBS boot instances. None of these benefits are available with instance-store, so even a single one of these can be an overriding factor for choosing EBS boot.

EBS boot instances store the root file system on an EBS volume which is persistent storage. If the instance hardware fails, the EBS volume remains accessible. It is also possible to request the EBS volume to persist beyond the termination of an EC2 instance. When an instance-store instance fails or is shut down, all of the data on the root disk is lost forever and can never be retrieved. Read more about protecting EC2 instances from accidental termination and loss of data.
EBS boot instances can be stopped and restarted at will. The “stopped” state suspends the hourly instance billing charges, preserving the information on the EBS volumes. The stopped instance can be started again a few minutes later or months later, restoring state just as if the instance was rebooted. An instance-store instance can only be left running with full charges or terminated, which causes you to lose all data on the disk. Read more about how stop/start of an EBS boot instance is similar to and different from a simple reboot.
When something goes wrong with an EBS boot instance so that it can’t be booted, or you lose access through ssh (e.g., lost keys, bad ssh config change), you can still view and modify or fix the EBS root volume by attaching it to another running instance. With an instance-store instance, everything on the root disk is lost and cannot be recovered. Read more about fixing files on the root EBS volume of an EC2 instance.
EBS boot instances can be run with a root EBS volume size from the default specified by the AMI (often 8GB) up to 1,000GB (1TB). The instance-store AMIs have a max root disk size of 10GB with no way to increase it. Read more about increasing the root disk size of an EBS boot AMI.
It is possible to grow the size of the root disk after an EBS boot instance has been started. An instance-store instance has no way to grow the root disk size. Read more about resizing the root disk on a running EBS boot EC2 instance.
It is possible to change the instance type for a running EBS boot instance without needing to start a new instance. For example, you can scale up from an m1.large to an m1.xlarge and then a few hours or days later, scale back down. An instance-store instance is stuck with the type on which it was originally run. Read more about changing the instance type of a running EBS boot instance.
You can easily replace the hardware for your instance if you are running an EBS boot instance. This is extremely valuable if your instance is having problems that you suspect may be related to the underlying hardware. An instance-store instance is bound to the hardware it started on and cannot be moved. Read more about using stop/start to replace EC2 hardware.
EBS boot AMIs are simpler and faster to create than instance-store AMIs. In fact, you can trigger the creation of an EBS boot AMI from a running instance in one command, API call, or console click. You need to copy sensitive AWS credentials to the instance when creating an instance-store AMI.
Amazon has stated that EBS boot AMIs boot up faster than S3 based AMIs (instance-store). In my recent experience, the difference is negligible, especially when testing popular AMIs that are likely to be cached, but we might as well chalk this up as another benefit.
The t1.micro instance type released recently by Amazon only supports EBS boot instances. This move is like a sign from Amazon that you really don’t want to run the legacy instance-store instances.
Some versions of Windows (Server 2008) only run on EBS boot instances. I believe this may be related to the disk size limitations of instance-store, but I don’t use Windows, so am not an expert in that area.

Possible benefits of instance-store instances

EBS volumes and EBS boot instances aren’t perfect. Running an instance-store instance might be preferable in some very specific cases where you don’t care so much about losing the data you are storing on the root disk.

I’m going to list some of the possible benefits of instance-store, but each of these may not be as beneficial as they appear at first glance and you must remember that running instance-store loses all of the above benefits.

There is a negigible cost savings with instance-store, as there is no charge for an EBS volume nor the I/O transactions. Note however, that the cost for an 8GB root EBS volume is only around 80 cents per month (depending on the region) and you get about a billion I/O requests for a dollar, billed by the penny increment. The cost savings to run instance-store is generally not worth the increased risk of losing your valuable data, especially when you compare it as a percentage of the cost of running the instance hours in the first place (tens or hundreds of dollars per month).
There may be some applications that perform somewhat better when running against ephemeral or instance-store disks than against EBS volumes. Some high end users have also reported inconsistency in the performance they see from EBS volumes at high I/O transaction rates for long periods of time. EBS volumes perform better for some applications and are generally good enough for most applications. If you don’t care about losing your application data, and you have tested to see that instance-store performs better than EBS volumes, then you could either run instance-store instances, or you could run EBS boot instances and drop your data on the ephemeral disks available to all instances above t1.micro. It is also becoming increasingly popular to run multiple EBS volumes in a mdadm RAID-0 or LVM striping configuration to improve performance and smooth out some experiences of performance volatility.
There are a couple historical failure modes that have happened with EBS volumes that could not happen for instance-store disks. You may hear people say this is a reason not to use EBS volumes, but EBS volumes are still far more reliable, persistent, and protected against failure than instance-store. The fact that it is possible for EBS to fail is not a reason to use the less persistent instance-store. It is a reason to create regular snapshots of your EBS volumes to improve reliability and act as backups.
If you are creating a “paid AMI” you can only do it as an instance-store AMI, not EBS boot. Only two of the thousands of people reading this article are creating paid AMIs and they already know this fact.

Why did I write this article?

I regularly provide consulting services in public communities like serverfault, the ##aws IRC channel on Freenode, the ec2ubuntu Google group, and Amazon’s EC2 forum. It’s a rare week that passes where I am not telling somebody that they would not have the problem they’re having if they had been using EBS boot instances.

The saddest response I can give to a plea for help is that the customer’s valuable data has been lost because they were running instance-store instead of EBS boot and they did not have real-time streaming backups.

Historical background

When Amazon launched EC2 in 2006, only instance-store AMIs were available (though they weren’t called instance-store at the time as they were the only kind). For years, Amazon customers learned to work with the server limitations of risking the loss of all data at any point in time.

In August of 2008, Amazon introduced the concept of EBS volumes, and there was much rejoicing. Data could finally be stored on persistent disks even though the root disk remained on ephemeral storage (another name for instance-store).

In December of 2009, Amazon released the ability to launch instances with EBS volumes as the root disk, or EBS boot instances. Now, the entire server could be persistent and all of the above benefits were realized.

Notes

Just because EBS boot volumes are “persistent” does not mean that they do not ever fail. Amazon has released figures about their failure rates, which are proportional to the number of blocks modified since the last EBS snapshot was created. Regular snapshots improve the intrinsic reliability of your EBS volumes in addition to acting as backups. I also recommend creating regular off-site backups (outside of Amazon) to eliminate your AWS account as a single point of failuire.

Even if you use an EBS boot instance, I still recommend keeping your data on a separate EBS volume. This has a number of benefits that could perhaps form the core of a followup article. Read an example of how to set up a MySQL database on a separate EBS boot volume. I still follow this approach with EBS boot instances, storing data on a second volume.

[Update 2011-01-04: Added benefit #11 (Windows Server 2008). Added note about potential (in)consistency of EBS IO at sustained, high rates.]

Original article: http://alestic.com/2012/01/ec2-ebs-boot-recommended

Retrieve Public ssh Key From EC2

2011-12-20T14:00:00Z

A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as it demonstrates a few handy features of EC2.

Challenge

The basic need is to get the public ssh key from a keypair that exists inside of EC2. You don’t have access to the private key at the moment (but somebody else does or you will at a different location).

The AWS console and EC2 API do not let you ask for the public ssh key associated with a keypair. However, EC2 does pass the public ssh key to a new EC2 instance when you run it with a specific keypair.

The problem is that we don’t currently have the private key, so we can’t log in to the EC2 instance to get the public key. (Besides, if we did have the private key, we could extract the public key from it directly.)

Solution

I proposed creating a user-data script that sends the public ssh key to the EC2 instance console output. You can retrieve the console output without logging in to the EC2 instance.

Save the following code to a file named output-ssh-key.userdata on your local computer. DO NOT RUN THESE COMMANDS LOCALLY!

#!/bin/bash -ex
exec> >(tee /var/log/user-data.log|logger -t user -s 2>/dev/console) 2>&1
adminkey=$(GET instance-data/latest/meta-data/public-keys/ | 
  perl -ne 'print $1 if /^0=[^a-z0-9]*([-.@\w]*)/i')
cat <<EOF
SSHKEY:===================================================================
SSHKEY:HERE IS YOUR PUBLIC SSH KEY FOR KEYPAIR "$adminkey":
SSHKEY:$(cat /home/ubuntu/.ssh/authorized_keys)
SSHKEY:===================================================================
SSHKEY:Halting in 50min ($(date --date='+50 minutes' +"%Y-%m-%d %H:%M UTC"))
EOF
sleep 3000
halt

Run a stock Ubuntu 10.04 LTS instance with the above file as a user-data script. Specify the keypair for which you want to retrieve the public ssh key:

ec2-run-instances \
  --key YOURKEYPAIRHERE \
  --instance-type t1.micro \
  --instance-initiated-shutdown-behavior terminate \
  --user-data-file output-ssh-key.userdata \
  ami-ab36fbc2

Keep requesting the console output from the instance until it shows your public ssh key. Specify the instance id returned from the run-instances command:

ec2-get-console-output YOURINSTANCEID | grep SSHKEY: | cut -f3- -d:

Repeat the above command a couple times a minute and within 2-10 minutes you will get output like this:

===================================================================
HERE IS YOUR PUBLIC SSH KEY FOR KEYPAIR "erich":
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6rn8cl41CkzaH4ZBhczOJZaR4xBBDI1Kelc2ivzVvCB
THcdJRWpDd5I5hY5W9qke9Tm4fH3KaUVndlcP0ORGvS3PAL4lTpkS4D4goMEFrwMO8BG0NoE8sf2U/7g
aUkdcrDC7jzKYdwleRCI3uibNXiSdeG6RotClAAp7pMflDVp5WjjECDZ+8Jzs2wasdTwQYPhiWSiNcfb
fS97QdtROf0AcoPWElZAgmabaDFBlvvzcqxQRjNp/zbpkFHZBSKp+Sm4+WsRuLu6TDe9lb2Ps0xvBp1F
THlJRUVKP2yeZbVioKnOsXcjLfoJ9TEL7EMnPYinBMIE3kAYw3FzZZFeX3Q== erich
===================================================================
Halting in 50min (2011-12-20 05:58 UTC)

The temporary instance will automatically terminate itself in under an hour, but you can terminate it yourself if you’d like to make sure that you aren’t charged more than the two cents this will cost to run.

Notes

If you currently have access to the private ssh key (not true in the above challenge) you can extract the public ssh key using a command like:
```
ssh-keygen -y -f KEYFILE.pem
```
but that’s obviously not as fun.
There is no way to retrieve the private ssh key if you have lost it. To protect your security, Amazon EC2 does not store a copy of this. If you are looking to get access to an EC2 instance where you have lost the private ssh key, I recommend following the approach I wrote about in this article: http://alestic.com/2011/02/ec2-fix-ebs-root
In seemingly-related-but-not news, Scott Moser is working on an enhancement to cloud-init (used by Ubuntu on EC2, Amazon Linux, and perhaps others) so that the public ssh host keys are output to the console output on instance startup. This cool feature will allow us to add the ssh host keys to our local known_hosts files, safely avoiding that pesky “Are you sure you want to continue connecting (yes/no)?” warning.

Original article: http://alestic.com/2011/12/ec2-public-ssh-key

Running EC2 Instances on a Recurring Schedule with Auto Scaling

2011-11-15T13:00:00Z

Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time?

Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance?

Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing).

In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minumum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job.

Here’s a high level overview for folks familiar with Auto Scaling:

The instance is started using a recurring schedule action that raises the min and max to 1.
The launch configuration is set to pass in a user-data script that runs the desired job on first boot. There’s no need to build our own AMI unless the software installation takes too long.
The user-data script runs shutdown at the end to move the instance to the “stopped” state, suspending hourly run charges.
Another recurring schedule action lowers the min and max to 0. This is done long after the job should have completed, just to clean up the stopped instance or to terminate an instance that perhaps failed to shut itself down. It also resets the min/max count so a new instance will be started the next time they are raised.

The missing key I found through experimentation is that we need to also suspend Auto Scaling’s (usually valid) desire to replace any unhealthy instances. If we don’t turn off this behavior, Auto Scaling will start another instance as soon as the first one shuts itself down, causing the job to be run over and over.

Prerequisites

The examples in this article assume that you have:

Installed and set up the EC2 command line tools
Installed and set up the EC2 Auto Scaling command line tools
Tell EC2 about your ssh keys using the approach described here:

Uploading Personal ssh Keys to Amazon EC2

User-data Script

The first step in setting up this process is to create a user-data script that performs the tasks you want your scheduled instance to execute each time it runs.

This user-data script can do anything you want as long as it runs this command after all the work has beeen completed:

shutdown -h now

I have created a demo user-data script for this article which you can download to your local computer now, saving it as demo-user-data-script.sh

wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh

WARNING: Do not attempt to run this user-data script on your local computer!

Edit the downloaded demo-user-data-script.sh file and change this line at the top of the script to reflect your email address:

EMAIL=youraddress@example.com

This demo user-data script:

Upgrades the Ubuntu instance
Installs Postfix with a generic configuration so that it can send email
Sends you a demo informative message about the instance at the email address you edited in the script.
Sleeps for 5 minutes giving the email a chance to be delivered
Shuts down the EC2 instance

The first time you run this demo, try using the script as it stands, only changing your email address. Then, try adjusting the script little by little to make it do tasks that you would find more useful.

If you’d like, you can test the user-data script by itself running an instance of Ubuntu 11.10. Please update the AMI id to use the latest release:

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
ec2-run-instances \
  --key $USER \
  --instance-type t1.micro \
  --instance-initiated-shutdown-behavior terminate \
  --user-data-file demo-user-data-script.sh \
  $ami_id

You should see an email from the instance and then it should terminate itself about 5 minutes later. Make sure you terminate it manually if it stays running after 10 minutes.

Auto Scaling Group

With the user-data script in hand, we are now ready to create the Auto Scaling setup.

Set some variables used in the commands below. Make sure you are using the latest release of the appropriate AMI.

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
region=us-east-1    # Region for running the demo

zone=${region}a     # A zone in that region
export EC2_URL=https://$region.ec2.amazonaws.com
export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com
launch_config=demo-launch-config
auto_scale_group=demo-auto-scale-group

This lauch configuration describes how we want our instance run each time including the AMI id, instance type, ssh key, and most importantly, our user-data script we edited above:

as-create-launch-config \
  --key $USER \
  --instance-type t1.micro \
  --user-data-file demo-user-data-script.sh \
  --image-id $ami_id \
  --launch-config "$launch_config"

The Auto Scaling group keeps track of many things including how many instances we want to have running, how they should be run (launch config above), and what instances are currently running.

as-create-auto-scaling-group \
  --auto-scaling-group "$auto_scale_group" \
  --launch-configuration "$launch_config" \
  --availability-zones "$zone" \
  --min-size 0 \
  --max-size 0

Here’s a non-obvious but key part of this approach! Tell the Auto Scaling group that we don’t want it to restart our instance right after the instance intentionally shuts down (or fails):

as-suspend-processes \
  "$auto_scale_group" \
  --processes ReplaceUnhealthy

Now we’re finally ready to tell EC2 Auto Scaling when we want to run the instance launch configuration in the Auto Scaling group.

Here’s an example that starts one instance four times a day to run the above user-data script:

# UTC: 1:00, 7:00, 13:00, 19:00
as-put-scheduled-update-group-action \
  --name "demo-schedule-start" \
  --auto-scaling-group "$auto_scale_group" \
  --min-size 1 \
  --max-size 1 \
  --recurrence "0 01,07,13,19 * * *"

And, we need to create a matching action to make sure the instance is terminated at some point after the longest time the job could take. For this demo, we’ll trigger it 55 minutes later, but it could just as easily be 3 hours and 55 minutes later:

# UTC: 1:55, 7:55, 13:55, 19:55
as-put-scheduled-update-group-action \
  --name "demo-schedule-stop" \
  --auto-scaling-group "$auto_scale_group" \
  --min-size 0 \
  --max-size 0 \
  --recurrence "55 01,07,13,19 * * *"

The recurrence value is in a cron format using UTC.

You are welcome to change the specs in the above commands to any time you want to run the demo, especially if you don’t want to wait up to six hours for it to trigger.

Before setting new schedules, make sure you delete the existing schedule (see the next section). Don’t forget to make the stop time(s) match up appropriately with the start time(s).

Clean up

Once you’re done with this demo, you can delete the AWS resources we created by following these steps:

Delete the schedule start and stop actions:

as-delete-scheduled-action \
  --force \
  --name "demo-schedule-start" \
  --auto-scaling-group "$auto_scale_group"
as-delete-scheduled-action \
  --force \
  --name "demo-schedule-stop" \
  --auto-scaling-group "$auto_scale_group"

Scale the Auto Scaling group down to zero instances. This will terminate any running instances:

as-update-auto-scaling-group \
  "$auto_scale_group" \
  --min-size 0 \
  --max-size 0

Delete the Auto Scaling group

as-delete-auto-scaling-group \
  --force-delete \
  --auto-scaling-group "$auto_scale_group"

Delete the Auto Scaling launch config:

as-delete-launch-config \
  --force \
  --launch-config "$launch_config"

You might now want to check to make sure nothing was left over. This works best in a wide terminal:

as-describe-launch-configs --headers
as-describe-auto-scaling-groups --headers
as-describe-scheduled-actions --headers
as-describe-auto-scaling-instances --headers

Timing

Everything takes a little time to filter through the system including:

scheduled action to raise min/max
triggering the start of an instance after a min/max is raised
starting an instance
booting an instance, installing software
running your job
shutting down an instance
scheduled action to lower min/max
triggering the termination of an instance after a min/max is lowered

When you set up the schedules, remember to make room for these things. For example, don’t schedule the termination of your instance too early or it could kill your job before it has a chance to complete.

Notes

Here are some great resources from Amazon for getting started with and learning about Auto Scaling:

The user-data script logging uses the approach described here:

Logging user-data Script Output on EC2 Instances

Watching the output of the user-data script lets you monitor its progress as well as debug where things might be going wrong.

I haven’t run the above approach except in testing, and would welcome any pointers or improvements folks might have to offer.

Original article: http://alestic.com/2011/11/ec2-schedule-instance

AWS Virtual MFA and the Google Authenticator for Android

2011-11-03T03:41:00Z

Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking up unwanted space in my pocket for two years.

Multi-factor authentication means that in order to log in to my AWS account using the AWS console or portal (including the AWS forums) you not only need my secret password, you also need access to a device that I carry around with me.

Before, this was a physical device attached to my key ring. Now, this is my smart phone which has the virtual (software) MFA device on it. I already carry my phone with me, so the software doesn’t take up any additional space.

To log in to AWS, I enter my password and then the current 6 digit access code displayed by the Android app on my phone. These digits change every 30 seconds in an unguessable pattern, so this enhances the security of my AWS account.

I started by using Amazon’s AWS Virtual MFA app for my Android phone, but had some complaints about it including:

You have to click on an account name to see the current digits instead of just having them shown when the app is run. There’s nothing else for the app to do but show me these digits. Just do it!
The digits disappear from the screen too fast. Sometimes I want to glance back and see if I typed them in correctly, but they’re gone and I have to click again, hoping that they haven’t changed yet.
It’s hard to choose your own account names so that you know which entry to use for different AWS accounts.

I then noticed some cryptic information in the announcements: the new feature will work with “any application that supports the open OATH TOTP standard”.

Hmmm…

Sure ‘nuff!

I already use the Google Authenticator app on my Android phone so that my Google logins can use MFA. As it turns out, Google Authenticator also works seamlessly with AWS Virtual MFA.

Google Authenticator shows the codes as soon as it is run with a little timer showing me when they will change.
Google Authenticator lets me easily edit the displayed name so that I know at a glance which code is for my personal AWS account and which one is for my company AWS account.

This also means that I only have to run one app to get access to my devices for Google accounts and for AWS accounts. Amazon may improve their Android app over time, but by using open standards users can pick whatever works best for them at the time.

I love the fact that Amazon now supports Virtual MFA. I’ve already thrown away my hardware token and my pocket feels less full.

I love the fact that Amazon implemented this as a service based on existing standards so that I can use Google’s Android app to access my account.

I love open standards.

Update: I just found this great starting page which even links to Google Authenticator as a client for Android and iPhone:

http://aws.amazon.com/mfa/

Original article: http://alestic.com/2011/11/aws-virtual-mfa

Updated EBS boot AMIs for Ubuntu 8.04 Hardy on Amazon EC2 (2011-10-06)

2011-10-08T03:45:00Z

Canonical has released updated instance-store AMIs for Ubuntu 8.04 LTS Hardy on Amazon EC2. Read Ben Howard’s announcement on the ec2ubuntu Google group.

I have released corresponding EBS boot AMIs for Ubuntu 8.04 LTS Hardy using the exact same image Canonical used to build the instance-store AMIs.

You can find the AMI ids for Canonical’s AMIs and for the AMIs I just built in the table at the top of Alestic.com. Just click on the region you are using to see the ids.

The updated code I used to build the EBS boot AMIs by downloading Canonical’s images is available on github.

If you’re not already using the venerable Ubuntu Hardy, please start with a more modern release, like Ubuntu 10.04 LTS Lucid or Ubuntu 11.04 Natty.

Original article: http://alestic.com/2011/10/ec2-hardy-ebs

New Release of Alestic Git Server

2011-10-05T01:13:36Z

New AMIs have been released for the Alestic Git Server. Major upgrade points include:

Base operating system upgraded to Ubuntu 11.04 Natty
Git upgraded to version 1.7.4.1
gitolite upgraded to version 2.1 (master branch)

You can find the latest Alestic Git Server AMI ids and documentation at:

http://alestic.com/alestic-git/

Questions and comments are welcomed on the Alestic Git group.

Original article: http://alestic.com/2011/10/ec2-git-server-release

Using ServerFault.com for Amazon EC2 Q&A

2011-09-26T20:45:00Z

The Amazon EC2 Forum has been around since the beginning of EC2 and has always been a place where you can get your EC2 questions in front of an audience of experts and Amazon employees.

Though I’m still listed as one of the top posters (scored by questioners marking my answers as helpful) I’ve slowed down my activity on that forum due to the sheer volume of support requests, many of which are commonly asked questions or things that only Amazon can help with.

I’ve started answering Amazon EC2 related questions on Stack Exchange sites like Server Fault, a place where system administrators can get questions answered by experts. Here’s my profile on serverfault which linkes to answers I’ve posted. Click “newest” to see the most recent.

http://serverfault.com/users/30542/eric-hammond

If you use an RSS reader, you can follow a feed for all questions tagged amazon-ec2:

[RSS] http://serverfault.com/feeds/tag/amazon-ec2

You can even follow all new answers posted by an individual like, say, that Eric Hammond fellow:

[RSS] http://serverfault.com/feeds/user/30542

Original article: http://alestic.com/2011/09/ec2-serverfault

Rebooting vs. Stop/Start of Amazon EC2 Instance

2011-09-24T14:00:00Z

When you reboot a physical computer at your desk it is very similar to shutting down the system, and booting it back up. With Amazon EC2, rebooting an instance is much the same as with a local physical computer, but a stop/start differs in a few keys ways that may cause some problems and definitely have some benefits.

When you stop an EBS boot instance you are giving up the physical hardware that the server was running on and EC2 is free to start somebody else’s instance there.

Your EBS boot volume (and other attached EBS volumes) are still preserved, though they aren’t really tied to a physical or virtual server. They are just associated with an instance id that isn’t running anywhere.

When you start the instance again, EC2 picks some hardware to run it on, ties in the EBS volume(s) and boots it up again.

Things that change when you stop/start include:

New internal IP address (though could randomly be the same).
New external IP address (though could randomly be the same).
If an Elastic IP address was associated with the instance before it was stopped, then you’ll need to re-associate it after the start. (Behavior may differ with VPC instances.)
Any contents on the instance’s former ephemeral storage were wiped and you are given fresh ephemeral storage (often mounted as /mnt).
You can leave an instance stopped for as long as you like and not get charged for run time (though you do get charged at a much lower rate for the EBS volume storage). See the next point.
A fresh billing hour is started for the instance when you start it again. E.g., if you start a new instance and then stop/start it 3 times within the first 60 minutes, you’ll get charged for 4 hours instead of 1.
There is a small chance that EC2 will not have available slots of the correct instance type to run your instance when you want to start it again. I’ve had this happen and temporarily switched to a different, available instance type to get it running again.

When you reboot, it’s a simple reboot at the OS level and the instance stays running on the same hardware, with the same private and public IP addresses, keeps the same Elastic IP address (if associated), and keeps the same ephemeral storage without getting wiped. No new billing hour is started on a reboot and you do not give up the instance hardware.

While an instance is stopped, you can do some cool things before starting it again. Here’s an article I wrote on changing the instance type of an instance while it’s stopped:

Moving an EC2 Instance to a Larger Size

Here’s an article I wrote on how to change the size of an EBS boot disk of an instance while it’s stopped:

Resizing the Root Disk on a Running EBS Boot EC2 Instance

Here’s an article I wrote on how to examine the root disk of an instance (while it’s stopped) when you can’t connect to it while it’s runnning:

Fixing Files on the Root EBS Volume of an EC2 Instance

Since the stop/start cycle has a good chance of moving your instance to new hardware, it’s an easy way to replace your instance hardware if you suspect that the current platform might be going bad and causing problems. Here’s an article I wrote about that:

A Simpler Way To Replace Instance Hardware on EC2

Warning: “Stopping” an instance is completely different from “terminating” an instance! When you terminate an EC2 instance, by default it deletes the EBS boot volume and other volumes that were created at run time. Make sure you understand the difference before you start doing one or the other.

[I originally posted this article as an answer to a serverfault question, but liked it enough to archive it on this blog.]

Original article: http://alestic.com/2011/09/ec2-reboot-stop-start

Upper Limits on Number of Amazon EC2 Instances by Region

2011-08-15T14:00:00Z

[Update: As predicted, these numbers are already out of date and Amazon has added more public IP address ranges for use by EC2 in various regions.]

Each standard Amazon EC2 instance has a public IP address. This is true for normal instances when they are first brought up and for instances which have had elastic IP addresses assigned to them. Your EC2 instance still has a public IP address even if you have configured the security group so that it cannot be contacted from the Internet, which happens to be the default setting for security groups.

Amazon has made public the EC2 IP address ranges that may be in use for each region.

From this information, we can calculate the absolute upper limit for the number of concurrently running standard EC2 instances that could possibly be supported in each region. At the time of this writing I calculate the hard upper limits to be:

EC2 Region	Max Instances*
us-east-1	585,704
us-west-1	98,298
eu-west-1	135,156
ap-southeast-1	43,000
ap-northeast-1	34,808

*Caveats:

An upper limit based on the IP address ranges does not tell you what the real number of possible instances is in a given EC2 region. It’s just an upper limit.
Amazon is sure to keep requesting, reserving, and announcing more IP addresses than is actively needed by EC2 at any point in time. Only they know the growth buffer percentage that they like to maintain.
Amazon may need to use different ranges of IP addresses for different groups of instances in different parts of their network, even in the same data center or availability zone, so they may publish and start using new ranges of IP addresses even before they are even near using up the capacity of previous ranges.
Amazon is free to add new IP address blocks to the list at any time as they keep growing, and they do. The specific numbers above could be out of date by the time you read this.
There are probably some IP addresses in each range that are reserved for various networking devices and protocols.
This calculation is for concurrently running EC2 instances. Many instances run for just a few minutes or hours and another instance, perhaps for another customer, can start up and use that same IP address moments later.
EC2 instances running inside Amazon VPC don’t necessarily use up an external IP address in Amazon’s EC2 public IP address ranges, so they are not included in the upper limits.
I am not a networking expert. I only play one at my startup.

Anything else I’m missing?

[Update 2012-12-27: Correct URL for Amazon’s latest IP address list.]

Original article: http://alestic.com/2011/08/ec2-max-instances

Unavailable Availability Zones on Amazon EC2

2011-08-06T18:30:33Z

I’m taking a class about using Chef with EC2 by Florian Drescher today and Florian mentioned that he noticed one of the four availability zones in us-east-1 is not currently available for starting new instances.

I’ve confirmed this in my own AWS accounts and found that one of the three availability zones in us-west-1 is also unavailable in addition to one of the four availability zones in us-east-1.

Here’s the error I get when I try to start an instance in the availability zone using an old AWS account:

Client.Unsupported: The requested Availability Zone is no longer supported. Please retry your request by not specifying an Availability Zone or choosing us-east-1d, us-east-1a, us-east-1b.

When I use an AWS account I created two days ago, I don’t even see the fourth availability zone at all:

$ ec2-describe-availability-zones --region us-east-1
AVAILABILITYZONE    us-east-1b  available   us-east-1   
AVAILABILITYZONE    us-east-1c  available   us-east-1   
AVAILABILITYZONE    us-east-1d  available   us-east-1

The exact name of the unavailable availability zones will vary between EC2 accounts. You can read more about that here:

Matching EC2 Availability Zones Across AWS Accounts

The availability zones that are unavailable in my AWS accounts map to the following identifiers using the method described in the above article:

us-east-1x ceb6a579-757c-474b-b09b-52c84b605767
us-west-1x e5a2ff3b-79b4-4217-8c93-ebf1d633dd6e

If my guess is correct based on old accounts I have, I believe these may be the oldest (original) availability zones in their respective regions.

Has there been any communication from Amazon about unsupported availability zones? Is this temporary or permanent? When I searched Google for the above error, I got back one result in Japanese and it appears to be somebody asking what the error is.

No longer supporting an availability zone in EC2 is something that Amazon is allowed to do under the EC2 SLA, especially with the way that they seem to phase them out. The SLA does not kick in until two availability zones are completely unavailable and “unavailable” includes your existing instances having no external connectivity. This is one reason we try to architect services with the ability to quickly move resources from one availability zone to another.

I’d love to hear if other people are able to start instances in these availability zones. Please also mention if you already have instances running in those zones.

Update 2011-08-06: According to a post in May from Amazon this seems to be a normal part of how AWS grows in an orderly manner, and if you already have instances running in a zone, you should be able to continue running instances in that zone. It isn’t clear to me how quickly you might lose a zone after your last remaining instance is stopped or terminated, but according to one user it sounds like it might be nearly immediate.

Original article: http://alestic.com/2011/08/ec2-unavailabiilty-zones

Desktop AMI login security with NX

2011-08-03T12:25:42Z

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

They just sent me a notice that all of my old Ubuntu and Debian desktop AMIs from 2008-2010 need to be taken down, but it was a misunderstanding of how the NoMachine NX software implements login security and I’m sure those AMIs just got caught up in their automated sweeps.

I sent an email to help Amazon to help explain why a fixed public ssh key in the AMI with a publicly known “private” ssh key is not a security risk in this instance (see command=”/usr/NX/bin/nxserver —login” in the authorized_keys2 file) but they might be sending out notices to users of these AMIs before they get my response.

You can read up on how NX authentication works securely here:

http://www.nomachine.com/ar/view.php?ar_id=AR02C00150

It’s the middle of the night for me, so I’ll publish more later, but I wanted to get the world out just in case there is alarm.

Note: There may be other reasons you should not run some of those AMIs, e.g., if they are of a release that is past end of life, but they don’t have open back doors that let me access them.

I’ve written about building AMIs securely recently and have been preaching this kind of stuff with EC2 for years.

Original article: http://alestic.com/2011/08/ec2-desktop-ami-security

Updated EBS boot AMIs for Ubuntu 8.04 Hardy on Amazon EC2

2011-06-24T13:00:00Z

For folks still using the old, reliable Ubuntu 8.04 LTS Hardy from 2008, Canonical has released updated AMIs for use on Amazon EC2. Read Scott Moser’s announcement on the ec2ubuntu Google group.

Though Canonical publishes both EBS boot and instance-store for recent Ubuntu releases, they only publish instance-store AMIs for the older Ubuntu 8.04, so…

I have published EBS boot AMIs for Ubuntu 8.04 Hardy using the exact image Canonical uses for the instance-store AMIs.

I also used the same ext3 file system with the same file system label and the same AKI and the same ephemeral disk mounts, so these should be pretty much identical except that the root disk is based on EBS instead of instance-store. The default root disk size is 8GB instead of 10GB to follow the Amazon and Ubuntu standards for EBS boot instances.

The code used to build the EBS boot AMIs using Canonical’s downloadable Hardy images is available in github.

If you’re not already using Hardy and you are just getting started with Ubuntu on EC2, I would recommend jumping straight to the most recent Ubuntu release. At the time of this writing it is Ubuntu 11.04 Natty.

You can find the AMI ids for both EBS boot and instance store for all active releases of Ubuntu (almost all published by Canonical) in the table at the top of Alestic.com.

[Update 2011-06-25: Corrected github link]

Original article: http://alestic.com/2011/06/ec2-hardy-ebs

Creating Public AMIs Securely for EC2

2011-06-17T14:20:00Z

Amazon published a tutorial about best practices in creating public AMIs for use on EC2 last week:

How To Share and Use Public AMIs in A Secure Manner

Though the general principles put forth in the tutorial are good, some of the specifics are flawed in how to accomplish those principles. (Comments here relate to the article update from June 7, 2011 3:45 AM GMT.)

The primary message of the article is that you should not publish private information on a public AMI. Excellent advice!

Unfortunately, the article seems to recommend or at least to assume that you are building the public AMI by taking a snapshot of a running instance. Though this method seems an easy way to build an AMI and is fine for private AMIs, it is is a dangerous approach for public AMIs because of how difficult it is to identify private information and to clear that private information from a running system in such a way that it does not leak into the public AMI.

The article recommends the use of the rm command to remove files containing confidential files. This is inadequate if you are building an EBS boot AMI for public use by taking a snapshot of the volume, as the deleted files are likely to remain in the EBS block device and are copied into the snapshot and public AMI.

Deleted files

In September, 2009, I published an article that describes this danger with an example of an EBS snapshot containing deleted files. (An EBS boot AMI is simply an EBS snapshot that has been registered).

Hidden Dangers in Creating Public EBS Snapshots on EC2

Almost immediately after I published that article with a sample public EBS snapshot, a reader found and restored the deleted file and redeemed the $100 Amazon gift certificate I had hidden in it. Now, imagine that the deleted file contained your AWS credentials with power over your entire infrastructure on EC2, power to charge tens of thousands of dollars to your credit card with a single API call, and that person finding them does not have the best of intentions. Ouch.

If you are generating a public AMI by creating an EBS snapshot of a running instance, then the use of rm is not sufficient to clear and protect secret information. As much as I love wipe, shred, and srm, even attempting to overwrite blocks may not be sufficient with modern journaling file systems.

History files

Amazon recommends “Always delete the shell history before creating your AMI” with an example of an rm of dot history files. Unfortunately, depending on exactly when this is run and what you do between that command and the creation of the AMI, a history file might be silently re-created and included on your AMI including commands that you typed before removing the history file. This is true even disregarding the fact that a file removed with rm isn’t really removed if you are creating an EBS boot AMI with a snapshot.

For example, simply exiting a shell will cause bash to re-create .bash_history with all the commands entered into that shell, perhaps including the ones where you had passed AWS credentials to commands.

You can test this with a sequence like:

ssh to your server
echo "my secret"
rm .bash_history
exit
ssh to your server again
cat .bash_history

It looks like you removed your secret, but there it is!

There are ways of configuring bash to not create history files, but bash is not the only program that creates history files, and you may have multiple users to worry about, and some information gets stored in system log files, and… It just gets messy. With security, you don’t want to say “I probably got everything”. You want to know that you are not at risk.

authorized_keys

Amazon recommends removing all authorized_keys files from your disk before creating the AMI. This is a great recommendation because you don’t want to release public AMIs with back doors letting you access your user’s servers. It’s bad form and makes you look suspicious and untrustworthy once discovered.

Unfortunately, this makes it difficult to reconnect to your running instance, so you need to make sure you keep some ssh connections alive. You would need to restore the authorized_keys to copy files or create new ssh connections to the instance, and then hope you remember to remove them again before taking the next snapshot.

This becomes error prone if you are in the test/development loop trying to perfect an AMI.

The article recommends you remove any unrecognized authorized_keys files and user login accounts when you run a public AMI. I would go further. If you run a public AMI that includes back doors like this, you should question what other security problems might exist with that AMI and get off of it as soon as possible. In fact, Amazon sends out alerts and turns off access when they find AMIs with pre-existing authorized_keys so it’s a bit odd this article seems to take such a security hole casually.

So now you’re sufficiently worried and are wondering how to create public AMIs without including your secret information or accidentally releasing AMIs with embarrassing back doors.

Recommendation 1

The cleanest and safest approach is to build the file system for the new image separate from the file system you are using with the running system. This means that you are creating a chroot environment with a complete operating system in a directory structure that is not the same root directory where you are logging in and running commands.

The great thing about this approach is that you know exactly what files you are putting on the new image; you only run the exact commands you want to in the chroot environment; no history or log files show up there that you don’t control; your authorized_keys files are never seen in that file system; and any files you delete are not leaked when you copy that file system onto a new EBS volume to snapshot and register the public AMI.

This is a reasonably advanced Linux concept and sometimes requires some special tools and knowledge, but the tools and knowledge are publicly available and folks in the community are standing by to help guide you when you run into problems.

I think this approach is made easiest with Ubuntu, as Canonical has provided downloadable file system images that are configured correctly for the EC2 environment and that can be easily customized to add software and configuration for making AMIs to your own specification.

I published a tutorial on how to build AMIs with Canonical’s downloadable images back in 2009 with Ubuntu Karmic (now past end of life). It is similar with modern versions of Ubuntu, and if there is interest I can publish a new article with the updated steps.

The code I use to build the Alestic Git Server AMIs uses this approach with Ubuntu 10.04 Natty and is available as a resource to study:

alestic-git-build-ami

Recommendation 2

If you aren’t building an Ubuntu AMI and your Linux distro does not provide clean, downloadable file systems for EC2 images, and you really believe you can identify which files on your running system contain private information, and you really want to create a public AMI from a running system, then here is how you can avoid releasing an AMI with recoverable deleted files.

Start with a fresh instance of a clean public AMI to install and configure your software. If your instance has been running for a while, you really don’t know what has leaked into the file system in visible and deleted files
Put as little private information on the running system as possible. If you need to have AWS credentials on the running system, drop them in a mounted ephemeral store disk (/mnt on some distros). Don’t type private information like keys or passwords into command lines where they might get dropped into history files.
Delete all sensitive files and all authorized_keys. (This is still risky as you might miss files, and history files can sometimes reappear as described above.)

Do not snapshot the live EBS volume as it still contains the deleted files and you don’t want to make them public in the new AMI. Instead,
Create a new EBS volume, attach, and mount it on the running instance, say under /image
Copy the root file system over to the new EBS volume. This only copies the current view of the undeleted files and does not copy the blocks containing the deleted files or any other modified file information. The command might look something like:
```
rsync -axvSHAX / /image/
```
umount and detach the new EBS volume.
Create an EBS snapshot of the new EBS volume.
Register the EBS snapshot as a new AMI.

Test by running an instance of the new AMI and verifying it works and contains no private information before you change its attributes so the public can run it. You can then terminate instances and delete the extra EBS volume. Only the EBS snapshot is required to be kept for the new AMI.

Warning: Even though these instructions are under a section titled “Recommendation 2” I would not build public AMIs this way myself. It just seems too risky that something sensitive might slip into the public AMI. It’s better than snapshotting a running system directly because it removes the possibility deleted files leak out, but “Recommendation 1” is a safer path.

Wrapping up

I’m a bit surprised that the above article’s rm/snapshot and auhorized_keys instructions came from Amazon as I’ve seen their security folks and documentation warn against these very practices in the past. Amazon knows better as an organization and I hope that this misinformation is corrected soon so that people building AMIs don’t follow the guidance and leave secret information or back doors on public AMIs.

This is a complicated topic, as security related issues tend to be. I welcome comments, clarifications, and questions on the original post. Articles from Alestic.com are republished on a couple specific sites with explicit permission, but I only read and respond to comments on my primary blog.

[Update 2011-06-17: Corrected comments about authorized_keys. Amazon does recommend removing all of these files before publishing a new public AMI and recommends removing “unrecognized” authorized_keys files when running public AMIs.]

Original article: http://alestic.com/2011/06/ec2-ami-security

Canonical Releases Ubuntu 11.04 Natty for Amazon EC2

2011-04-28T16:35:35Z

As steady as clockwork, Ubuntu 11.04 Natty is released on the day scheduled at least eleven months ago; and thanks to Canonical, tested AMIs for Natty are already published for use on Amazon EC2.

The Ubuntu AMI table at the top of Alestic.com always reflects the latest official Ubuntu AMIs from Canonical and now includes Ubuntu 11.04 Natty.

Click on the EC2 region tab (e.g., us-west-1) to see the list of AMI ids. AMIs are listed for both 32-bit and 64-bit architectures (depending on what instance type you want to run) and for both EBS boot and instance-store. I strongly recommend EBS boot unless you have a specific situation that requires instance-store and you really know what you’re doing. EBS boot has many advantages over instance-store that you may not be aware of until after you need them.

As part of this release, I have added launch buttons to the right of each AMI id on Alestic.com. If you click on one of these buttons you will be taken to the AWS console and dropped directly into the process of launching an EC2 instance of that AMI. Thanks to Scott Moser for pointing out that AWS had implemented support for this cool feature—and for building the Ubuntu AMIs themselves :-)

I’ve been running Ubuntu Natty on my laptop for a couple weeks with good success and look forward to trying out this latest release of Ubuntu on EC2 as well.

Original article: http://alestic.com/2011/04/ec2-ubuntu-natty

EC2 Reserved Instance Offering IDs Change Over Time

2011-04-26T01:14:59Z

This article is a followup to Matching EC2 Availability Zones Across AWS Accounts written back in 2009. Please read that article first in order to understand any of what I write here.

Since I wrote that article, Amazon has apparently changed the reserved instance offering ids at least once. I haven’t been tracking this, so I don’t know if this was a one time thing or if the offering ids change on a regular basis.

Interestingly, the offering ids still seem to match across accounts and still map to different availability zones, so perhaps they can still be used to map the true, underlying availability zones as the original article proposes.

To document for posterity and comparison, here are my 2009 availability zone offering ids as calculated by the procedure in the above mentioned article:

“Blue” Account (2009):

eu-west-1a 649fd0c8-75d4-4e16-88c7-1ddb83f66062
eu-west-1b 438012d3-0440-480a-9f5c-eb7e55dd5a37
us-east-1a 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1b 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1c c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

“Red” Account (2009):

eu-west-1a 438012d3-0440-480a-9f5c-eb7e55dd5a37
eu-west-1b 649fd0c8-75d4-4e16-88c7-1ddb83f66062
us-east-1a c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1b 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1c 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

And here are the new, 2011-05-25 availability zone offering ids.

“Blue” Account (2011):

eu-west-1a c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
eu-west-1b d586503b-7025-44e8-8487-09907b6b0e7e
us-east-1a 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1b 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1c ceb6a579-757c-474b-b09b-52c84b605767
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

“Red” Account (2011):

eu-west-1a d586503b-7025-44e8-8487-09907b6b0e7e
eu-west-1b c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
us-east-1a ceb6a579-757c-474b-b09b-52c84b605767
us-east-1b 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1c 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

Note: I have excluded the regions and availability zones that Amazon has added since 2009 (quite a few).

Though all of the offering ids have changed since 2009, it appears that the availability zone mappings between these two accounts has remained the same across the years. This is a small sampling and I still think that Amazon could change this, especially for zones where no resources are being used (instances running, EBS volumes created), so don’t depend on it remaining so.

Please let me know if you do any further experimentation or future tracking of these ids. There’s not a ton of practical application, but I still find this interesting.

Original article: http://alestic.com/2011/04/ec2-offering-ids