Alestic.com

SCALE 8x Talk Notes: EC2 Beginners Workshop

2010-02-22T09:24:05Z

At SCALE 8x (Southern California Linux Expo, Feb 2010) I did a walkthrough demonstrating how to use the AWS console to run, connect to, and terminate your first Ubuntu server on Amazon EC2.

Though they do not include my talking points or the Q&A discussion during and following the session, you can download my slides and workshop handout:

http://cdn.anvilon.com/20100220-scale/scale8x-ec2.pdf

http://cdn.anvilon.com/20100220-scale/scale8x-steps.pdf

Notes:

These slides were not created for use outside of the workshop, so they are not complete in themselves. I am making them available for the workshop participants who may wish to refer to them to refresh their memory on what was discussed.
The instructions talk about removing the private ssh key at the end. This would not be done in a normal secure environment.
This demo used a temporary Ubuntu AMI which had wikimedia installed. This AMI is not designed for use in production systems as it includes a known wikimedia admin password.
This AMI is subject to being deleted at any time without notice. Did I mention you should not use it in a production system?

The BitSource Interview of Eric Hammond (SCALE, EC2)

2010-02-19T01:44:38Z

Matthew Sacks, of The BitSource, made the mistake of asking me some questions about Amazon EC2, so I rambled on far too long and the results are posted on the SCALE blog:

SCALE: Eric Hammond on Deploying Linux on EC2

In addition to some high level concepts, I point out that Ubuntu is, at the moment, the best choice for a Linux distro on EC2 if you want up-to-date images with modern, consistent kernels.

Resizing the Root Disk on a Running EBS Boot EC2 Instance

2010-02-11T03:47:55Z

In a previous article I described how to run an EBS boot AMI with a larger root disk size than the default. That’s fine if you know the size you want before running the instance, but what if you have an EC2 instance already running and you need to increase the size of its root disk without running a different instance?

As long as you are ok with a little down time on the EC2 instance (few minutes), it is possible to change out the root EBS volume with a larger copy, without needing to start a new instance.

Let’s walk through the steps on a sample Ubuntu 9.10 Karmic EBS boot instance. I tested this with ami-6743ae0e but check Alestic.com for the latest AMI ids.

On the instance we check the initial size of the root file system (15 GB):

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G  675M   14G   5% /

The following commands are all run on a system other than the one we are resizing. Pick a new size (in GB) that is larger than the current size:

instanceid=<YOURINSTANCEID>
size=20

Get the root EBS volume id and availability zone for this instance:

oldvolumeid=$(ec2-describe-instances $instanceid |
  egrep "^BLOCKDEVICE./dev/sda1" | cut -f3)
zone=$(ec2-describe-instances $instanceid | egrep ^INSTANCE | cut -f12)
echo "instance $instanceid in $zone with original volume $oldvolumeid"

Stop (not terminate!) the instance:

ec2-stop-instances $instanceid

Detach the original volume from the instance:

while ! ec2-detach-volume $oldvolumeid; do sleep 1; done

Create a snapshot of the original volume:

snapshotid=$(ec2-create-snapshot $oldvolumeid | cut -f2)
while ec2-describe-snapshots $snapshotid | grep -q pending; do sleep 1; done
echo "snapshot: $snapshotid"

Create a new volume from the snapshot, specifying a larger size:

newvolumeid=$(ec2-create-volume \
  --availability-zone $zone     \
  --size $size                  \
  --snapshot $snapshotid |
  cut -f2)
echo "new volume: $newvolumeid"

Attach the new volume to the instance:

ec2-attach-volume        \
  --instance $instanceid \
  --device /dev/sda1     \
  $newvolumeid
while ! ec2-describe-volumes $volumeid | grep -q attached; do sleep 1; done

Start the instance and find its new public IP address/hostname. (If you were using an elastic IP address, re-assign it to the instance.)

ec2-start-instances $instanceid
while ! ec2-describe-instances $instanceid | grep -q running; do sleep 1; done
ec2-describe-instances $instanceid

Connect to the instance with ssh (not shown) and resize the root file system to fill the new EBS volume:

# ext3 root file system (most common)
sudo resize2fs /dev/sda1

# XFS root file system (less common):
sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /

Show that the root file system is the new, larger size (20 GB):

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  679M   19G   4% /

Delete the old EBS volume and snapshot if you no longer need them, though I recommend you at least keep the snapshot for a while just in case:

ec2-delete-volume $oldvolumeid
ec2-delete-snapshot $snapshotid

Note: Since you manually created the new volume and attached it to the instance yourself, it will not be deleted automatically when the instance is terminated. You can modify the instance attributes to change the delete-on-termination flag for the volume if you wish.

New Ubuntu 8.04.3 Hardy AMIs for Amazon EC2

2010-01-30T05:38:11Z

Scott Moser (Canonical) built and released new Ubuntu 8.04.3 LTS Hardy images and AMIs for Amazon EC2. I also published new EBS boot AMIs using the same images. I’ve listed all of the AMI ids on http://alestic.com (pick your region at the top).

These AMIs should work better in the us-west-1 region (apt sources.list) and have updated software packages so upgrades on new instances should be faster.

Southern California Linux Expo - Februrary 19-21, 2010 at the Westin LAX

2010-01-29T01:59:46Z

The 8th Southern California Linux Expo (aka SCaLE 8x) is a community organized, non-profit event. Those words and the incredibly cheap price might lead you to believe that it is not worth going to, but if this is your first time you’ll be amazed by the size, scope, and professionalism of the event with nearly a hundred exhibits and dozens of informative talks.

Even though you’re not paying hundreds of dollars for the conference fee, it’s still worth traveling to if you’re not in Los Angeles. If you are in LA, then you have no excuse.

Just like last year at SCaLE, I will be leading another “Try-It Lab” where we’ll help folks get started with using Amazon EC2 and Ubuntu Linux. More information about preparation will be posted on the SCaLE blog, so be sure to review it before attending if you’re interested in a hands-on, guided, workshop experience with EC2. The lab seats “sold out” quickly last year, so make sure you get in early.

Deal for readers of Alestic.com: When you register for SCaLE, use the code “ERIC” for 50% off of the listed price. If you sign up today, that gives you a full access pass for a ridiculously low $35. Prices may go up as the weekend gets closer.

[Upate 2010-02-16: Link to preparation instructions on SCALE blog]

Public EBS Boot AMIs for Ubuntu on Amazon EC2

2010-01-25T14:33:58Z

If you’ve been following along, you probably know that I have been recommending that folks using EC2 switch to the official Ubuntu AMIs published by Canonical (Hardy or Karmic). I have been building and publishing Ubuntu AMIs since 2007 (including Dapper, Edgy, Feisty, Gutsy, Hardy, Intrepid, Karmic), but the last year my focus on this project has been to transition these responsibilities to Canonical who have more time and resources to support the initiative.

I’m happy to say that I’ve finally followed my own advice. For my personal Amazon EC2 servers (including for the Alestic.com web site) I am using Ubuntu 9.10 Karmic images published for EC2 by Canonical.

While I was making the transition, I also switched to EBS boot AMIs. However, since it sounds like Canonical is not planning to publish EBS boot AMIs until Lucid, I decided to continue in service to the community and make available EBS boot AMIs for running Ubuntu on EC2.

I have published EBS boot AMIs for Ubuntu 9.10 Karmic and Ubuntu 8.04 Hardy, both 32- and 64-bit architectures, in all current EC2 regions, for a total of a dozen new AMIs.

I chose to use the exact Ubuntu images which Canonical built for running Ubuntu on EC2. This means that these EBS boot AMIs work exactly the same as the official Canonical AMIs including ssh to the ubuntu user. Again, even though I’m publishing the EBS boot AMIs for Karmic and Hardy, the contents of the image were built by Canonical.

The EBS boot AMIs are listed on Alestic.com. I have restructured the table to better feature Canonical AMIs, and now you need to pick an EC2 region to see the IDs.

Give the EBS boot AMIs a spin and let me know if you run into any issues.

How to Report Bugs with Ubuntu on Amazon EC2: ubuntu-bug

2010-01-25T10:00:00Z

The official Ubuntu AMIs published by Canonical for EC2 starting in October have proven to be solid and production worthy. However, you may still on occasion run into an issue which deserves to be brought to the attention of the Ubuntu server team developing these AMIs and the software which enables Ubuntu integration with EC2.

The easiest, most efficient, and most complete way to report problems with Ubuntu on EC2 is to use the ubuntu-bug tool which comes pre-installed on all Ubuntu systems.

The ubuntu-bug command requires a single argument which is one of:

the name of an Ubuntu software package experiencing a problem,
the path to a program related to the problem,
the process id of the program experiencing the problem, or
the path of a crash file.

When reporting EC2 startup issues with an Ubuntu instance, the involved package is generally ec2-init so the command to run would be:

ubuntu-bug ec2-init

This command should be run on the EC2 instance that is experiencing the problem. The ubuntu-bug command will collect relevant information about the instance and file it with the bug report to assist in tracking down and correcting the issue.

If the instance with the problem is no longer running or accessible, try to run another instance of the same AMI to report the bug. This will help submit the correct AMI information with the bug report.

If ubuntu-bug reports “This is not a genuine Ubuntu package” you might have to first run

sudo apt-get update

and then try again.

Unfortunately, ubuntu-bug is an interactive program which does not accept command line options to set choices, so you will need to respond to a couple prompts and then copy and paste a URL it provides to you. First, it asks:

What would you like to do? Your options are:
  S: Send report (1.5 KiB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  C: Cancel
Please choose (S/V/K/C): S

Respond by hitting the “S” key because you really do want to report a problem.

ubuntu-bug then displays a URL and asks if you would like to launch a browser.

Choices:
  1: Launch a browser now
  C: Cancel
Please choose (1/C): C

Respond by hitting the “C” key as ubuntu-bug running on the EC2 instance can’t launch the web browser on your local system and you probably don’t want to use a terminal based browser.

Make a note of the URL displayed in:

*** To continue, you must visit the following URL:
  https://bugs.launchpad.net/ubuntu/+source/ec2-init/+filebug/LONGSTRINGHERE?

Copy the URL and paste it into your web browser. You will continue reporting the problem through your browser and the system information will be attached after you submit.

If this is the first time you have used Launchpad.net, you will be prompted to create an account. Use a valid email address as you will need to confirm it.

Launchpad will prompt you to enter a “Summary” which should be a short description of the bug. If it is not a duplicate of one of the bugs already entered, click “No I need to report a new bug” and enter the “Further Information”. Include as much information as possible relevant to the issue. If a developer can reproduce the bug using this description, then it will be addressed more easily.

For general information on submitting bugs in Ubuntu, please see:

https://help.ubuntu.com/community/ReportingBugs

You can see also see a current list of open ec2-images bugs.

If you are reporting Ubuntu on EC2 bugs directly using Launchpad without ubuntu-bug (not recommended) make sure you include the AMI id and tag the bug with “ec2-images”.

Note that ubuntu-bug is not a mechanism to support general support questions. One place to get help with running Ubuntu on EC2 is from the community in the ec2ubuntu Google group and there’s always the general Amazon EC2 forum. You can occasionally get live help with Ubuntu on EC2 on the #ubuntu-server IRC channel on irc.freenode.net

Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data

2010-01-11T19:48:55Z

Here are a few little-publicized benefits that were launched with Amazon EC2’s new EBS boot instances: You can lock them from being accidentally terminated; you can prevent termination even when you try to shutdown the server from inside the instance; and you can automatically save your data storage when they are terminated.

In discussions with other EC2 users, I’ve found that it is a common feeling of near panic when you go to terminate an instance and you check very carefully to make sure that you are deleting the right instance instead of an active production system. Slightly less common but even worse is the feeling of dread when you realize you just casually terminated the wrong EC2 instance.

It is always recommended to set up your AWS architecture so that you are able to restart production systems on EC2 easily, as they could, in theory, be lost at any point due to hardware failure or other actions. However, new features released with the EBS boot instances help reduce the risks associated with human error.

The following examples will demonstrate with the EC2 API command line tools ec2-run-instances, ec2-modify-instance-attribute, and ec2-terminate-instances. Since AWS is Amazon’s “web service” all of these features are also available through the API and should be coming available (if they aren’t already) in other tools, packages, and interfaces using the web service API.

1. Shutdown Behavior

First, let’s look at what happens when you run a command like the following in an EC2 instance:

sudo shutdown -h now
# or, equivalently and much easier to type:
sudo halt

Using the legacy S3 based AMIs, either of the above terminates the instance and you lose all local and ephemeral storage (boot disk and /mnt) forever. Hope you remembered to save the important stuff elsewhere!

A shutdown from within an EBS boot instance, by default, will initiate a “stop” instead of a “terminate”. This means that your instance is not in a running state and not getting charged, but the EBS volumes still exist and you can “start” the same instance id again later, losing nothing.

You can explicitly set or change this with the --instance-initiated-shutdown-behavior option in ec2-run-instances. For example:

ec2-run-instances                             \
  --instance-initiated-shutdown-behavior stop \
  [...]

This is the first safety precaution and, as mentioned, should already be built in, though it doesn’t hurt to document your intentions with an explicit option.

2. Delete on Termination

Though EBS volumes created and attached to an instance at instantiation are preserved through a “stop”/”start” cycle, by default they are destroyed and lost when an EC2 instance is terminated. This behavior can be changed with a delete-on-termination boolean value buried in the documentation for the --block-device-mapping option of ec2-run-instances.

Here is an example that says “Don’t delete the root EBS volume when this instance is terminated”:

ec2-run-instances                          \
  --block-device-mapping /dev/sda1=::false \
  [...]

If you are associating other EBS snapshots with the instance at run time, you can also specify that those created EBS volumes should be preserved past the lifetime of the instance:

  --block-device-mapping /dev/sdh=SNAPSHOTID::false

When you use these options, you’ll need to manually clean up the EBS volume(s) if you no longer want to pay for the storage costs after an instance is gone.

Note: EBS volumes attached to an instance after it is already running are, by default, left alone on termination (i.e., not deleted). The default rules are: If the EBS volume is created by the creation of the instance, then the termination of the instance deletes the volumes. If you created the volume explicitly, then you must delete it explicitly.

3. Disable Termination

This is my favorite new safety feature. Years ago, I asked Amazon for the ability to lock an EC2 instance from being accidentally terminated, and with the launch of EBS boot instances, this is now possible. Using ec2-run-instances, the key option is:

ec2-run-instances           \
  --disable-api-termination \
  [...]

Now, if you try to terminate the instance, you will get an error like:

Client.OperationNotPermitted: The instance 'i-XXXXXXXX' may not be terminated.
Modify its 'disableApiTermination' instance attribute and try again.

Yay!

To unlock the instance and allow termination through the API, use a command like:

ec2-modify-instance-attribute     \
  --disable-api-termination false \
  INSTANCEID

Then end it all with:

ec2-terminate-instances INSTANCEID

Oh, wait! did you make sure you unlocked and terminated the right instance?! :)

Note: disableApiTermination is also available when you run S3 based AMIs today, but since the instance can still be terminated from inside (shutdown/halt) I am moving towards EBS based instances for all around security.

Put It Together

Here’s a command line I just used to start up an EC2 instance of an Ubuntu 9.10 Karmic EBS boot AMI which I intend to use as a long-term production server with strong uptime and data safety requirements. I wanted to add all the protection available:

ec2-run-instances                                    \
  --key $keypair                                     \
  --availability-zone $availabilityzone              \
  --user-data-file $startupscript                    \
  --block-device-mapping /dev/sda1=::false           \
  --block-device-mapping /dev/sdh=$snapshotid::false \
  --instance-initiated-shutdown-behavior stop        \
  --disable-api-termination                          \
  $amiid

With regular EBS snapshots of the volumes, copies to off site backups, and documented/automated restart processes, I feel pretty safe.

Runtime Modification

If you have a valuable running EC2 instance, but forgot to specify the above options to protect it when you started it, or you are now ready to turn a test system into a production system, you can still lock things after the fact using the ec2-modify-instance-attribute command (or equivalent API call).

For example:

ec2-modify-instance-attribute --disable-api-termination true INSTANCEID
ec2-modify-instance-attribute --instance-initiated-shutdown-behavior stop INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sda1=::false INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sdh=::false INSTANCEID

Notes:

Only one type of option can be specified with each invocation.
The --disable-api-termination option has no argument value when used in in ec2-run-instances, but it takes a value of true or false in ec2-modify-instance-attribute.
You don’t specify the snapshot id when changing the delete-on-terminate state of an attached EBS volume.
You may change the delete-on-terminate to “true” for an EBS volume you created and attached after the instance was running. By default it will not be deleted since you created it.
The above --block-device-mapping options currently generate a Server.InternalError for me. I reported it to Amazon.

You can find out the current state of the options using ec2-describe-instance-attribute, which only takes one option at a time:

ec2-describe-instance-attribute --disable-api-termination INSTANCEID
ec2-describe-instance-attribute --instance-initiated-shutdown-behavior INSTANCEID
ec2-describe-instance-attribute --block-device-mapping INSTANCEID

Unfortunately, the block-device-mapping output does not currently show the state of delete-on-termination value, but thanks to Andrew Lusk for pointing out in a comment below that it is available through the API. Here’s a hack which extracts the information from the debug output:

ec2-describe-instance-attribute -v --block-device-mapping INSTANCEID | 
  perl -0777ne 'print "$1\t$2\t$3\n" while 
  m%<deviceName>(.*?)<.*?<volumeId>(.*?)<.*?<deleteOnTermination>(.*?)<%sg'

While we’re on the topic of EC2 safety, I should mention the well known best practice of separating development and production systems by using a different AWS account for each. Amazon lets you create and use as many accounts as you’d like even with the same credit card as long as you use a unique email address for each. Now that you can share EBS snapshots between accounts, this practice is even more useful.

What additional safety precautions do you take with your EC2 instances and data?

Building EBS Boot AMIs Using Canonical's Downloadable EC2 Images

2010-01-06T13:19:58Z

In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.

In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.

Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.

Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:

# 32-bit
instanceid=$(ec2-run-instances   \
  --key YOURKEYPAIR              \
  --availability-zone us-east-1a \
  ami-1515f67c |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"


# 64-bit
instanceid=$(ec2-run-instances   \
  --key YOURKEYPAIR              \
  --availability-zone us-east-1a \
  --instance-type m1.large       \
  ami-ab15f6c2 |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"

Wait for the instance to move to the “running” state, then note the public hostname:

while host=$(ec2-describe-instances "$instanceid" | 
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

rsync                            \
  --rsh="ssh -i YOURKEYPAIR.pem" \
  --rsync-path="sudo rsync"      \
  ~/.ec2/{cert,pk}-*.pem         \
  ubuntu@$host:/mnt/

Connect to the instance:

ssh -i YOURKEYPAIR.pem ubuntu@$host

Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get dist-upgrade -y &&
sudo -E apt-get install -y ec2-api-tools

Set up some parameters:

codename=karmic
release=9.10
tag=server
if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
  arch2=amd64
  ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
  ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
  arch=i386
  arch2=i386
  ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
  ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi

Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.

imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
image=/mnt/$codename-uec-$arch2.img
imagedir=/mnt/$codename-uec-$arch2
wget -O- $imagesource |
  sudo tar xzf - -C /mnt
sudo mkdir -p $imagedir
sudo mount -o loop $image $imagedir

[OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.

In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.

# Allow network access from chroot environment
sudo cp /etc/resolv.conf $imagedir/etc/
# Fix what I consider to be a bug in vmbuilder
sudo rm -f $imagedir/etc/hostname
# Add multiverse
sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
  $imagedir/etc/ec2-init/templates/sources.list.tmpl
# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
  sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
sudo chroot $imagedir \
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
sudo chroot $imagedir \
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
# Upgrade the system and install packages
sudo chroot $imagedir mount -t proc none /proc
sudo chroot $imagedir mount -t devpts none /dev/pts
cat <<EOF > $imagedir/usr/sbin/policy-rc.d
#!/bin/sh
exit 101
EOF
chmod 755 $imagedir/usr/sbin/policy-rc.d
DEBIAN_FRONTEND=noninteractive
sudo chroot $imagedir apt-get update &&
sudo -E chroot $imagedir apt-get dist-upgrade -y &&
sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
sudo chroot $imagedir umount /proc
sudo chroot $imagedir umount /dev/pts
rm -f $imagedir/usr/sbin/policy-rc.d

Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.

Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

size=15 # root disk in GB
now=$(date +%Y%m%d-%H%M)
prefix=ubuntu-$release-$codename-$tag-$arch-$now
description="Ubuntu $release $codename $tag $arch $now"
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
  cut -f2)
instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
sudo mkfs.ext3 -F /dev/sdi
ebsimage=$imagedir-ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage
sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
sudo umount $ebsimage
ec2-detach-volume "$volumeid"
snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
ec2-delete-volume "$volumeid"
while ec2-describe-snapshots "$snapshotid" | grep -q pending
  do echo -n .; sleep 1; done
ec2-register                   \
  --architecture $arch         \
  --name "$prefix"             \
  --description "$description" \
  $ebsopts                     \
  --snapshot "$snapshotid"

Depending on what you want to keep from the above process, there are various things that you might want to clean up.

If you no longer want to use an EBS boot AMI:
```
ec2-deregister $amiid
ec2-delete-snapshot $snapshotid
```
When you’re done with the original instance:
```
ec2-terminate-instance $instanceid
```

In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]

[Update 2010-01-22: New location for downloadable Ubuntu images.]

Building EBS Boot and S3 Based AMIs for EC2 with Ubuntu vmbuilder

2010-01-04T11:33:33Z

Here’s my current recipe for how to build an Ubuntu 9.10 Karmic AMI, either the new EBS boot or the standard S3 based, using the Ubuntu vmbuilder software. The Ubuntu vmbuilder utility replaces ec2ubuntu-build-ami for building EC2 images and it can build images for a number of other virtual machine formats as well.

There is a lot of room for simplification and scripting in the following instructions, but I figured I’d publish what is working now so others can take advantage of the research to date. Happy New Year!

Some sections are marked [For EBS boot AMI] or [For S3 based AMI] and should only be followed when you are building that type of AMI. The rest of the sections apply to either type. It is possible to follow all instructions to build both types of AMIs at the same time.

Run an instance of Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. I prefer the c1.* instance types to speed up the builds, but you can get by cheaper with the m1.* instance types. Make a note of the resulting instance id:

# 32-bit
instanceid=$(ec2-run-instances   \
  --key YOURKEYPAIR              \
  --availability-zone us-east-1a \
  --instance-type c1.medium      \
  ami-1515f67c |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"


# 64-bit
instanceid=$(ec2-run-instances   \
  --key YOURKEYPAIR              \
  --availability-zone us-east-1a \
  --instance-type c1.xlarge      \
  ami-ab15f6c2 |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"

Wait for the instance to move to the “running” state, then note the public hostname:

while host=$(ec2-describe-instances "$instanceid" | 
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

rsync                            \
  --rsh="ssh -i YOURKEYPAIR.pem" \
  --rsync-path="sudo rsync"      \
  ~/.ec2/{cert,pk}-*.pem         \
  ubuntu@$host:/mnt/

Connect to the instance:
```
ssh -i YOURKEYPAIR.pem ubuntu@$host
```
Install the image building software. We install the python-vm-builder package from Karmic, but we’re going to be using the latest vmbuilder from the development branch in Launchpad because it has good bug fixes. We also use the EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:
```
export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get upgrade -y &&
sudo -E apt-get install -y \
  python-vm-builder ec2-ami-tools ec2-api-tools bzr &&
bzr branch lp:vmbuilder
```
You can ignore the “Launchpad ID” warning from bzr.

Fill in your AWS credentials:

export AWS_USER_ID=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)

Set up parameters and create files to be used by the build process. The bucket value is only required for S3 based AMIs:

bucket=...
codename=karmic
release=9.10
tag=server
if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
  arch2=amd64
  pkgopts="--addpkg=libc6-i386"
  kernelopts="--ec2-kernel=aki-fd15f694 --ec2-ramdisk=ari-c515f6ac"
  ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
  ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
  arch=i386
  arch2=i386
  pkgopts=
  kernelopts="--ec2-kernel=aki-5f15f636 --ec2-ramdisk=ari-0915f660"
  ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
  ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi
cat > part-i386.txt <<EOM
root 10240 a1
/mnt 1 a2
swap 1024 a3
EOM
cat > part-x86_64.txt <<EOM
root 10240 a1
/mnt 1 b
EOM

Create a script to perform local customizations to the image before it is bundled. This is passed to vmbuilder below using the --execscript option:

cat > setup-server <<'EOM'
#!/bin/bash -ex
imagedir=$1
# fix what I consider to be bugs in vmbuilder
perl -pi -e "s%^127.0.1.1.*\n%%" $imagedir/etc/hosts
rm -f $imagedir/etc/hostname
# Use multiverse
perl -pi -e 's%(universe)$%$1 multiverse%' \
  $imagedir/etc/ec2-init/templates/sources.list.tmpl
# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
  tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
chroot $imagedir \
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
chroot $imagedir \
  sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
# Install packages
chroot $imagedir apt-get update
chroot $imagedir apt-get install -y runurl
chroot $imagedir apt-get install -y ec2-ami-tools
EOM
chmod 755 setup-server

Build the image:

now=$(date +%Y%m%d-%H%M)
dest=/mnt/dest-$codename-$now
prefix=ubuntu-$release-$codename-$arch-$tag-$now
description="Ubuntu $release $codename $arch $tag $now"
sudo vmbuilder/vmbuilder xen ubuntu       \
  --suite=$codename                       \
  --arch=$arch2                           \
  --dest=$dest                            \
  --tmp=/mnt                              \
  --ec2                                   \
  --ec2-version="$description"            \
  --manifest=$prefix.manifest             \
  --lock-user                             \
  --part=part-$arch.txt                   \
  $kernelopts                             \
  $pkgopts                                \
  --execscript ./setup-server             \
  --debug

[For S3 based AMI] include the following options in the vmbuilder command above. This does not preclude you from also building an EBS boot AMI with the same image. Make a note of the resulting AMI id output by vmbuilder:

  --ec2-bundle                            \
  --ec2-upload                            \
  --ec2-register                          \
  --ec2-bucket=$bucket                    \
  --ec2-prefix=$prefix                    \
  --ec2-user=$AWS_USER_ID                 \
  --ec2-cert=$EC2_CERT                    \
  --ec2-key=$EC2_PRIVATE_KEY              \
  --ec2-access-key=$AWS_ACCESS_KEY_ID     \
  --ec2-secret-key=$AWS_SECRET_ACCESS_KEY \

[For EBS boot AMI] Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

size=15 # root disk in GB
volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
  cut -f2)
instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
sudo mkfs.ext3 -F /dev/sdi
ebsimage=$dest/ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage
imageroot=$dest/root
sudo mkdir $imageroot
sudo mount -oloop $dest/root.img $imageroot
sudo tar -cSf - -C $imageroot . | sudo tar xvf - -C $ebsimage
sudo umount $imageroot $ebsimage
ec2-detach-volume "$volumeid"
snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
ec2-delete-volume "$volumeid"
while ec2-describe-snapshots "$snapshotid" | grep -q pending
  do echo -n .; sleep 1; done
ec2-register                   \
  --architecture $arch         \
  --name "$prefix"             \
  --description "$description" \
  $ebsopts                     \
  --snapshot "$snapshotid"

Depending on what you want to keep from the above process, there are various things that you might want to clean up.

If you no longer want to use an S3 based AMI:

ec2-deregister $amiid
ec2-delete-bundle                     \
  --access-key $AWS_ACCESS_KEY_ID     \
  --secret-key $AWS_SECRET_ACCESS_KEY \
  --bucket $bucket                    \
  --prefix $prefix

If you no longer want to use an EBS boot AMI:

ec2-deregister $amiid
ec2-delete-snapshot $snapshotid

When you’re done with the original instance:

ec2-terminate-instance $instanceid

In the above instructions I stray a bit from the defaults. For example, I add the runurl package from the Alestic PPA so that it is available for use in user-data scripts on first boot. I enable multiverse for easy access to more software, and I install ec2-ami-tools which works better for me than the current euca2ools.

I also set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance.

Explore and set options as you see fit for your applications. Go wild with the --execscript feature (similar to the ec2ubuntu-build-ami --script option) to customize your image.

The following vmbuilder options do not currently work with creating EC2 images: --mirror, --components, --ppa. I have submitted bug 502490 to track this.

As with much of my work here, I’m simply explaining how to use software that others have spent a lot of energy building. In this case a lot of thanks go to the Ubuntu server team for developing vmbuilder, the EC2 plugin, the ec2-init startup software, and the code which builds the official Ubuntu AMIs; especially Søren Hansen, Scott Moser, and Chuck Short. I also appreciate the folks who reviewed early copies of these instructions and provided feedback including Scott Moser, Art Zemon, Trifon Trifonov, Vaibhav Puranik, and Chris.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

Call for testers (building EBS boot AMIs with Ubuntu vmbuilder)

2010-01-01T11:21:29Z

I’m polishing up an article about how to build images from scratch with Ubuntu vmbuilder, both for S3 based AMIs and for EBS boot AMIs. Since nobody is paying any attention this new year’s weekend (except for you seven and you know who you are) I figured I’d wait until Monday or so to publish the article.

However, if you have nothing better to do this long weekend and you’d like a preview copy of the article, drop me a note with your email address. All I ask for in return is that you review (and perhaps even test) the instructions and send me feedback where things aren’t clear or don’t work.

ec2-consistent-snapshot release 0.1-9

2009-12-18T01:49:21Z

Thanks to everybody who submitted bug reports and feature requests for ec2-consistent-snapshot, software which can be used to create consistent EBS snapshots on Amazon EC2 especially for use with XFS and/or MySQL.

A new release (0.1-9) has been published to the Alestic PPA. This release provides the following fixes and enhancements:

Read MySQL “host” parameter from .my.cnf if provided. (closes: bug#485978)
Support quoted values in .my.cnf (closes: bug#454184)
Require Net::Amazon::EC2 version 0.11 or higher. (closes: bug#490686)
Require xfsprogs package. (closes: bug#493420)
Replace “/etc/init.d/mysql stop” with “mysqladmin shutdown”. (closes: bug#497557)
Document —description option which was added earlier. (closes: bug#487692)

If you already have ec2-consistent-snapshot installed, you can upgrade using commands like:

sudo apt-get update
sudo apt-get install ec2-consistent-snapshot

If you don’t yet have the Alestic PPA set up, run these commands first:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main"|
  sudo tee /etc/apt/sources.list.d/alestic-ppa.list    
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

If you find bugs or think of feature requests, please submit them.

Listing Recent Prices for EC2 Spot Instances

2009-12-14T10:00:10Z

The new spot instances on EC2 are a great way to get some extra compute power at a price you can live with, especially if you are flexible on exactly when the instances run. On the other hand maybe you won’t get the compute power if the spot instance price never drops to the max price you are willing to pay.

The best way to approach auction type situations like this is often to simply list the maximum price you can afford. Your instance(s) will get run if and when the spot instance price reaches that price and you will regularly get charged less depending on what other users are bidding for their instances.

Though I don’t recommend trying to chase the spot instance price around, it is natural to be curious about what others have been paying and whether or not you might have a chance to get in with your bid.

The ec2-describe-spot-price-history command lists the historical values of the spot instance price for all sizes and types of instances. This is useful for studying fluctuations over time, but sometimes you’ll just want to know what the latest spot instance price is, even though this is not necessarily the price you would get charged if you opened new spot instance requests.

The following command shows the most recent spot instance price for each size of the Linux instances:

ec2-describe-spot-price-history -d Linux/UNIX |
  sort -rk3 |
  perl -ane 'print "$F[1]\t$F[3]\n" unless $seen{$F[3]}++' |
  sort -n

To check the spot price in a different region, simply add an option like

--region us-west-1

--region eu-west-1

As of the writing of this article, the prices returned for the us-east-1 region are as follows:

0.026   m1.small
0.05    c1.medium
0.11    m1.large
0.25    c1.xlarge
0.265   m1.xlarge
0.443   m2.2xlarge
0.997   m2.4xlarge

The exact spot instance prices will vary significantly from these samples over time and likely even during the course of a day. In fact, the spot instance price may occasionally exceed the on demand (standard) EC2 instance price.

Who’s going to be the first person to provide handy, real time, spot price history graphs by EC2 region, instance class (Linux/Windows), and instance type with correct X-axis date/time scaling?

Good luck bidding!

Increasing Root Disk Size of an "EBS Boot" AMI on EC2

2009-12-11T10:25:49Z

Amazon EC2’s new EBS Boot feature not only provides persistent root disks for instances, but also supports root disks larger than the previous limit of 10GB under S3 based AMIs.

Since EBS boot AMIs are implemented by creating a snapshot, the AMI publisher controls the default size of the root disk through the size of the snapshot. There are a number of factors which go into deciding the default root disk size of an EBS boot AMI and some of them conflict.

On the one hand, you want to give users enough free space to run their applications, but on the other hand, you don’t want to increase the cost of running the instance too much. EBS volumes run $0.10 to $0.11 per GB per month depending on the region, or about $10/month for 100GB and $100/month for 1TB.

I suspect the answer to this problem might be for AMI publishers to provide a reasonable low default, perhaps 10GB as per the old standard or 15GB following in the footsteps of Amazon’s first EBS Boot AMIs. This would add $1.00 to $1.50 per month to running the instance which seems negligible for most purposes. Note: There are also IO charges and charges for EBS snapshots, but those are more affected by usage and less by the size of the original volume.

For applications where the EBS boot AMI’s default size is not sufficient, users can increase the root disk size at run time all the way up to 1 TB. Here’s a quick overview of how to do this.

Example

The following demonstrates how to run Amazon’s getting-started-with-ebs-boot AMI increasing the root disk from the default of 15GB up to 100GB.

Before we start, let’s check to see the default size of the root disk in the target AMI and what the device name is:

$ ec2-describe-images ami-b232d0db
IMAGE   ami-b232d0db    amazon/getting-started-with-ebs-boot    amazon  available   public      i386    machine aki-94c527fd    ari-96c527ff        ebs
BLOCKDEVICEMAPPING  /dev/sda1       snap-a08912c9   15

We can see the EBS snapshot id snap-a08912c9 and the fact that it is 15 GB attached to /dev/sda1. If we start an instance of this AMI it will have a 15 GB EBS volume as the root disk and we won’t be able to change it once it’s running.

Now let’s run the EBS boot AMI, but we’ll override the default size, specifying 100 GB for the root disk device (/dev/sda1 as seen above):

ec2-run-instances                       \
  --key KEYPAIR                         \
  --block-device-mapping /dev/sda1=:100 \
  ami-b232d0db

If we check the EBS volume mapped to the new instance we’ll see that it is 100GB, but when we ssh to the instance and check the root file system size we’ll notice that it is only showing 15 GB:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G  1.6G   13G  12% /

There’s one step left. We need to resize the file system so that it fills up the entire 100 GB EBS volume. Here’s the magic command for ext3. In my early tests it took 2-3 minutes to run:

$ sudo resize2fs /dev/sda1
resize2fs 1.40.4 (31-Dec-2007)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 7
Performing an on-line resize of /dev/sda1 to 26214400 (4k) blocks.
The filesystem on /dev/sda1 is now 26214400 blocks long.

Finally, we can check to make sure that we’re running on a bigger file system:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              99G  1.6G   92G   2% /

Note: The output reflects “99” instead of “100” because of slight differences in how df and EBS calculate “GB” (e.g., 1024 MB vs 1000 MB).

XFS

If it were possible to create an EBS boot AMI with an XFS root file system, then the resizing would be near instantaneous using commands like

sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /

The Ubuntu kernels built for EC2 by Canonical have XFS support built in, so XFS based EBS boot AMIs might be possible. This would also allow for more consistent EBS snapshots.

Toolset

Make sure you are running the latest version of the ec2-run-instances command. The current version can be determined with the command

ec2-version

To use EBS boot features, the version should be at least 1.3-45772.

Updated 2009-12-11: Switch instructions to default us-east-1 since all regions now support this feature.

Ubuntu Karmic Desktop on EC2

2009-11-16T10:42:41Z

As Thilo Maier pointed out in comments on my request for UDS input, I have been publishing both server and desktop AMIs for running Ubuntu on EC2 up through Jaunty, but the official Karmic AMIs on EC2 only support server installations by default.

Ubuntu makes it pretty easy to install the desktop software on a server, and NX from NoMachine makes it pretty easy to access that desktop remotely, with near real-time interactivity even over slowish connections.

Here’s a quick guide to setting this up, starting with an Ubuntu 9.10 Karmic AMI on Amazon EC2:

Create a user-data script which installs runurl (not on Karmic AMIs by default) and then runs a simple desktop and NX server installation script. Examine the desktop script to see what it’s doing to install the software.
```
cat <<EOM >install-desktop
#!/bin/bash -ex
wget -qO/usr/bin/runurl run.alestic.com/runurl
chmod 755 /usr/bin/runurl
runurl run.alestic.com/install/desktop
EOM
```
Start an instance on EC2 telling it to run the above user-data script on first boot. The following example uses the current 32-bit Karmic server AMI. Make sure you’re using the latest AMI id.
```
ec2-run-instances                   \
  --key YOURKEY                     \
  --user-data-file install-desktop  \
  ami-1515f67c
```
Connect to the new instance and wait for it to complete the desktop software installation (when sshd is restarted). This takes about 30 minutes on an m1.small instance and 10 minutes on a c1.medium instance. Then generate and set a secure password for the ubuntu user using copy/paste from the pwgen output. Save the secure password so you can enter it into the NX client later.
```
ssh -i YOURKEY.pem ubuntu@THEHOST
tail -f /var/log/syslog | egrep --line-buffer user-data:
pwgen -s 16 1
sudo passwd ubuntu
```
If anybody knows how to use ssh keys with NX, I’d love to do this instead of using passwords.
Back on your local system, install and run the NX client. For computers not running Ubuntu, download the appropriate software from NoMachine.
```
wget http://64.34.161.181/download/3.4.0/Linux/nxclient_3.4.0-5_i386.deb
sudo dpkg -i nxclient_3.4.0-5_i386.deb
/usr/NX/bin/nxclient --wizard &
```
Point the NX Client to the external hostname of your EC2 instance. Enter the Login “ubuntu” and the Password from above. Choose the “Gnome” desktop.

If all goes well, you should have a complete and fresh Ubuntu desktop filling most of your screen, available for you to mess around with and then throw away.

ec2-terminate-instances INSTANCEID

If you want to have a persistent desktop with protection from crashes, you’ll need to learn how to do things like placing critical directories on EBS volumes.

If you’d like to run KDE on EC2, replace the package “ubuntu-desktop” with “kubuntu-desktop” in the installation script.