Building EBS Boot AMIs Using Canonical's Downloadable EC2 Images

By Eric Hammond Jan 6, 2010 EC2 Ubuntu

In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.

In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.

Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.

Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:

 # 32-bit
 instanceid=$(ec2-run-instances   \
   --key YOURKEYPAIR              \
   --availability-zone us-east-1a \
   ami-1515f67c |
   egrep ^INSTANCE | cut -f2)
 echo "instanceid=$instanceid"

 # 64-bit
 instanceid=$(ec2-run-instances   \
   --key YOURKEYPAIR              \
   --availability-zone us-east-1a \
   --instance-type m1.large       \
   ami-ab15f6c2 |
   egrep ^INSTANCE | cut -f2)
 echo "instanceid=$instanceid"

Wait for the instance to move to the “running” state, then note the public hostname:

 while host=$(ec2-describe-instances "$instanceid" | 
   egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
 echo host=$host

Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

 rsync                            \
   --rsh="ssh -i YOURKEYPAIR.pem" \
   --rsync-path="sudo rsync"      \
   ~/.ec2/{cert,pk}-*.pem         \
   ubuntu@$host:/mnt/

Connect to the instance:

 ssh -i YOURKEYPAIR.pem ubuntu@$host

Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

 export DEBIAN_FRONTEND=noninteractive
 echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
   sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
 sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
 sudo apt-get update &&
 sudo -E apt-get dist-upgrade -y &&
 sudo -E apt-get install -y ec2-api-tools

Set up some parameters:

 codename=karmic
 release=9.10
 tag=server
 if [ $(uname -m) = 'x86_64' ]; then
   arch=x86_64
   arch2=amd64
   ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
   ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
 else
   arch=i386
   arch2=i386
   ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
   ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
 fi

Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.

 imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
 image=/mnt/$codename-$tag-uec-$arch2.img
 imagedir=/mnt/$codename-uec-$arch2
 wget -O- $imagesource |
   sudo tar xzf - -C /mnt
 sudo mkdir -p $imagedir
 sudo mount -o loop $image $imagedir

[OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.

In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.

 # Allow network access from chroot environment
 sudo cp /etc/resolv.conf $imagedir/etc/
 # Fix what I consider to be a bug in vmbuilder
 sudo rm -f $imagedir/etc/hostname
 # Add multiverse
 sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
   $imagedir/etc/ec2-init/templates/sources.list.tmpl
 # Add Alestic PPA for runurl package (handy in user-data scripts)
 echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
   sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
 sudo chroot $imagedir \
   apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
 # Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
 echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
   sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
 sudo chroot $imagedir \
   apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
 # Upgrade the system and install packages
 sudo chroot $imagedir mount -t proc none /proc
 sudo chroot $imagedir mount -t devpts none /dev/pts
 cat <<EOF > $imagedir/usr/sbin/policy-rc.d
 #!/bin/sh
 exit 101
 EOF
 chmod 755 $imagedir/usr/sbin/policy-rc.d
 DEBIAN_FRONTEND=noninteractive
 sudo chroot $imagedir apt-get update &&
 sudo -E chroot $imagedir apt-get dist-upgrade -y &&
 sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
 sudo chroot $imagedir umount /proc
 sudo chroot $imagedir umount /dev/pts
 rm -f $imagedir/usr/sbin/policy-rc.d

Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.

Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

 size=15 # root disk in GB
 now=$(date +%Y%m%d-%H%M)
 prefix=ubuntu-$release-$codename-$tag-$arch-$now
 description="Ubuntu $release $codename $tag $arch $now"
 export EC2_CERT=$(echo /mnt/cert-*.pem)
 export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
 volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
   cut -f2)
 instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
 ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
 while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
 sudo mkfs.ext3 -F /dev/sdi
 ebsimage=$imagedir-ebs
 sudo mkdir $ebsimage
 sudo mount /dev/sdi $ebsimage
 sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
 sudo umount $ebsimage
 ec2-detach-volume "$volumeid"
 snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
 ec2-delete-volume "$volumeid"
 while ec2-describe-snapshots "$snapshotid" | grep -q pending
   do echo -n .; sleep 1; done
 ec2-register                   \
   --architecture $arch         \
   --name "$prefix"             \
   --description "$description" \
   $ebsopts                     \
   --snapshot "$snapshotid"

Depending on what you want to keep from the above process, there are various things that you might want to clean up.

If you no longer want to use an EBS boot AMI:
```
 ec2-deregister $amiid
 ec2-delete-snapshot $snapshotid
```
When you’re done with the original instance:
```
 ec2-terminate-instance $instanceid
```

In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]

[Update 2010-01-22: New location for downloadable Ubuntu images.]

[Update 2010-03-26: Path tweak, thanks to paul.]

Show Comments

Alestic.com - A Personal AWS Blog by Eric Hammond

A Personal AWS Blog

by Eric Hammond