Firstly, a word on security. I’ve been careful to ensure that there’s no trace of my machines or public keys on these images by not adding them in the first place. Many of the CentOS AMIs I tried out had entries in the last log from previous logins, and some even had root passwords set. Using the kickstart file I created, the root account on the machine is locked and no password based logins are allowed over SSH.

Anyway, here’s what I did:

Create suitable packages

An AMI isn’t so useful without cloud-init, so I rebuilt it for el6. I also rebuilt ec2-ami-tools, modifying the udev script so that devices appear where they’re supposed to. Without it, a drive specified as sda will appear as sde, because the EL6 Xen kernel reserves the first four drive letters. If you’re interested, take a look at the SRPMs.

Create the instance-store image

Install the image

I did this on a remote machine I had handy running Xen, although there’s no reason you couldn’t do this on a local machine with KVM – there’s no separate -xen kernel with EL6, Xen is fully merged in. I used a remote machine because it was in a data centre with a very speedy Internet connection, something that’s very useful when you’re going to be uploading large images.

I created the image as 5GB. You can use any size you like from 1GB-10GB for an instance store image, but remember that the large it is, the longer it’ll take to start the machine. Plus you’ll be using ephemeral or EBS storage for data on the machine anwyay.
I created a kickstart file which installs the image and carries out the necessary modifications. Specifically it sorts out networking by removing mac addresses from the ifcfg-eth0 file and udev, labels the partition and fixes the fstab, and modifies the grub config. Anyway, enough of the chat, on with the install:

$ sudo virt-install -n "centos-ami" -r 1024 --nographics -l http://mirrors.melbourne.co.uk/centos/6/os/x86_64/ -x "ks=http://bashton.com/downloads/centos-ami/ami-kickstart.cfg" -f /dev/dsk/centosami --noreboot

Select a suitable kernel

We’re going to be using the EL6 kernel within the image, so we use the ‘PV Grub’ loader within Amazon to boot this.
You can find a suitable kernel by running:

$ ec2-describe-images -a -F image-type=kernel -F manifest-location=*pv-grub*

For instance store images, use an image tagged hd0. For EBS backed images, use an image tagged hd00. Note that the Amazon Kernel Images (AKIs) are region specific.

Bundle the image and upload the image to S3

We need to expose the first (and only) partition of the image:

$ kpartx -a /dev/dsk/centosami

Now we bundle this into an image using the kernel we selected above. This will write the image files into ~bundle

$ ec2-bundle-image -r x86_64 -d ~/bundle -p myimagename -u 561795456677 -k ~/amazonkey.pem --kernel aki-825ea7eb -c amazoncert.pem --image /dev/mapper/centosamip1

Now we upload it to a S3 bucket:

$ ec2-upload-bundle -b mybucketname -m ~/bundle/myimagename.manifest.xml -a S3_ACCESS_KEY -s S3_SECRET

And register it as an AMI:

$ ec2-register mybucketname/myimagename.manifest.xml

Create the EBS backed image

We need an EC2 instance to write to the EBS volume, so spin one up. This can use the instance store AMI you created in step 1, but it doesn’t have to.

Create a volume and attach

Again, I’m creating an EBS image of 5GB. You can create EBS root images of up to 100GB, so do whatever suits you.
Make sure you create the volume in the same availability zone as the EC2 instance you just started up.

$ ec2-create-volume -s 5 -z eu-west-1a
$ ec2-attach-volume vol-abcdwxyz -i i-1234abcd -d sdg

Partition and format the volume

We’re going to create two partitions, a root and a small swap. The small swap is useful on t1.micro instances, where there is no ephemeral storage.

$ fdisk /dev/sdg
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd94fa889.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Calling ioctl() to re-read partition table.
Syncing disks.

Format the partition and label, then set fsck never to be run on boot

$ sudo mkfs.ext4 -L '/' /dev/sdg1
$ sudo tune2fs -c 0 -i 0 /dev/sdg1

Format and label swap

$ sudo mkswap -L 'ebs-swap' /dev/sdg2


Copy disk image across

We need to copy the disk image across. You can either use instance storage to temporarily store this image, or create an EBS volume. Either way, make a directory that’s writeable by the ec2-user.
On the machine you did the bare install on for the instance store:

$ dd if=/dev/mapper/centosamip1 of=- bs=1m | ssh -C ec2-user@newawsinstance.amazonaws.com "cat - > /media/ephemeral0/img/theimage.img"

Copy image files

Mount the image file as loopback
$ sudo mkdir /loop
$ sudo mount -o loop /media/ephemeral0/theimage.img /loop


Mount the destination EBS partition and copy the files across

$ sudo mount /dev/sdg1 /mnt
$ sudo rsync -avHx /loop/ /mnt

Modify grub config and add EBS swap to fstab


$ sudo sed -i -e 's/hd0/hd0,0/' /mnt/boot/grub/menu.lst
$ echo "LABEL=ebs-swap none swap sw 0 0" | sudo tee -a /mnt/etc/fstab

We’re done! Unmount the image.


$ umount /mnt


Create a snapshot

Detach the volume and create a snapshot:


$ ec2-detach-volume vol-abcdwxyz
$ ec2-create-snapshot -d "My really great AMI snapshot" vol-abcdwxyz


Find a suitable kernel to use, selecting an ‘hd00′ one this time:

$ ec2-describe-images -a -F image-type=kernel -F manifest-location=*pv-grub*

Last step – register your image!

$ ec2-register -b "/dev/sda=snap-12345678::false" -b "/dev/sdb=ephemeral0" -n "My Great AMI" -a x86_64 --kernel aki-12345678

This post is prompted by this question posted on Twitter. At the closing talk given by Domas of Facebook, he mentioned that one should use FORCE INDEX when optimising for performance.

Why is this?

If you look at the manual page explaining how MySQL uses indexes you’ll see that they are broadly selected based on which will find the smallest number of rows. This means that as additional data is added to a table the ‘best’ index use to can change.

Using FORCE INDEX tells MySQL to use a specific index (or indexes) for a query. By using it, you’re avoiding the need to calculate which index to use for each query. You’re also ensuring (or at least making it more likely) that the indexes that are most used stay in the buffer pool, thus removing the need to read them from disk. MySQL also gets the opportunity to replace infrequently used indexes with data in the buffer pool.

I suspect that the benefits of FORCE INDEX don’t apply so much for those with smaller data sets, but when one has as massive a set of data as Facebook, many small optimisations make a big difference.

Update: See also this post by the big man himself

Download it here.

On other systems we’ve used supervisor to launch the Django fastcgi process, but on Ubuntu 10.04, and now on Red Hat/CentOS 6, we have the wonderful upstart system built in which provides everything we need.

Below, you can see the upstart config I’ve written to start fastcgi. It assumes you have python-flup installed.
We have one of these per Django project hosted on the machine – you’ll need to vary the port number specified for each project you have. All you’ll need to alter otherwise is the path to run in, and the path to manage.py. If you’re running under EL6, you’ll probably want to change www-data to apache, but beware that I haven’t actually tested this config in such an environment.
Save the following in “/etc/init/djangoproject.conf”, where djangoproject is the name of your django project.


description "Basic Django fcgi Upstart config"
author "Sam Bashton"

respawn
chdir /var/www/djangoproject
exec su -s /bin/sh -c 'exec "$0" "$@"' www-data -- \
/var/www/djangoproject/manage.py runfcgi \
method=prefork host=127.0.0.1 port=32769 minspare=4 maxspare=8 \
maxchildren=128 maxrequests=65535 daemonize=false


Once you’ve added the file, you can start it with the command “start djangoproject”, stop it with “stop djangoproject”, and restart it (for example, if you’ve updated the code) with “restart djangoproject”, replacing djangoproject with the name you gave the config of course.
Should the fastcgi process die for some reason it will be automatically be restarted.

Day to day you’ll be working with a wide range of Open Source software, but we’ve deliberately not included a list of apps – if you’re the right person for us, you’ll be able to pick up new technologies very quickly. The interview process will contain a full technical assessment.

If you’re interested, you can see more details and apply by visiting our job page. Please note that we’re really, really not interested from hearing from recruitment agencies and the like.

Red Hat 5 was getting rather long in the tooth, and I’m sure many users will be happy to see the inclusion of more up to date packages – MySQL 5.1, Python 2.6 and PHP 5.3 amongst the more notable that will have been giving some users of RHEL5 headaches.

Red Hat 6 provides much more than just a version bump to packages though, there are some key changes. KVM is now the default virtualisation technology. Xen is still supported, and RHEL6 will work as a RHEL5 guest out of the box, as long as one uses an ext3 /boot partition. The para-virt drivers for VMWare are also included, making those deployments simpler too.

ext4 is now the default file system, which provides support for file systems up to 1 exabyte in size and files up to 16TB. Red Hat only officially supported file systems of up to 8TB in size on RHEL5, although it was possible to make them larger than this. ext4 also brings much better performance, and removes the limit of 32000 subdirectories in a single directory.

Upstart is now used for the boot process, replacing the old /etc/init.d scripts. Upstart will be familiar to those who have used more recent versions of Ubuntu Linux, where the project was started. It provides support for automatically respawning failed daemons – something that will be music to the ears of anyone who has ever tried to log into a machine that has had the SSH daemon killed by the out of memory killer. It also always for parallel execution of startup scripts, meaning that RHEL6 boots significantly faster than previous versions.

One other piece of good news for sysadmins is that sendmail is now no longer the default MTA – I can almost hear the cheers of readers from here! Postfix is now provided by default, with the distinct advantage of configuration files that are easily distinguishable from line noise.

Overall, Red Hat Enterprise Linux 6 is a massive step forward from the previous version – as you’d expect, given RHEL5 was released in 2007.

Do the S100 and S300 Software RAID controllers in my Dell PowerEdge Server work with Red Hat Enterprise Linux?

To paraphrase, the solution is to take the RAID card out the server and throw it in the bin. The S300 isn’t a real RAID card, it’s a fakeraid. You’re better off just using Linux software RAID.

Unfortunately, the cabling in the server doesn’t allow for connecting to the on-board SATA ports, or rather for powering them. Speaking to Dell support, they initially suggested that just disabling the card in the BIOS would be sufficient; they later admitted it isn’t, and offered to send out suitable cables.

I seem to remember Dell making a big deal of their entire server line offering full Linux support some time ago, so I’m surprised to see them making this retrograde step – particularly as Linux represents a large percentage of the server market. In Dell’s defence they do make it reasonably clear that the S300 card isn’t supported under Linux at the ordering stage.

Red Hat Enterprise releases have been traditionally 18-24 months apart, and on that basis one might have expected RHEL6 to have been released sometime between October 2008 and March 2009, as RHEL5 was released in March of 2007. So what’s taking Red Hat so long, and when can we expect RHEL6?

There are a number of reasons for the delay, but the biggest is related to the Fedora Project, who create Fedora, on which Red Hat Enterprise is based.
For Fedora releases up to version 6, on which RHEL5 was based, the Fedora Project was run very much as an in-house Red Hat project, with contributions from the wider Open Source community. Post FC6, the Fedora Project was much more run by the Open Source community, with Red Hat contributors. This seems to have led to a shift in focus from fixing bugs to implementing new features. This isn’t necessarily a bad thing – Fedora exists to be the ‘bleeding edge’ of development, pushing at the boundaries in a way Red Hat are obviously unable to do with an enterprise focussed product like RHEL. Unfortunately, Fedora 9, on which RHEL6 was to be based, released with a large number of critical bugs and stability issues. Red Hat made the decision that they therefore couldn’t base RHEL6 on this release, and instead poured man-power into the Fedora Project, fixing bugs and generally improving stability as much as possible.

The evidence is that RHEL6 will now be based on Fedora 12, which was released in November of last year. The was a gap of just under a year between Fedora 6 and RHEL5, so this would point to a release in Autumn of this year.

The other issue that has caused delays to RHEL is that of virtualisation. RHEL4/RHEL5 utilised the Xen hypervisor for their virtualisation. In September 2008 Red Hat purchased Qumranet, whose staff included the leaders of the KVM project, an alternate virtualisation technology. KVM is very much seen as the future for Linux virtualisation, and Red Hat immediately made it clear that this was the direction they would be pursuing. RHEL 5.4 included KVM as a ‘technology preview’, so clearly much progress has been made integrating it into the main Red Hat release. It’s therefore likely that this is no longer holding up RHEL6.

I expect the final release date for RHEL6 to be announced at the Red Hat summit at the end of June. There have been some suggestions that RHEL6 itself will be released at the summit. Unless Red Hat release a beta within the next couple of weeks, I’d say this looks distinctly unlikely.