How to increase the size of an Amazon EC2 EBS Instance (with no downtime)
0I'm often asked by people who read my posts on AWS, how they can increase the hard drive size of their EBS instance. i.e. how do they change the size of their / partition.
Luckily, for most systems with up-to-date software runnning on them, this is actually quite easy to achieve.
The following method worked for me but please be careful when doing this on your own system as it could cause irretrievable loss of data.
As a result create a full backup before attempting this.
The following assumes you are only running one instance that you want to upgrade, but the principles apply whatever. It was carried out on an Amazon Linux 64 bit image (originally 8GB, resized to 250GB) which had all software updated via yum update before starting.
Take a backup and create a load balanced system - if you're not already using one
- Create an AMI of your current running instance, this acts as a perfect backup of your current system and allows to deploy a copy to test this process with.
- Launch a new instance from the AMI you just made
- Create an ELB with this new instance behind it and update your app, DNS whatever to filter traffic through the ELB
- When you're sure your new instance is taking the traffic for your app add the existing instance to the load balancer, so you're now load balancing across both instances. Having two instances to work with allows us to avoid down time as instance two can take the traffic while we work on instance one.
Resize the hard disk
- Stop the instance: AWS Management Console -> EC2 -> Instances, Right click instance name and choose "Stop"
- Note down the name of the volume attached to the instance: AWS Management Console -> EC2 -> Instances, click the instance name and click the link next to "Block Devices" in the panel (e.g. labelled sda1) which gives you the detailes of the selected instance. Note down the "EBS ID" value (e.g. vol-12a3599a).
- Create a snapshot of the current HDD: AWS Management Console -> EC2 -> Elastic Block Store -> Volumes, search for the EBS ID you noted down, right click it and choose create snapshot. Also note down the device name from the Attachment information column, this will be something like /dev/sda1
- Switch to the snapshots list (AWS Management Console -> EC2 -> Elastic Block Store -> Snapshots) and look for the one with a status of pending - this is your new snapshot being created. Note down the Snapshot ID and wait for the operation to complete.
- Go back to the volumes list: AWS Management Console -> EC2 -> Elastic Block Store -> Volumes, and hit create volume. Set the disk size you want and select the snapshot from the list with the ID you noted down.
- Wait for the volume to be created (you should be able to sort the volume list by status and see one which is available to find it if you have a lot of volumes).
- Disassociate the existing volume from your instance: Right click the volume you created the snapshot from and choose disassociate.
- Attach the new bigger volume to your instrance: Right click the availible volume you created and choose associate, select the instance you want to attach it to from the list and enter the device name as that which you noted down earlier (e.g. /dev/sda1). Note though this may be the original device name, when started back up this may be renamed to something /dev/xvda1 on the instance itself due to the way modern linux kernels reference the devices.
- Start your instance: AWS Management Console -> EC2 -> Instances, Right click instance name and choose "Start"
- ssh into your instance and confirm that the instance now has more space availible. To do this run fdisk -l /dev/xvda1 (replacing /dev/xvda1 with your device name). You should get something back like this:Disk /dev/xvda1: 268.4 GB, 268435456000 bytes
255 heads, 63 sectors/track, 32635 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000 - However if you run something like df -H you'll see that the partition still currently has the size originally allocated to it, so we have to expand the partition to cover the additional space
Expand Your Partition
- Check your instance file system type by running: fsck -N /dev/xvda1
- If it is one of ext2, ext3, or ext4 you can use resize2fs to expand the filesystem into the availible space. You can do this with the instance running if your file system is ext3 or ext4 and you're using kernel version 2.6+ by simply executing something like this (this example resizes to 240Gb):resize2fs /dev/xvda1 240G
- Other filesystem types have different on-line (running instance) resizing capabilities and you should check the man pages for your filesystem to check which you should use.
- After a short amount of time your instance should be able to access the extra space. At this point I generally like to reboot my instance to ensure clean running but this is probably unnecessary.
- Add the instance back into your load balancer if it's not there already and when happy remove or update the other instance and remove the ELB if required.
Google Insights Charts
0I've just noticed that you can now embed charts from Google insights e.g. this one that compares intrest in the BBC, ITV, Channel 4, and Channel 5...
Interesting huh? I wonder what else this could be used for...
AWS Rolling Thunder In CentOS Linux
1I've been using a common technique to deploy source code updates to my Amazon EC2 instances for some time now, which makes use of S3 as a central source code file-store, and syncs updates out to instances on a restart.
I didn't however know that this technique had a name: Rolling Thunder (Thanks AWS Tech Summit)
As it was such a cool buzz wordy term, I thought I'd post a guide to how I achieve Rolling Thunder on AWS using CentOS Linux based instances (though this should work with any *NIX variant) in case anyone new to the game wanted a how-to.
The method is as follows
- Clone a running live instance of your web app to use a test server
- Set up a private S3 bucket in which to store the source code for your web app(s)
- Zip up your source code into an archive file (this makes the permissions management of the source code of multiple apps, or versions of multiple apps easier on S3 BUT you could simply use the bucket to store an export of the latest version of your code)
- Upload your source to this bucket using the AWS web tool or (better) setup a sync using a tool such as cyberduck Remember to set permissions appropriately on all uploaded files i.e. don't make anything public, but don't remove your rights to access the uploaded content either!
- Log into your test instance and install the latest version of the fuse filesystem using the following instructions (these are all cli commands):
yum remove fuse fuse* fuse-devel yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap cd /usr/local/src wget "https://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz?r=&ts=1299709935&use_mirror=cdnetworks-us-1" tar -xzvf fuse-2.8.4.tar.gz rm fuse-2.8.4.tar.gz mv fuse-2.8.4 fuse cd fuse/ ./configure --prefix=/usr make make install export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ ldconfig modprobe fuse #confirm that 2.8.4 is the version displayed pkg-config --modversion fuse #set the fuse files system to load on startup (make the kernel module load) echo modprobe fuse >> /etc/rc.modules chmod +x /etc/rc.modules
- Install s3fs, using the following instructions (these are all cli commands):
wget http://s3fs.googlecode.com/files/s3fs-1.40.tar.gz (get URL for latest version) tar -xzvf s3fs-X.XX.tar.gz rm s3fs-X.XX.tar.gz mv s3fs-X.XX s3fs cd s3fs ./configure --prefix=/usr make make install
- Create an /etc/passwd-s3fs file, and add a single line like this:
accessKeyId:secretAccessKey
Obviously replacing accessKeyId:secretAccessKey with access credentials for your AWS account
- chmod this file to 640 for security (s3fs won't work unless permissions are set at this level)
- Test that you can mount the s3 bucket as a filesystem folder by running the following commands (these are all cli commands):
mkdir /thunder #test if /thunder is a mountpoint (it shouldn't be) mountpoint /thunder #mount the bucket /usr/bin/s3fs my-bucketname /thunder #test again (it should be a mountpoint now) mountpoint /thunder #try listing the source files you uploaded to the bucket ls /thunder #unmount the bucket umount /thunder #show that the files no longer exist in /thunder ls /thunder
Note: as a test you might want to try writing a file inside /thunder too to do this use your favourite text editor or an program like touch, and then look for the file's existence in s3 throught he web interface, or cyberduck, or whatever
- Next install Rsync (again, this is a cli command):
yum install rsync
- Then write a script to use rsync to update your web-app's code directory e.g. /var/www/html/ with the source code from the s3 bucket. Here is an example of the kind of script I use:
#!/bin/sh ############ # Config # ############ archive_name=my-sourcecode-archive.zip ################# # Main script # ################# #output the start time echo 'Started: ' date; #make a backup of your existing web app directory #this is just a precaution but can be useful for versioning deployments on the instances ts=`date '+%d-%m-%Y_%H-%M-%S'` cp -R /var/www/html /var/www/html_rt_bak_$ts #initialise the fuse filesystem modprobe fuse #check to see if /thunder is a mountpoint, if not mount it mountpoint /thunder/ || /usr/bin/s3fs source-a /thunder #copy and extract the zip file into /tmp/thunder, then remove the original archive mkdir -p /tmp/thunder cp /thunder/$archive_name /tmp/thunder/ umount /thunder cd /tmp/thunder/ unzip $archive_name rm /tmp/thunder/$archive_name #remove any stray .svn files/folders find . -name ".svn" -type d -exec rm -rf {} \; #rsync the contents of /thunder with /var/www/html rsync -avI --exclude-from=/tmp/thunder/thunder-exclude-list.txt /tmp/thunder/ /var/www/html/ #remove the temp dir, and the exclude list rm -rf /tmp/thunder #chown and chmod everything to ensure synced files have the correct permissions chown -R apache:apache /var/www/html chmod -R 754 /var/www/html #output the end time echo 'Finsihed: ' date;Note: The rsync exclude file. This is a file I put into my web app home directory that allows me to exclude various files from the rsync process, which can be useful to avoid copying files such as database connection settings or the source code archive itself into the web root. By making the exclude list a file in the web app, control of what's synced can be easily managed by the developer on upload of the source code to S3 rather than requiring extra edits to server side scripts on every instance.I tend to store my scripts in a /scripts directory and in my case I've called this one rolling-thunder.sh Remember to run a command such as this to make the script executable once written:
chmod +x /scripts/rolling-thunder.sh
- Next add this script to the startup sequence for your instance. There are a number of ways of doing this, but I choose to add it to the /etc/rc.local file (which is for executing user commands after the main system has started up). All you need to do is open /etc/rc.local in a text editor like nano and add a line like this to call your shell script:
/scripts/rolling-thunder.sh > /var/log/rolling-thunder.log 2>&1
Note: The additional line after the call to the script
> /var/log/rolling-thunder.log 2>&1
Creates and sends all output caused by the script (including errors) to a log file called rolling-thunder.log, which can be useful for debugging any errors that may occur
- Reboot the test instance and take a look at your rolling-thunder.log file and check your web app code, your update should have been deployed smoothly!
- Last, create an image of this instance to allow you to fire up further instances as necessary.
Notes:
- s3fs seems to start to choke when a single directory contains more than 1000 files, I think this is because of the time it takes to produce the list and parse it. Another good reason to package the code up into an archive
- Example thunder-exclude-list.txt ** are standard wildcard matches:
*my-sourcecode-archive.zip* *thunder-exclude-list.txt* */some/specific-file-to/exclude.php* */some-folder-to-exclude/*
- A better deployment option for easy management/automated instances is to break the rolling-thunder.sh script in two and have the main part which does the sync sitting on s3 or in the deployment archive and a minimal script sitting on the server which calls that script. This route would give yet more flexibility/control on the deployment front from a central location as opposed to having to edit the deployment script on multiple servers