I'm often asked by people who read my posts on AWS, how they can increase the hard drive size of their EBS instance. i.e. how do they change the size of their / partition.

Luckily, for most systems with up-to-date software runnning on them, this is actually quite easy to achieve.

The following method worked for me but please be careful when doing this on your own system as it could cause irretrievable loss of data.

As a result create a full backup before attempting this.

The following assumes you are only running one instance that you want to upgrade, but the principles apply whatever. It was carried out on an Amazon Linux 64 bit image (originally 8GB, resized to 250GB) which had all software updated via yum update before starting.

Take a backup and create a load balanced system - if you're not already using one

1. Create an AMI of your current running instance, this acts as a perfect backup of your current system and allows to deploy a copy to test this process with.
2. Launch a new instance from the AMI you just made
3. Create an ELB with this new instance behind it and update your app, DNS whatever to filter traffic through the ELB
4. When you're sure your new instance is taking the traffic for your app add the existing instance to the load balancer, so you're now load balancing across both instances. Having two instances to work with allows us to avoid down time as instance two can take the traffic while we work on instance one.

Resize the hard disk

1. Stop the instance: AWS Management Console -> EC2 -> Instances, Right click instance name and choose "Stop"
2. Note down the name of the volume attached to the instance: AWS Management Console -> EC2 -> Instances, click the instance name and click the link next to "Block Devices" in the panel (e.g. labelled sda1) which gives you the detailes of the selected instance. Note down the "EBS ID" value (e.g. vol-12a3599a).
3. Create a snapshot of the current HDD: AWS Management Console -> EC2 -> Elastic Block Store -> Volumes, search for the EBS ID you noted down, right click it and choose create snapshot. Also note down the device name from the Attachment information column, this will be something like /dev/sda1
4. Switch to the snapshots list (AWS Management Console -> EC2 -> Elastic Block Store -> Snapshots) and look for the one with a status of pending - this is your new snapshot being created. Note down the Snapshot ID and wait for the operation to complete.
5. Go back to the volumes list: AWS Management Console -> EC2 -> Elastic Block Store -> Volumes, and hit create volume. Set the disk size you want and select the snapshot from the list with the ID you noted down.
6. Wait for the volume to be created (you should be able to sort the volume list by status and see one which is available to find it if you have a lot of volumes).
7. Disassociate the existing volume from your instance: Right click the volume you created the snapshot from and choose disassociate.
8. Attach the new bigger volume to your instrance: Right click the availible volume you created and choose associate, select the instance you want to attach it to from the list and enter the device name as that which you noted down earlier (e.g. /dev/sda1). Note though this may be the original device name, when started back up this may be renamed to something /dev/xvda1 on the instance itself due to the way modern linux kernels reference the devices.
9. Start your instance: AWS Management Console -> EC2 -> Instances, Right click instance name and choose "Start"
10. ssh into your instance and confirm that the instance now has more space availible. To do this run fdisk -l /dev/xvda1 (replacing /dev/xvda1 with your device name). You should get something back like this:Disk /dev/xvda1: 268.4 GB, 268435456000 bytes
    255 heads, 63 sectors/track, 32635 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00000000
11. However if you run something like df -H you'll see that the partition still currently has the size originally allocated to it, so we have to expand the partition to cover the additional space

Expand Your Partition

1. Check your instance file system type by running: fsck -N /dev/xvda1
2. If it is one of ext2, ext3, or ext4 you can use resize2fs to expand the filesystem into the availible space. You can do this with the instance running if your file system is ext3 or ext4 and you're using kernel version 2.6+ by simply executing something like this (this example resizes to 240Gb):resize2fs /dev/xvda1 240G
3. Other filesystem types have different on-line (running instance) resizing capabilities and you should check the man pages for your filesystem to check which you should use.
4. After a short amount of time your instance should be able to access the extra space. At this point I generally like to reboot my instance to ensure clean running but this is probably unnecessary.
5. Add the instance back into your load balancer if it's not there already and when happy remove or update the other instance and remove the ELB if required.

Related posts:

1. How to mount an Amazon EBS disk as a drive in Linux (CentOS) These instructions on how to mount an Amazon EBS...
2. Amazon CloudFront Streaming In WordPress With JW Media Player I'm writing this post because a simple search on...
3. How To Add Export To Excel To Internet Explorer Note: This tutorial is only for people comfortable with...

I'm writing this post because a simple search on how to achieve CloudFront video streaming in wordpress using the jw media player plugin returns all kinds of results which are irrelevant if you just want to serve basic cloudfront streams!

So, here's how to do it (assuming you already know how to setup cloudfront to stream a video):

1. Install the latest version of the JW Player Plugin (Plugins -> Add New -> Search for "JW Player Plugin for WordPress")
2. Activate the plugin, you'll notice that a new menu called "JW Player" appears in your admin area
3. Click on JW Player, you'll get a list of the players which currently exist (each player being a different customisation of the default JW Media Player)
4.  Click the copy button next to the out-of-the-box player to create a new player you can customise. Rember to give the new player a meaningful name instead of the default "out-of-the-box_copy". In my case I'm going to call this player "Test"
5.  Edit any of the player settings you like to create your new player (but don't click save - yet - if you do just click the edit button next to your new custom player to re-open the interface)
6. Click the Advanced Settings tab of the player editor, scroll down the page until you see the streaming section and enter "rtmp" (without the quotes) in the provider box and the url to your clodfront streamer in the streamer box. e.g. if I setup a streaming distribution with the cname "videos.coderchris.com" my streamer url would be "rtmp://videos.coderchris.com:1935/cfx/st" that said you could just use the cloudfront default url for your distribution e.g. "hjh4887yh7h7.cloudfront.net" which would have the this streamer url: "rtmp://hjh4887yh7h7.cloudfront.net:1935/cfx/st"
7. Now JW player knows where to stream your video files from you need to embed them in the page, which is best done using the jwplayer shorcode. So assuming I uploaded a video called "My Test File.mp4" to the S3 bucket feeding the cloudfront distribution and have set it's permissions to be public then I should be able to enter the following shortcode into a post or page "[jwplayer config="Test" file="My Test File.mp4"]" and that's it!

Related posts:

1. JW Media Player – A Godsend For Flash Streaming So, you want or need to stream multimedia content from...
2. Meld WordPress And Facebook Together With Wordbook Now this is a pretty neat plugin: Wordbook allows...
3. WordPress 2.3 has been released WordPress 2.3 has been released and I can already...

I've been using a common technique to deploy source code updates to my Amazon EC2 instances for some time now, which makes use of S3 as a central source code file-store, and syncs updates out to instances on a restart.

I didn't however know that this technique had a name: Rolling Thunder (Thanks AWS Tech Summit)

As it was such a cool buzz wordy term, I thought I'd post a guide to how I achieve Rolling Thunder on AWS using CentOS Linux based instances (though this should work with any *NIX variant) in case anyone new to the game wanted a how-to.

The method is as follows

1. Clone a running live instance of your web app to use a test server
2. Set up a private S3 bucket in which to store the source code for your web app(s)
3. Zip up your source code into an archive file (this makes the permissions management of the source code of multiple apps, or versions of multiple apps easier on S3 BUT you could simply use the bucket to store an export of the latest version of your code)
4. Upload your source to this bucket using the AWS web tool or (better) setup a sync using a tool such as cyberduck Remember to set permissions appropriately on all uploaded files i.e. don't make anything public, but don't remove your rights to access the uploaded content either!
5. Log into your test instance and install the latest version of the fuse filesystem using the following instructions (these are all cli commands):

   yum remove fuse fuse* fuse-devel
   
   yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap
   
   cd /usr/local/src
   
   wget "https://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz?r=&ts=1299709935&use_mirror=cdnetworks-us-1"
   
   tar -xzvf fuse-2.8.4.tar.gz
   
   rm fuse-2.8.4.tar.gz
   
   mv fuse-2.8.4 fuse
   
   cd fuse/
   
   ./configure --prefix=/usr
   
   make
   
   make install
   
   export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
   
   ldconfig
   
   modprobe fuse
   
   #confirm that 2.8.4 is the version displayed
   
   pkg-config --modversion fuse
   
   #set the fuse files system to load on startup (make the kernel module load)
   
   echo modprobe fuse >> /etc/rc.modules
   
   chmod +x /etc/rc.modules
   

6. Install s3fs, using the following instructions (these are all cli commands):

   wget http://s3fs.googlecode.com/files/s3fs-1.40.tar.gz (get URL for latest version)
   
   tar -xzvf s3fs-X.XX.tar.gz
   
   rm s3fs-X.XX.tar.gz
   
   mv s3fs-X.XX s3fs
   
   cd s3fs
   
   ./configure --prefix=/usr
   
   make
   
   make install
   
   

7. Create an /etc/passwd-s3fs file, and add a single line like this:

   accessKeyId:secretAccessKey
   

   Obviously replacing accessKeyId:secretAccessKey with access credentials for your AWS account

8. chmod this file to 640 for security (s3fs won't work unless permissions are set at this level)
9. Test that you can mount the s3 bucket as a filesystem folder by running the following commands (these are all cli commands):

   mkdir /thunder
   
   #test if /thunder is a mountpoint (it shouldn't be)
   mountpoint /thunder
   
   #mount the bucket
   /usr/bin/s3fs my-bucketname /thunder
   
   #test again (it should be a mountpoint now)
   mountpoint /thunder
   
   #try listing the source files you uploaded to the bucket
   ls /thunder
   
   #unmount the bucket
   umount /thunder
   
   #show that the files no longer exist in /thunder
   ls /thunder
   

   Note: as a test you might want to try writing a file inside /thunder too to do this use your favourite text editor or an program like touch, and then look for the file's existence in s3 throught he web interface, or cyberduck, or whatever

10. Next install Rsync (again, this is a cli command):

    yum install rsync
    

11. Then write a script to use rsync to update your web-app's code directory e.g. /var/www/html/ with the source code from the s3 bucket. Here is an example of the kind of script I use:

    #!/bin/sh
    
    ############
    #  Config  #
    ############
    
    archive_name=my-sourcecode-archive.zip
    
    #################
    #  Main script  #
    #################
    
    #output the start time
    echo 'Started: '
    date;
    
    #make a backup of your existing web app directory
    #this is just a precaution but can be useful for versioning deployments on the instances
    ts=`date '+%d-%m-%Y_%H-%M-%S'`
    cp -R /var/www/html /var/www/html_rt_bak_$ts
    
    #initialise the fuse filesystem
    modprobe fuse
    
    #check to see if /thunder is a mountpoint, if not mount it
    mountpoint /thunder/ || /usr/bin/s3fs source-a /thunder
    
    #copy and extract the zip file into /tmp/thunder, then remove the original archive
    mkdir -p /tmp/thunder
    cp /thunder/$archive_name /tmp/thunder/
    umount /thunder
    
    cd /tmp/thunder/
    unzip $archive_name
    rm /tmp/thunder/$archive_name
    
    #remove any stray .svn files/folders
    find . -name ".svn" -type d -exec rm -rf {} \;
    
    #rsync the contents of /thunder with /var/www/html
    rsync -avI --exclude-from=/tmp/thunder/thunder-exclude-list.txt /tmp/thunder/ /var/www/html/
    
    #remove the temp dir, and the exclude list
    rm -rf /tmp/thunder
    
    #chown and chmod everything to ensure synced files have the correct permissions
    chown -R apache:apache /var/www/html
    chmod -R 754 /var/www/html
    
    #output the end time
    echo 'Finsihed: '
    date;
    
    

    Note: The rsync exclude file. This is a file I put into my web app home directory that allows me to exclude various files from the rsync process, which can be useful to avoid copying files such as database connection settings or the source code archive itself into the web root. By making the exclude list a file in the web app, control of what's synced can be easily managed by the developer on upload of the source code to S3 rather than requiring extra edits to server side scripts on every instance.I tend to store my scripts in a /scripts directory and in my case I've called this one rolling-thunder.sh Remember to run a command such as this to make the script executable once written:

    chmod +x /scripts/rolling-thunder.sh
    

12. Next add this script to the startup sequence for your instance. There are a number of ways of doing this, but I choose to add it to the /etc/rc.local file (which is for executing user commands after the main system has started up). All you need to do is open /etc/rc.local in a text editor like nano and add a line like this to call your shell script:

    /scripts/rolling-thunder.sh > /var/log/rolling-thunder.log 2>&1
    

    Note: The additional line after the call to the script

     > /var/log/rolling-thunder.log 2>&1
    

    Creates and sends all output caused by the script (including errors) to a log file called rolling-thunder.log, which can be useful for debugging any errors that may occur

13. Reboot the test instance and take a look at your rolling-thunder.log file and check your web app code, your update should have been deployed smoothly!
14. Last, create an image of this instance to allow you to fire up further instances as necessary.

Notes:

• s3fs seems to start to choke when a single directory contains more than 1000 files, I think this is because of the time it takes to produce the list and parse it. Another good reason to package the code up into an archive
• Example thunder-exclude-list.txt ** are standard wildcard matches:

  *my-sourcecode-archive.zip*
  *thunder-exclude-list.txt*
  */some/specific-file-to/exclude.php*
  */some-folder-to-exclude/*
  

• A better deployment option for easy management/automated instances is to break the rolling-thunder.sh script in two and have the main part which does the sync sitting on s3 or in the deployment archive and a minimal script sitting on the server which calls that script. This route would give yet more flexibility/control on the deployment front from a central location as opposed to having to edit the deployment script on multiple servers

Related posts:

1. How to mount an Amazon EBS disk as a drive in Linux (CentOS) These instructions on how to mount an Amazon EBS...
2. PHP5 Universal File Download Class After finding different hosting companies having wildly different policies...
3. Apache Live Log (ALiveLog) A really useful tool for providing simple stylised views...

The following two presentations are really useful for people just getting started with hosting websites on the Amazon AWS infrastructure.

These presentations were from guest speakers, the take home points for me were that Eagle Genomics have security tested AWS hosting and believe it to be secure enough for the kinds of sensitive data they are dealing with and that Servicetick were able to make good use of the data processing facilities

Related posts:

1. Amazon ELB SSL Termination IS Secure (According to Amazon) Today I contacted Amazon about a question that's been...
2. Server Administration I have been administering Linux and FreeBSD servers in...
3. Development Though now a much more operations focused individual I...

When trying to evaluate the security of AWS over the last few years this document would have been REALLY useful.

It's a couple of years old now and there may well be a more up to date version out there, but I've just stumbled up on it so I thought I'd repost it here for anyone else who needs a reference/is looking into AWS structure and security for their applications.

http://s3.amazonaws.com/aws_blog/AWS_Security_Whitepaper_2008_09.pdf

Related posts:

1. Some Recent Stuff on Web Application Security I subscribe to the Security Focus web application security...
2. Excel SYLK: File format is not valid error I came across this error earlier this week so...
3. How to mount an Amazon EBS disk as a drive in Linux (CentOS) These instructions on how to mount an Amazon EBS...

These instructions on how to mount an Amazon EBS volume apply to CentOS Linux specifically but with little modification can be applied to all Linux distributions. By attaching EBS volumes (aka. disks) to your instance you can get around the majority file space issues that might encounter when hosting popular websites or those which contain a lot of data (e.g. image galleries, music sites, podcasts, etc.)

1. Note down the instance id of the instance you want to add more storage to
2. In your AWS account go to the volumes link (under Elastic Block Store) click the Create Volume Button and choose how big you want this device to be.
3. Attach the new EBS volume to your instance by right clicking it and choosing Attach Volume. Select the instance id from the list that you noted down in step 1. and give the new device a name to reference it on the instance, e.g. /dev/sdf
4. Login into your instance on the command line and do and run (# represents the command prompt):
   # ls /dev
   You should see that /dev/sdf has been created for you
5. Format /dev/sdf by running:
   # mkfs.ext3 /dev/sdf
   It will warn you that this an entire device. You should type y to allow the process to continue unless you want to create specific partitions on this device
6. Create a directory to mount your new drive as on the filesystem, for example we’ll use /files:
   # mkdir /files
7. Add a reference in the fstab file to mount the newly formatted drive onto the /files directory by running the following command:
   #  echo "/dev/sdb /files ext3 noatime 0 0" >> /etc/fstab
8. Mount the drive by running:
   # mount /files
9. Check your drive has mounted correctly with the expected amount of file space by running:
   # df -h /files

It really is that simple, within a few cli commands you can simply add 1GB to 1TB of storage at the drop of a hat!

Questions? Leave me a comment and I’ll do my best to answer them for you

Related posts:

1. How to increase the size of an Amazon EC2 EBS Instance (with no downtime) I'm often asked by people who read my posts...
2. How to Mount a Linux LVM2 Partition in Windows I have a dual boot Windows Vista / Linux...
3. AWS Rolling Thunder In CentOS Linux I've been using a common technique to deploy source...

Today I contacted Amazon about a question that's been bothering me ever since they announced that SSL termination was possible on their Elastic load Balancers.

Basically, in network terms, as the instances sit behind the firewall and the firewall behind the load balancers, if you terminate the SSL at the load balancer, what exposure is your data at risk of between the load balancer and the firewall?

Amazon's response (though lacking in actual detail) is that data is secure between the load balancer and the firewall, though post termination it is transmitted in clear text (as you'd generally expect). You can see the original question and response here: https://forums.aws.amazon.com/message.jspa?messageID=209121

So all in all Amazon are saying that their load balancer-to-firewall connection is secure.

As they don't give any detail, to a point it depends on how much you trust Amazon on this issue but in my opinion if you're going to implement an SSL termination process on this scale and exposed to this level globally you're gonna do it properly, and if it does has security flaws it's going to be a marketing disaster for your service.

So, all in all, though I don't like not knowing the specific process they use, in future I'm going to trust it for most cases (though all the while keeping an eye out for related reported issues or concerns).

Related posts:

1. How to increase the size of an Amazon EC2 EBS Instance (with no downtime) I'm often asked by people who read my posts...
2. How to Reduce Amazon S3 Costs If you're new to Amazon S3 or have been...
3. Woo Wishlists are now Universal Thanks to recent improvements over at Amazon, you can...