Chris Dean

I've been using a common technique to deploy source code updates to my Amazon EC2 instances for some time now, which makes use of S3 as a central source code file-store, and syncs updates out to instances on a restart.

I didn't however know that this technique had a name: Rolling Thunder (Thanks AWS Tech Summit)

As it was such a cool buzz wordy term, I thought I'd post a guide to how I achieve Rolling Thunder on AWS using CentOS Linux based instances (though this should work with any *NIX variant) in case anyone new to the game wanted a how-to.

The method is as follows

1. Clone a running live instance of your web app to use a test server
2. Set up a private S3 bucket in which to store the source code for your web app(s)
3. Zip up your source code into an archive file (this makes the permissions management of the source code of multiple apps, or versions of multiple apps easier on S3 BUT you could simply use the bucket to store an export of the latest version of your code)
4. Upload your source to this bucket using the AWS web tool or (better) setup a sync using a tool such as cyberduck Remember to set permissions appropriately on all uploaded files i.e. don't make anything public, but don't remove your rights to access the uploaded content either!
5. Log into your test instance and install the latest version of the fuse filesystem using the following instructions (these are all cli commands):

   yum remove fuse fuse* fuse-devel
   
   yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap
   
   cd /usr/local/src
   
   wget "https://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz?r=&ts=1299709935&use_mirror=cdnetworks-us-1"
   
   tar -xzvf fuse-2.8.4.tar.gz
   
   rm fuse-2.8.4.tar.gz
   
   mv fuse-2.8.4 fuse
   
   cd fuse/
   
   ./configure --prefix=/usr
   
   make
   
   make install
   
   export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
   
   ldconfig
   
   modprobe fuse
   
   #confirm that 2.8.4 is the version displayed
   
   pkg-config --modversion fuse
   
   #set the fuse files system to load on startup (make the kernel module load)
   
   echo modprobe fuse >> /etc/rc.modules
   
   chmod +x /etc/rc.modules
   

6. Install s3fs, using the following instructions (these are all cli commands):

   wget http://s3fs.googlecode.com/files/s3fs-1.40.tar.gz (get URL for latest version)
   
   tar -xzvf s3fs-X.XX.tar.gz
   
   rm s3fs-X.XX.tar.gz
   
   mv s3fs-X.XX s3fs
   
   cd s3fs
   
   ./configure --prefix=/usr
   
   make
   
   make install
   
   

7. Create an /etc/passwd-s3fs file, and add a single line like this:

   accessKeyId:secretAccessKey
   

   Obviously replacing accessKeyId:secretAccessKey with access credentials for your AWS account

8. chmod this file to 640 for security (s3fs won't work unless permissions are set at this level)
9. Test that you can mount the s3 bucket as a filesystem folder by running the following commands (these are all cli commands):

   mkdir /thunder
   
   #test if /thunder is a mountpoint (it shouldn't be)
   mountpoint /thunder
   
   #mount the bucket
   /usr/bin/s3fs my-bucketname /thunder
   
   #test again (it should be a mountpoint now)
   mountpoint /thunder
   
   #try listing the source files you uploaded to the bucket
   ls /thunder
   
   #unmount the bucket
   umount /thunder
   
   #show that the files no longer exist in /thunder
   ls /thunder
   

   Note: as a test you might want to try writing a file inside /thunder too to do this use your favourite text editor or an program like touch, and then look for the file's existence in s3 throught he web interface, or cyberduck, or whatever

10. Next install Rsync (again, this is a cli command):

    yum install rsync
    

11. Then write a script to use rsync to update your web-app's code directory e.g. /var/www/html/ with the source code from the s3 bucket. Here is an example of the kind of script I use:

    #!/bin/sh
    
    ############
    #  Config  #
    ############
    
    archive_name=my-sourcecode-archive.zip
    
    #################
    #  Main script  #
    #################
    
    #output the start time
    echo 'Started: '
    date;
    
    #make a backup of your existing web app directory
    #this is just a precaution but can be useful for versioning deployments on the instances
    ts=`date '+%d-%m-%Y_%H-%M-%S'`
    cp -R /var/www/html /var/www/html_rt_bak_$ts
    
    #initialise the fuse filesystem
    modprobe fuse
    
    #check to see if /thunder is a mountpoint, if not mount it
    mountpoint /thunder/ || /usr/bin/s3fs source-a /thunder
    
    #copy and extract the zip file into /tmp/thunder, then remove the original archive
    mkdir -p /tmp/thunder
    cp /thunder/$archive_name /tmp/thunder/
    umount /thunder
    
    cd /tmp/thunder/
    unzip $archive_name
    rm /tmp/thunder/$archive_name
    
    #remove any stray .svn files/folders
    find . -name ".svn" -type d -exec rm -rf {} \;
    
    #rsync the contents of /thunder with /var/www/html
    rsync -avI --exclude-from=/tmp/thunder/thunder-exclude-list.txt /tmp/thunder/ /var/www/html/
    
    #remove the temp dir, and the exclude list
    rm -rf /tmp/thunder
    
    #chown and chmod everything to ensure synced files have the correct permissions
    chown -R apache:apache /var/www/html
    chmod -R 754 /var/www/html
    
    #output the end time
    echo 'Finsihed: '
    date;
    
    

    Note: The rsync exclude file. This is a file I put into my web app home directory that allows me to exclude various files from the rsync process, which can be useful to avoid copying files such as database connection settings or the source code archive itself into the web root. By making the exclude list a file in the web app, control of what's synced can be easily managed by the developer on upload of the source code to S3 rather than requiring extra edits to server side scripts on every instance.I tend to store my scripts in a /scripts directory and in my case I've called this one rolling-thunder.sh Remember to run a command such as this to make the script executable once written:

    chmod +x /scripts/rolling-thunder.sh
    

12. Next add this script to the startup sequence for your instance. There are a number of ways of doing this, but I choose to add it to the /etc/rc.local file (which is for executing user commands after the main system has started up). All you need to do is open /etc/rc.local in a text editor like nano and add a line like this to call your shell script:

    /scripts/rolling-thunder.sh > /var/log/rolling-thunder.log 2>&1
    

    Note: The additional line after the call to the script

     > /var/log/rolling-thunder.log 2>&1
    

    Creates and sends all output caused by the script (including errors) to a log file called rolling-thunder.log, which can be useful for debugging any errors that may occur

13. Reboot the test instance and take a look at your rolling-thunder.log file and check your web app code, your update should have been deployed smoothly!
14. Last, create an image of this instance to allow you to fire up further instances as necessary.

Notes:

• s3fs seems to start to choke when a single directory contains more than 1000 files, I think this is because of the time it takes to produce the list and parse it. Another good reason to package the code up into an archive
• Example thunder-exclude-list.txt ** are standard wildcard matches:

  *my-sourcecode-archive.zip*
  *thunder-exclude-list.txt*
  */some/specific-file-to/exclude.php*
  */some-folder-to-exclude/*
  

• A better deployment option for easy management/automated instances is to break the rolling-thunder.sh script in two and have the main part which does the sync sitting on s3 or in the deployment archive and a minimal script sitting on the server which calls that script. This route would give yet more flexibility/control on the deployment front from a central location as opposed to having to edit the deployment script on multiple servers