Symetrik Domino Consulting

Until Pantheon Mercury 1.1 or 1.2 comes out publicly, I wanted to setup a plain vanilla Drupal server for development on Amazon’s EC2 environment.

I wanted the server to have:

• Ubuntu 10.04
• Apache
• MySQL
• PHP
• Drupal
• Drush
• Webmin
• PhpMyAdmin

There are a ton of AMIs out there, so I wanted to find one that was already available and easy to spin up and configure with my server name and database name/users.

NOTE: The preferred method for me, is still going to be to use Pantheon. I was only looking for a short term, temporary image for a development server while we wait for Pantheon to go public/live.

The final result for me was that I spun up a Ubuntu 10.04 LTS server. The latest official Canonical Ubuntu AMIs are are all listed at http://alestic.com for each region. The list is automatically updated with the latest images.
From there, I installed LAMP all in one go with one command, then added Mail servers, PhpMyAdmin, Webmin, Drupal, and finally Drush.

It actually wouldn’t have taken me less time to just do that instead of going out and looking for other AMIs and trying out the ones I found.

Here’s the command to get you started:

$ sudo tasksel install lamp-server

I found 3 main public AMIs for Drupal. Here’s what I thought about each one of them.

I tried spinning up one of the free Bitnami AMIs on a micro instance. I decided that it was not a viable solution due to poor documentation and non-standard, non-Ubuntu way software is installed such as Apache and Drupal.

If you run on the Bitnami cloud, it’s easy to spin up an instance and backup your instance. There is a flat fee for using their system. For a single server, it’s not worth it ($49/month up to 3 servers, on top of Amazon charges) – http://bitnami.org/cloud/pricing

PROS:

• Up-to-date versions of Drupal 6.20 and 7 available.
• Can run on micro instance.
• AMIs are EBS boot.
• Have a free AMI that you can use in addition to the cloud hosted version with a backup that is similar to Turnkey Linux’s Hub setup.

CONS:

• I found the Bitnami stack unusuable because of the customized installation of Apache.
• It is installed in /opt/bitnami/apache2
• sudo service apache2 restart didn’t work – there is a bitnami specific command to restart it
• Drupal was installed in a non-tradtional way /opt/bitnami/apps/drupal
• By defualt apache was setup such that www.example.com was a bitnami splash page, whereas to get to Drupal, I had to go to http://www.example.com/drupal.
• Changing the default apache root was not-inuitive (I couldn’t figure it out).
• No username/password supplied for PhpMyAdmin.
• Poor documentation

I run another, non-Drupal turnkey linux appliance and it works pretty well, so I was familiar with how their AMIs and general setup works. Their software images are open source and you can download and install on your own hardware. They do not offer a free public Amazon AMI. If you use their Amazon AMI, you kick it off from their Hub. After logging into hub.turnkeylinux.org, you can spin up an instance, and configure automatic backups, similar to Bitnami’s cloud backup feature. They charge 10% on top of the Amazon charges, so it’s pretty affordable for a single server.

I decided against the Drupal AMI without installing it because of the following two cons.

CONS:

• No free AMI version to try like Bitnami has.
• The AMI is an instance boot.
• The AMIs do not support micro instance, so you have to run a small instance at a minimum.
• The AMI was built on a much older version of Drupal 6.16 (not really a big issue).
• The following statement is on their page: http://www.turnkeylinux.org/drupal6

Note: Drupal 6 includes a built-in update status module.
* When enabled could produce confusing/annoying version notifications.
* This is due to drupal6 being installed and updated via the APT package manager, instead of manually from the upstream source code.

I didn’t want to deal with having Ubuntu’s apt-get manage my Drupal updates, I would rather use Drush.

They didn’t have a free version to try, so I didn’t try their service. It looks similar in setup and price to Bitnami.
http://www.jumpbox.com/pricing

Here’s some instructions for doing a really quick Drupal install on Ubuntu server 10.04.

This assumes that you are well versed with Drupal and have already created your database and database user and given the rights to the DB user to access the DB.

Step 1 download, extract, and copy the files into the /var/www root

$ cd /
$ sudo wget http://ftp.drupal.org/files/projects/drupal-6.20.tar.gz
$ sudo tar xvzf drupal-6.20.tar.gz
$ sudo mv drupal-6.20/* drupal-6.20/.htaccess /var/www
$ sudo rm -rf drupal-6.20.tar.gz
$ sudo rm -rf drupal-6.20

Create the files directory and give it the proper ownership

$ sudo mkdir /var/www/sites/default/files
$ sudo chown www-data:www-data /var/www/sites/default/files

Copy default settings.php file to settings.php so that you can enter your own server and database details

$ sudo cp /var/www/sites/default/default.settings.php /var/www/sites/default/settings.php
$ sudo chown www-data:www-data /var/www/sites/default/settings.php

From here you would need to edit /var/www/sites/default/settings.php and enter in our server, database, and database user details.

I would also like to note that you can also install Drupal with aptitude on Ubuntu, but I would much prefer to install Drupal manually and use Drush to update drupal whenever I want to have it updated.

I only want Drupal updated when I say so and with what versions that I decide on.

$ sudo apt-get install drupal6

Installing Drush (at the time of this writing 4.2 is the latest version)

$ sudo apt-get install php5-cli
$ cd /usr/local/
$ sudo wget http://ftp.drupal.org/files/projects/drush-All-versions-4.2.tar.gz
$ sudo tar zxvf drush-All-versions-4.2.tar.gz
$ sudo rm -rf drush-All-versions-4.2.tar.gz (remove downloaded and extracted tar file)
$ cd drush (to make sure the files are there)
$ sudo chmod 555 drush

Create symbolik link
$ cd /usr/local/bin
$ sudo ln -s /usr/local/drush/drush drush

Last week, some of our developers complained that they could not access our server that we have hosted at Amazon Web Services (AWS) / Elastic Cloud Computing (EC2) environment intermittently every few minutes, but I was having no problems.

The next day, I was now unable to get to the server.
HTTP, HTTPS, SSH all failed.

Cloudwatch showed all metrics as “flat” for the few minutes after I stopped getting a response.

I would like to note that we have been using this server for approximately 6 months without any problems whatsoever. We have a very standard procedure for accessing the server, and we had tested many things before reporting the problem to Amazon. We were sure that we were not mis-typing the SSH host name, and were using the correct instance ID.

Additionally, I saw lots of other posts in the Amazon EC2 support discussion forum which were similar to this post about suddenly not being able to access their server that same morning and/or the night before.

The end result was that Amazon notified us that quote: “There does appear to be a problem with the underlying hardware.” and “I would relaunch the instance using your latest AMI. Since this instance is not EBS, what ever changes you made may be lost.”

Here are some hard lessons that I learned from the experience.

A) As redundant and high availability oriented we thought Amazon was, IT IS NOT.

B) Amazon’s support in such incidents is actually just a discussion forum for the various services that it offers. In this case, it is the EC2 discussion forum, where you are lumped in with thousands of other people running potentially hundreds of other AMIs that contain unlimited number of “popular software” on several popular operating systems. In other words, there is alot of traffic on there because of people running Windows, Oracle, WordPress, Linux (all the flavors you can think of), etc. etc. etc. The traffic is so high that your chances of getting responded to depend on the Amazon staff manning the forum, and whether your needle doesn’t get lost in the high traffic hay stack.

C) Amazon does offer phone support, but this is only available from the Gold and Platinum support levels. Gold costs $400/month, so unless you are enterprise, you can forget it. Bronze and Silver cost $49 and $100/month respectively.

Full details: http://aws.amazon.com/premiumsupport/

I was really surprised that there was nothing for the average joe running a couple of servers. It costs me more a month to run at Amazon than it does my dedicated server hosting company, yet at least I can get someone on the phone at the dedicated server hosting company.

D) Amazon’s attitude in their response and the response I’ve seen many times since having this issue is that you should never rely on an instance whatsoever. Count on it going down at any time. Instead, have load balancers and custom built, EBS boot AMIs ready to go at any time.

This is a difficult philosophy to totally grasp, but one that must be grasp if you are moving to the cloud.

E) We actually had an EBS volume attached where our /var/www and MySQL resides. We had previously tested disaster recovery in the event the instance went away and had successfully detached, spun up a new instance, and re-attached the volume and ran a script to let the server know where the /var/www and MySQL data was.

This is a good practice for the data, but it should also be the case for the server.

F) Instance store is no way to go, unless you are literally spinning up a temporary test server to run for a few minutes or a few hours, or you are spinning up an instance for the CPU cycles.

G) EBS Boot instance should really be the rule of thumb, and generate a master AMI from it. If you make changes (periodic operating system updates, etc.) – also periodically re-bundle your AMI.

H) Make sure your data is also stored on a separate EBS volume, and make automatic nightly snapshots of that EBS volume. Think of EBS snapshotting in the same way you think of incremental backups in the tradition IT infrastructure sense.

By having your data on a separate EBS volume, you don’t necessarily have to re-bundle your AMI every night.

Someone out in the websphere correct me if I’m wrong, but it seems to me that snapshotting a data EBS volume every night is easier than re-bundling your instance every night in the case that you store your data on the same EBS volume as your instance operating system and software.

Thoughts and feedback appreciated.

I’m writing this blog post in the hopes that some wise soul out there happens to read this post and responds with the answer. I am truly stumped.

I’m running an instance store instance at Amazon. I would like to bundle a custom AMI in case something happens to the instance as it did to our instance a few days ago.

Our instance is built from a high availability, high performance AMI built by Pantheon. After we spin up the instance, we still have a bit of customizations. Namely setting up the mail/mailboxes, setting the hostname, installing Webmin, etc. etc.

We wanted to bundle our customizations into a new AMI in case something happened to the instance again and we could simply spin up a new instance from the AMI, change the public DNS and MX record, and re-attach the EBS volume where our /var/www and MySQL data is located.

After searching various and sundry locations on the web on how to re-bundle an AMI from a running instance, I came up with the following commands.

I’m using this command where 123456789123 is not my actual Amazon user ID, and I’ve copied my Amazon private key and certificate to the /mnt/ids directory.

$ sudo ec2-bundle-vol -d /mnt/amis -k /mnt/ids/pk-*.pem -c /mnt/ids/cert-*.pem -u 123456789123 -r i386

I get the following:

Copying / into the image file /mnt/amis/image...
Excluding:
/sys/kernel/debug
/sys/kernel/security
/sys
/var/log/mysql
/var/lib/mysql
/mnt/mysql
/proc
/dev/pts
/dev
/dev
/media
/mnt
/proc
/sys
/etc/udev/rules.d/70-persistent-net.rules
/etc/udev/rules.d/z25_persistent-net.rules
/mnt/amis/image
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00273375 s, 384 MB/s
mkfs.ext3: option requires an argument -- 'L'
Usage: mkfs.ext3 [-c|-l filename] [-b block-size] [-f fragment-size]
[-i bytes-per-inode] [-I inode-size] [-J journal-options]
[-G meta group size] [-N number-of-inodes]
[-m reserved-blocks-percentage] [-o creator-os]
[-g blocks-per-group] [-L volume-label] [-M last-mounted-directory]
[-O feature[,...]] [-r fs-revision] [-E extended-option[,...]]
[-T fs-type] [-U UUID] [-jnqvFKSV] device [blocks-count]
ERROR: execution failed: "mkfs.ext3 -F /mnt/amis/image -U 2c567c84-20a1-44b9-a353-dbdcc7ae863b -L "

Here are the results of df -h, so I know I’ve got plenty of room on /mnt:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  2.1G  7.4G  22% /
devtmpfs              834M  124K  834M   1% /dev
none                  851M     0  851M   0% /dev/shm
none                  851M   96K  851M   1% /var/run
none                  851M     0  851M   0% /var/lock
none                  851M     0  851M   0% /lib/init/rw
/dev/sda2             147G  352M  139G   1% /mnt
/dev/sdf             1014M  237M  778M  24% /vol

Does anybody have any clue what the mkfs.ext3 errors are all about? I’ve posted this over at the Amazon EC2 discussion forum and the Pantheon Drupal group with no response and I have scoured google and I can’t find anything like this.

I have spent quite a bit of time figuring out how to make this blog easier to read visually. One thing I came up with is CSS to make the commands or code that I describe easier to pick out and easier to see due to formatting.

Here’s the CSS I used for the

 tag.  This will allow nicely formatted pre sections with your code or commands.  I opted to wrap the words for really long lines instead of using the horiztonal scroller.

I’ve tested this on Mac OS X with Opera, Safari, Chrome, and Firefox and it works great.
I like the added effect of the background and rounded corners as well.
pre {
color: white;
overflow-Y: hidden;
overflow: auto;
font-family: “Consolas”,monospace;
font-size: 9pt;
text-align:left;
background-color: black;
overflow-x: auto; /* Use horizontal scroller if needed; for Firefox 2, not needed in Firefox 3 */
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
_white-space: pre; /* IE only hack to re-specify in addition to word-wrap */
margin: 0px 0px 0px 0px;
padding:6px 5px;
-moz-border-radius: 5px;
-webkit-border-radius: 5px;
-khtml-border-radius: 5px;
border-radius: 5px;
}