How To: Getting Started with Amazon EC2
Amazon EC2 is among the more potent items in Amazon’s web services arsenal. You’ve probably heard of many of the other services such as S3 for storage and FPS for payments. EC2 is all about the “elastic compute cloud.” In layman’s terms, it’s a server. In slightly less layman’s terms, EC2 lets you easily run and manage many instances (like servers) and given the proper software and configurations, have a scalable platform for your web application, outsource resource-intensive tasks to EC2 or for whatever you would use a server farm.
There are three different sizes of EC2 instances you can summon and they’re all probably more powerful than the server currently running your blog. Unless you’re offloading video processing or something intense to EC2, the default small instance with its 1.7GB of RAM and 160GB disk should be more than fine. It’s just nice to know that if for any reason I need a farm of machines each with 15GB of RAM, I can get that easily.
EC2 has been around for a while but has gained interest in the last few weeks as Amazon released an elastic IP feature. One of the larger EC2 issues deals with data persistence on instances. There are many limitations with EC2 that make it difficult to use unless you carefully build around the EC2 architecture and don’t just assume that you can move your app to EC2 flawlessly. If an instance crashes and you run it again, you’ll loose data and when the instance comes back up it will have a new IP, adding another hurdle with DNS issues. Fortunately, the elastic IP feature lets you assign a static IP address to your instances.
As the title of this article implies, this article is meant to be a beginner’s look into tinkering with EC2. Just because you will be able to host a page on EC2 at the end of this article does not mean you should start using it as your only server. Many considerations need to be made when using EC2 to get around the data persistence issue. If your startup is looking to use EC2 as a scalable platform, fortunately there are many services that have already built stable systems on top of EC2, ready for your consumption: WeoCeo, Scalr and RightScale. Enough talk, shall we jump right in?
Note: Most of the information below (and more) is available in the EC2 API doc if you enjoy reading those things.
Getting Started
In order to interact with any EC2 instances, you’ll need to install Amazon’s command line tools and download your X.509 certificate from Amazon. Let’s start with the certificate. Login to your Amazon account and visit the AWS Access Identifiers page. In the X.509 certificate section near the bottom, click Create New. You’ll be greeted with a page allowing you to download both the private key file and X.509 certificate. Both of these are very important, download them to your desktop so you don’t lose them.
Next up, grab the EC2 command line tools. Extract them and you should be left with a folder named something like ec2-api-tools-1.3-19403. We’ll move those to a directory where we will also store the private and public keys.
The commands below assume you are working on an OS X machine in the Terminal.
cd ~/Desktop
mv *.pem ~/.ec2
cd ~/Desktop/ec2-api-tools-1.3-19403/ #depends on ec2 tools folder name
mv * ~/.ec2
This is what your .ec2 folder should have now.
Next up, we’ll set some paths in your bash profile so the OS knows where the EC2 tools are located.
Add these lines, replacing “YOURKEYNAME” with the actual file name of your private and public keys, then save.
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=pk-YOURKEYNAME.pem
export EC2_CERT=cert-YOURKEYNAME.pem
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
To get the changes noticed by the OS immediately, run source.
Now we can actually use those helpful EC2 command line tools.
AMIs, Keypairs and Instances, Oh My!
Before proceeding, you’ll need to grok the concept of AMIs. They are Amazon Machine Images and whenever you create an EC2 instance, an AMI is quickly loaded on the machine. They’re essentially images of the OS. If you terminate an instance and bring it up again, your machine will only have the data initially included in the image. That’s why lots of work goes into making (”bundling”) a good image you will always use that has the configurations and software you need so you don’t have to do much whenever you load the image. This article won’t delve into creating your own AMIs but fortunately there are many great, public AMIs available for use.
To SSH into the instance we’ll create from an AMI we find, we’ll need to create a keypair. This is a different key from the one provided to us by Amazon. That was for using the EC2 tools and interacting with the instances in terms of creation and management. To actually SSH into an instance, a separate keypair is required as there are no passwords by default.
ec2-add-keypair pstam-keypair
That will print out the private key, which you’ll need to copy and paste into a file manually.
#now paste the private key and save
sudo chmod 600 id_rsa-pstam-keypair
Now we can find which AMI we want to toss on our yet-to-be-created EC2 instance.
ec2-describe-images -a
Using the -a option will list all of the AMIs you have access to, and there are a lot. Alternatively you can list just the images Amazon has:
I found an AMI that I’ll try out.
ami.yyang.info/gentoo-nginx-php-mysql-06feb2008.manifest.xml
It’s a Gentoo Linux install with PHP, MySQL and nginx. When looking at AMIs, you need to find the AMI ID. In this case, it’s ami-6138dd08.
The instance is now being loaded with the AMI I selected and booting up. It should output some text with “RESERVATION” and “INSTANCE” rows. On the instance row, it will say something like pending pstam-keypair until it has fully booted up. When an instance is ready to go, its URL and internal address will be supplied with the ec2din command below.
If you try to access the URL in a browser, nothing will happen just yet as the firewall blocks all ports by default. You’ll have to open up the ones you need. We’ll do port 80 for HTTP and 22 for SSH. If the AMI you are running doesn’t have a web server installed, accessing the EC2 URL in a browser won’t bring up anything regardless.
ec2-authorize default -p 80
If you want to undo any port authorizations you’ve made, you can use ec2-revoke. Now if you access your EC2 URL in a browser you’ll get something like a default Apache page, or in the case of the AMI I’m using, a phpinfo() page.
The next step to actually using your new EC2 instance is SSHing into it to get full root access. Run the line below and replace the EC2 URL with the one provided to you by the ec2-describe-images command earlier.
If you run into the problem I did, this won’t work and you’ll be asked for an EC2 instance password that doesn’t exist. I found out this was because I initially created the id_rsa-pstam-keypair file as the root user but ran the ssh command as a regular user which was not able to access that keypair. That was easily fixed with sudo chown Paul id_rsa-pstam-keypair but you won’t have this issue if you followed this guide.
Otherwise, you should be logged into your EC2 instance as root over SSH. Now that we’re in, we can tinker with the system however we like and even see what kind of hardware we’re running on, setup FTP and drop a web app into /var/www/localhost/htdocs or whatever. Having full root access in any OS you wish is one of the boons of using Amazon EC2.
Static IP Time
If you plan on running your instance for good, you’ll want a static IP. Let’s get one for you.
Now we just need to tie that IP address to the instance ID of the instance you wish to give a static IP. You can grab the instance ID (not to be confused with the ami-* AMI ID) by running ec2-describe-instances.
In the line below, replace XXX.XXX.XXX.XXX with the IP address you were given above and replace i-yourinstance with your actual instance ID.
Give it a few minutes and your instance will be accessible through that new IP in addition to the longer EC2 URL we were previously using. Please note that if you terminate the instance, the IP does not remain tied to the instance, to the best of my knowledge. Terminating an instance seems to be a nuclear option compared to simply rebooting an instance via regular unix commands over SSH.
Now that the instance has an IP you can setup a domain name with it if you want. The easiest way I’ve found of doing this is through a DNS service like EveryDNS. Just provide your domain name registrar with EveryDNS’s domain name servers, create an EveryDNS account, add your domain and create an A record with your newly associated EC2 instance IP address.
The TTL on EveryDNS seems to be fixed at 3600 so it might take a while for propagation, especially if you’re used to pushing down TTL to 300 when doing DNS work.
Terminating Your EC2 Instance
Killing your instance for good can be done, like every other action, through an EC2 command line tool. This time, it’s the appropriately named ec2-terminate-instances (ec2kill). Just provide it with the instance ID of your instance (get it from running ec2din).
It should return with a “shutting-down” status but you’ll definitely want to check back in a few minutes with ec2-describe-instances to make sure it shutdown successfully and comes up as “terminated”. There are a few cases where instances will hang on shutdown and you will continue to be charged for instance hours.
The Next Step
Now that you’ve successfully launched your first Amazon EC2 instance, you’re ready to begin exploring the endless EC2 possibilities. I’m still learning about taking the next step but overall, EC2 really isn’t something to mess with unless you have quite a bit of sysadmin and development experience. Actually using EC2 as an elastic compute cloud usually involves setting up an instance as a load balancer and giving that instance access to an array of active EC2 instances which it can hand work to. Other hurdles include providing multiple instances access to the same database, using Amazon S3 as a persistent filesystem and employing highly redundant backup systems given the relatively volatile nature of instances.
Thoughts? Do you have anything you’d like to use with Amazon EC2?
This post brought to you by a rainy day, a MacBook Air and a Corona.
I think my brain just exploded.
Sick man. No need for this now, but I know where to come when the need arises!
In English please, LOL but seriously that is a crazy description but very nicely done, I really wish I knew what it was all about ;)
Whoosh! Right over my head, but it created an appealing breeze.
Great how-to, Paul!
By the way,
“If an instance crashes and you run it again, you’ll loose data and when the instance” <— loose = lose
Bravo Paul, nice article! Though, I’ll go over this again, in smaller chunks…
I’ve recently tried EC2 too, but just played around with it for a day - it would be too expensive for a small website.
A small instance would cost 0.10*24*31 = ~$75 just for running (backups, traffic, etc. not included).
Amazon provides a nice calculator:
http://calculator.s3.amazonaws.com/calc5.html
So one thing I’m a little unclear on is the instance hours…is that actual CPU usage, or just time your instance is running?
P.S. THANK YOU for writing something on this, I haven’t been able to find a nice article on actual EC2 experiences.
P.P.S. Nice background image
@Tim Trueman:
Instance hours is the time the instance is running, not the actual CPU time usage.
It’s sunday morning here Paul….. Sunday is the day when people try to rest their brain and relax a bit, why couldn’t you of done a simple Air update or something equally easy….
I’d rather ask my techie-friends to do this for me.
Great article.
btw… We have started an AWSome group (AWS Meetup) in Atlanta. You should check it out. Also if you are interestde you could give a pitch (like this blog entry) at our first meeting. Here are some lings…
AWSome Atlanta (Cloud Computing User’s Group)
Two useful tools to know about to help make managing EC2 instances much easier, one is a Firefox extension and the other a paid service (but with a free account for developers).
First the Firefox extension..
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609
..and then the RightScale service …
http://www.rightscale.com
Tom
well written and illustrated. looking forward to a time when all my personal computing requirements are met online. (then i will be kewl enough to have a MacBook Air). best…skip
Paul,
If a user donates ($20) or just emails me their username (heh) they can adjust their TTLs. :-)
-David
Good to know David. EveryDNS is definitely worth donating to.. friends that use it have never had any DNS downtime.
This is what I was looking for. Very nice introduction. Thanks alot. Greetings from Germany
After signing up and receiving your EC2 credentials goto Cohesive Flexible Technologies Elastic Server On-Demand.
You can select a custom list of software components, specify your target deployment (in this case Amazon’s EC2 AMI), build, and auto-deploy to the cloud for free.
All you need is the FireFox EC2 UI mentioned by Tom above and Elastic Server On-Demand and you can do all the steps above without typing a single command.
Could this be used to send video files to for encoding?
@David - absolutely. It’s advertised as both a scalable platform as well as one for offloading batch processing that would require resources you don’t have easy access to.
Nice writeup. I got tired of messing around with EC2 and found Slicehost. Pretty similar service, decent price and much easier to get going.
Nice one Paul - this is a great guide!
Btw - you post this and EC2 was offline for an hour today… just how many readers do you have exactly? ;)
EC2 has been “stammied”.
Excellent, excellent post!
I’m using EC2 myself and I have to say your description of the service is just perfect and comprehensive.
Good stuff :)
i take it the video encoding would need to be done via command line though?
I mean is there no way of having a GUI OS Install on this thing and you can install a app to Q a load of files up to encode in batch?
found this on a google advert:
http://download2.3tera.net/demo/applogic20demo.html
Wow! That is a great detailed set of instructions. I’m definitely bookmarking this on delicious. How is everybody’s success rate with this?
Just remember, EC2 might not be for everyone. [begin shameless plug] We just launched a new product into public beta called GoGrid which those of you who want true root access (on linux) or admin access (on windows) might be interested in. You can deploy servers in under 5 minutes and even load balance them for free.[end shameless plug]
EC2 is a pretty amazing bit of technology, especially with the release of elastic IPs and availability zones. But be sure you look “beyond the clouds”.
-Michael
Wonderful post! Thank you for this, someday none of us will need our own servers anymore, things will just scale easily with us.
Thanks for writing this. The word about this article is spreading around Twitter :)
I’m curious as to why you picked Gentoo for EC2? I’ve been a fan of Ubuntu for a while and it seems the following is growing. Here’s an Ubuntu 7.10 Gutsy base install AMI for EC2 for those who are interested
http://ec2gutsy.notlong.com
Full disclosure: I maintain a series of Ubuntu AMIs as a community service and you can join an Ubuntu on EC2 community here:
http://groups.google.com/group/ec2ubuntu
“download them to your desktop so you don’t loose them.” keep ‘em tight, all right!
ahem. Loose != lose. no matter how many people spell it wrong.
agree with the over the head + breeze comments above.
Man I’d love to get this running w/ Final Cut’s Qmaster. Rendering video on a laptop is the suck.
Nice Post Paul,
@ Adrian: ha ha ha!
I wonder if this could be used to host a game server?
Thanks for the great post!
Amazing post…
One thing that I would note is that when you terminate an instance that you have allocated an IP address to… you will be billed 1 cent for every hour that the IP address is not associated with an instance.
To release an address from your account if you don’t plan on using it again or you just don’t want to be billed, type in this command:
ec2-release-address xx.xxx.xxx.xxx
more info here:
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1346
@Brandon - thanks for the heads up about releasing the IP address, good to know
Nice work, great post.
You have a typo in your description of terminating instances:
It should return with a “shutting-down” status but you’ll definitely want to check back in a few minutes with >>>ec2-describe-images<<< to make sure it shutdown successfully and comes up as “terminated”
Should read ‘ec2describe-instances’
I’ve just created a EC2 guide for Windows user here http://arope99.blogspot.com/2008/05/getting-started-with-amazon-elastic.html
Feel free to try it out….
this is also a good guide…
you should check it out
http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/
Hi
Thanks for a useful post ! i need to know one thing
using the below command we allow the port 22 to our image i.e modify the firewall..
PROMPT>ec2-authorize default -p 22
My question is:
how can we check the list of ports opened so far at our image or instance ?
Here is another flash based tutorial on Amazon EC2 - http://www.searchblox.com/tutorial/sb_amazon_demo/sb_amazon_demo.htm
very nice article and “Terminating an instance seems to be a nuclear option compared to simply rebooting an instance via regular unix commands over SSH.” is gold :-)
@brandon thanks for the heads up!
thanks! was a great kick start intro.
Thanks Paul
I was bogged down with the O Reilly Programming AWS book and your writeup is very lucid. Enabled me to get up and running quickly.
@Brandon
BIG thanks for the tip about the static IP needing to be released.
Great tutorial man, thanks for it.
Nice detailed article on using EC2. It will surely help me setting up EC2 service I’m purchasing next week for my client.
Thanks for the awesome write up. Yea.. that sudo part killed me for the longest time.
Paul,
I wanted to let you know that I referenced your “How to get started with Amazon EC2″ post today on my Technical Manager blog. I wrote about Cloud Computing Examples and referenced you and a couple of your posts. I recently found your blog and enjoy reading about live in the Cloud.
Thanks - Kevin Mullins
Hello,
I’m just wondering how it is with a outgoing ports from AMI instance? Are they all blocked be default. What If I want to enable all ports? I have found another howto and they also talk only about port 22:
http://www.linuxconfig.org/Howto_CREATE_BUNDLE_UPLOAD_and_ACCESS_custom_Debian_AMI_using_ubuntu
thanks
@Kevin Mullins - thanks for the coverage!
Hi,
A few weeks back I setup ec2 server and now I set up a new one. But how can I get the instance I am running for old one.?>
whenever I issue the command ec2-describe-instances
I get the details for the new server I setup.
How can I get the details for the old one.
Thanks,
Anp
Hi,
First of all thank you for this wonderfull tutorial, it really was a very good help, when i was lost searching for some.
I have a small problem.
i couldn’t find the “id_rsa-pstam-keypair” file and so i am unable to connect to the server, it gives me a “Permission denied (publickey,gssapi-with-mic).” erorr when trying to connect.
when is this file generated? and where is it supposed to be saved? and how can i regenerate it if possible?!
thanks in advance, and really good work.
thanks, you helped me a lot. Does anybody know of a good ec2 gui client for mac os x?
Great work..Appreciate your time. Helped me to resolve Issues I was facing.
Thanks a lot