Infrastructure Internet Markets - Then, now and next.

The garden of markets

The internet is like a garden. Things grow, as long as you water them. In fact, the internet is really like a continuously expanding cluster of gardens, each with a different environment (aka market). Sales, marketing, finance, communication, and everything in between.

What I'll be talking about here are the infrastructure markets that grew to meet the needs of the internet.

Some new markets (gardens) of the past ...  

Connectivity, Dialup and beyond

First, there was the need for people and companies to be connected to the Internet. An untold amount of ISPs (internet service providers) were created to meet this need. They ranged from the size of your local mom and pop shop all the way to the size of AOL. Hundreds, thousands, hundreds of thousands of these companies were created world wide in order to meet the demand of humans wanting to communicate with one another. The ISP market will always exist and it will continue to grow and evolve. The speed of our connections will increase and along with it, what we do on the internet will change as well.

Web Hosting. Anyone remember geocities?

How many web hosting companies have existed in the last 15 years? A lot. If companies wanted to get on the internet, the most common way to do that at the time, was to pay a web hosting company to manage the server and internet connectivity for you. In this method, all you had to do was worry about creating and maintaining the content (which was a whole other market).

Today's new garden is... IaaS and PaaS

Infrastructure as a Service (Iaas)

Remember the ISP and web hosting markets exploding? Well, IaaS and PaaS is next! It is now!


When Amazon web services came out, it caused somewhat of a distribution in the industry. Why was this such a big event? Well, it was the first time internet infrastructure could be programatically controlled (can anyone say api). So simple, yet so game changing. Infrastructure as a service (IaaS) is really just web operations (data center, servers, storage, network) with an api in front of it. The benefits are huge to both internet based companies and tradition enterprise IT organizations.

We are now in a gold rush. Everyone under the sun wants to offer an IaaS solution. No one wants to be left behind. ISPs, Data center providers, enterprises, web hosting companies, governments, education, and even individuals are all getting on the boat. All of these people are interesting in providing IaaS features to their clients. Both publicly and privately.

There are now CMS (cloud management solutions) that allows these organizations to create an IaaS solution without the investment in writting their own code. Open Stack, Cloud stack, and eucalyptus seem to be the leading options available today. An organization can have an inhouse proof of concept up and running in a matter of hours. A production ready system can be ready in a few months.

Platform as a Service (Paas)

This is up a layer from IaaS. Instead of offering raw infrastructure, it offers a more of an application platform to its users. Someone only needs to upload their application, ie source code, and the PaaS will run and scale their solution as required.

Salesforce, Engine Yard, AppEngine, and Azure are a few of the leading public PaaS providers. Now, there are also PaaS software solutions that will allow people to create their own PaaS. Openshift and cloudfoundry are a couple in this category.

If I where you

I would get into this scene today. In 5 years from now, no one will be provisioning infrastructure manually anymore.

 

 

 What Next?

Obviously, more growth in IaaS and PaaS is coming. Software and solutions to help secure, monitor, provision, deploy all this stuff is currently being developed. But, what is the next garden? What is the next culture shift?

If I knew the answer to this, I probably wouldn't be talking publicly about it. The point is, some new internet garden will arise in which we will all see ourselves shifting onto.

Do you have a guess? Do you know?

Thanks
Chris

Cloudstack Integration - Detecting instance creation and deletion

So, you want to integrate your cloud?

Let us assume you have a cloud built on cloudstack. Now that you have this cloud, you may need integrate some of your other systems with cloudstack. Why? Well, for all those compute instances that you will run in the cloud, perhaps you will need to tell your other systems about them?

Perhaps you have a monitoring system that you need to configure so that it knows about your new instances?

Perhaps you need to register a dns entry?

Perhaps you need to tell puppet or chef about your new instance?

Perhaps you need to send an email to your boss asking for a raise every time a cloud instance is created?

I had some of these issues and I found out that cloudstack didn't have a great way, out of the box, to run 'custom jobs' when an instance was created or destroyed. However, cloudstack does do a great job giving us the information we need to make this happen. So, with a little help from the folks @ cloudstack, here is what I came up with.

Create and destroy?

Let us assume we want to configure our monitoring system and tell it to start monitoring instances the moment they get created. And, since we care about cleaning up our monitoring system, we also care about removing the instance from the system when the instance is destroyed.

So, how can we find out from cloudstack when these things occur?

At first, I thought about putting something in the OS template itself so that the instance itself would do these things. However, I realized that this might not be the best thing for me a few reasons. Those reasons were:

• The instance will have no way of knowing if it is being destroyed. (versus just stopped). This would prevent the instance from calling the 'remove me' jobs. 
• I didn't want to manage and troubleshoot things on a lot of instances (versus a central location). 
• It might not be secure (assuming a lot of people would have access to the instances). 

So, I decided to query cloudstack itself for this information. Here are some of the options that I came up with:

• Query the DB directly
• Parse the log file for output
• Access the cloud stack events via the api
• Access the cloud stack instance information via the api

I ended up choosing to use the event information via the api. It had all the inforamtion I needed and using the api is always the best coarse if it is available to you.

So, I wrote a script that polls for event info every X seconds. The script determines if there has been an instance that was created or destroyed successfully. And, if so, the script runs a bunch of integration tasks (like configure my monitoring system).

I wrote the script in python using the python cloudstack library found here:

https://github.com/jasonhancock/cloudstack-python-client

How to get the events?

In order to get the list of events, I decided to use the api method 'listAsyncJobs to get the events. If we want to get the events for all users (not just the account's events), we need to pass the isrecursive and listall parameters correctly).



Code snipnet:


#!/usr/bin/python
import CloudStack

api = 'http://10.1.2.3:8080/client/api'
apikey = 'xxxxxxxxxxxxxxxx'
secret = 'xxxxxxxxxxxxxx'

cloudstack = CloudStack.Client(api, apikey, secret)

jobs = cloudstack.listAsyncJobs({'isrecursive': 'true', 'listall': 'true'})
for job in jobs:

How to detect new instances?

Basically, you need to check for these things:

- event cmd =  ' com.cloud.api.commands.DeployVMCmd'
- event jobstatus = 1  (this means the event was a success)

If these two items are true, we know we have a new instance in our cloud.


Code snipnet:


       # check for new vm jobs
       if ((job['cmd'])=="com.cloud.api.commands.DeployVMCmd"):
         if ((job['jobstatus'])>1):
           #print "SOMETHING WENT WRONG WITH NEW INSTANCE"
         if ((job['jobstatus'])<1):
           #print "VM being created now............"
        if ((job['jobstatus'])==1):
           #print "NEW VM was created successfully........."
           # Here is where we would add the part to configure our monitoring system

How to detect destroyed instances?

Basically, you need to check for these things:

- event cmd =  'com.cloud.api.commands.DestroyVMCmd'

- event jobstatus = 1  (this means the event was a success)


If these two items are true, we know that a cloud instance was destroyed in our cloud.

How to get information about the instance in question?

Now that we know an instance was created or destroyed, our script will need to know some information about the instance in question. No problem!

The data returned in the event itself will have this information!

Below, you will see how I get the instance ID, hostname, and IP address. (other information is available if you need it)

Now that your have that information, you call go create that dns entry or setup monitoring or whatever you need to!


Code snipet:

           vmname=job['jobresult']['virtualmachine']['name']
           vmzoneid=job['jobresult']['virtualmachine']['zoneid']
           for nic in job['jobresult']['virtualmachine']['nic']:
               vmip=nic['ipaddress']
           zones = cloudstack.listZones({'id': vmzoneid})
           for zone in zones:
                  vmdomain=zone['domain']
           vmhostname=vmname+"."+vmdomain

            print vmid, vmhostname, vmip

To sum it up

Now, if you want, you can take your cloudstack cloud and integrate it so that your instances in your cloud are fully and automatically integrated to all your other systems too.

If someone is interesting in the full code I used, let me know and perhaps I can post it up somewhere.

Thanks and happy hunting
Chris

Cloudstack instance authentication using ssh keys

SSH Keys... 

Background: I am currently building a private IaaS cloud using the open source cloudstack software.

Did you know that cloudstack comes with the ability to offer cloud clients the ability to authenticate into their instance's using ssh keys? I didn't.

So, I'm reading the cloudstack api documentation and ... Before you say anything, yes, I was actually reading an api doc. So, I see an api method entitled createSSHKeyPair and I wondered if it did what I think it did. It did.

Unfortunately, I couldn't find any documentation on this, so I thought I'd help the community out and share what I learned.

I should also mention that Cloudstack also supports another form of user authentication. That is random root/admin password generation. A user is given a random password for use on their instance. I've tested this an it seems to work fine. This works on linux, unix, and windows. This is not covered in this article however.

A quick primer on SSH keys and why this would be useful:

If you know all about ssh keypairs, then please feel free to skip to the next section.

If you don't know, an ssh key pair can be used to login into a system with a 'ssh key file' versus using a known password. Please feel free to reference other material to learn about ssh keys.

Using ssh keys and infrastructure as a service (iaas) clouds is useful for a couple reasons. First, every cloud user would be using their own ssh key. This, in itself, offers some additional security to your cloud solution. This ensures that one cloud user can not access another cloud user's instances (unless they shared their ssh key files). Secondly, when people choose to use clouds, they often are doing it because they know the value of programmatically being able to access and modifuy thier infrastructure (instances). And, one way to do that is to use ssh keys. This would allow to easily manage many instances with a simple, secure method.

By the way, Amazon's EC2 uses SSH KeyPairs pretty heavily.

Now, what work is needed to make this happen on a cloudstack cloud?

First, some restrictions, notes and disclaimers. This should work on most unix/linux variants. I don't think this works with Windows atm (since I don't think is has a built in SSH server =]). I've personally tested it on CentOs linux. Also, SSH Key management is not available via the cloudstack GUI (currently). That means that the only way to create instances that use ssh keys is via the API (for now). My work was done on cloudstack 3.0.0. However, I think it also works on older versions as well.

The only thing a cloudstack administrator needs to do is:

1. Create a instance template that supports SSH Keys

The process for the cloud user would be:

1. Create an SSH Key in cloudstack
2. Create an instance using the SSHKey enabled temaplate and specifying their personal ssh key
3. Login to the instance using their ssh key (not their password)

 Create a instance template that supports SSH Keys

The folks @ Cloudstack published a script that enables SSH Key support on a linux instance. What does this script do? Well, the script will run on instance startup. It will communicate to the instance's virtual router and ask cloudstack for the SSH key private key. The script will then download the ssh key and install it on the instance for the cloud user to authenticate against. What we, as the cloud admin, will need to do is install this script on an instance and save the instance as a template. This way, cloud users will be able to use this feature.

 Here is what I did on a CentOs instance in order to get this to work:

• Create a new instance using the CentOs template provided by cloudstack.
• Download the cloudstack script from here to the centos instance: http://sourceforge.net/projects/cloudstack/files/SSH%20Key%20Gen%20Script/
• Copy the file to /etc/init.d and have it run on startup.
• Stop the instance (via the cloudstack GUI)
• Create a template (via the cloudstack GUI)

Here are the commands I ran on the CentOs host before I created a template.

# download the script
wget http://downloads.sourceforge.net/project/cloudstack/SSH%20Key%20Gen%20Script/cloud-set-guest-sshkey.in?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FSSH%2520Key%2520Gen%2520Script%2F&ts=1331225219&use_mirror=iweb

 # copy the script to the proper place
cp cloud-set-guest-sshkey.in /etc/init.d/

# set permissions on the script
chmod +x /etc/init.d/cloud-set-guest-sshkey.in

# tell the OS to start the script on start up
chkconfig --add cloud-set-guest-sshkey.in

The Cloud User work flow

1. Create an ssh key:

We will need to make a call to the createSSHKeyPair api method. I personally use the cloudstack python api library that jason hancock published. But, for a better example of how to do this via the api, I'll show example curl commands to the cloudstack api.

Note: I am making the below call from my cloudstack server itself. If you are making the api call from a different server, your URL/PORT will be different and you will need to use api keys.

Here, I am creating a ssh keypair called "keypair-test1" for the admin account in the root domain. (Please adjust these values to meet yoru needs)

My curl command:

curl --globoff "http://localhost:8096/?command=createSSHKeyPair&name=keypair-test1&account=admin&domainid=5163440e-c44b-42b5-9109-ad75cae8e8a2"

The output will look something like this:

<?xml version="1.0" encoding="ISO-8859-1"?><createsshkeypairresponse cloud-stack-version="3.0.0.20120228045507"><keypair><name>keypair-test1</name><fingerprint>f6:77:39:d5:5e:77:02:22:6a:d8:7f:ce:ab:cd:b3:56</fingerprint><privatekey>-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
-----END RSA PRIVATE KEY-----
</privatekey></keypair></createsshkeypairresponse>

What you will want to do, is copy the key data into a file so that file looks like this:

 -----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci
dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB
AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu
mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy
QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7
VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK
lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm
nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14
4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd
KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3
-----END RSA PRIVATE KEY-----

Save the file. I called my file  keypair-test1

2. Create an instance

Now, we need to create a new instance in cloudstack. We have to use the template we created above. We also will have to use the ssh key name from above. Note: you can not create the instance via the gui at this time and associate the instance with the newly created ssh keypair. A sample curl command to create a new instance would be:

Note: Please substitute the template, service offering and security group IDs (you may not need to specify a security group if you are not using that feature) that are in your cloudstack environment:

curl --globoff http://localhost:8096/?command=deployVirtualMachine\&zoneId=1\&serviceOfferingId=18727021-7556-4110-9322-d625b52e0813\&templateId=e899c18a-ce13-4bbf-98a9-625c5026e0b5\&securitygroupids=ff03f02f-9e3b-48f8-834d-91b822da40c5\&account=admin\&domainid=1\&keypair=keypair-test1

3. Login using the ssh key

Assuming everything above went well, you should be able to login into you new instance without using a password, but instead using your keypair:

From a linux cli, I ran this:

ssh -i ~/.ssh/keypair-test1 192.168.1.100

The -i parameter tells the ssh client to use a ssh key found at ~/.ssh/keypair-test1.

Notice it just works and doesn't ask for a password:

 [root@cloudstack1]# ssh -i ~/.ssh/keypair-test1 192.168.1.100
Last login: Mon Mar 26 19:58:30 2012 from 1.2.3.4
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# hostname
faa9ccca-30fe-4e4b-9645-d878514f0966
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# cat /etc/redhat-release
CentOS release 5.6 (Final)
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# uname -a
Linux faa9ccca-30fe-4e4b-9645-d878514f0966 2.6.18-238.el5xen #1 SMP Thu Jan 13 16:41:45 EST 2011 x86_64 x86_64 x86_64 GNU/Linux




Good luck and happy hunting.
Chris

Welcome to my new blog. I'll be writing about internet infrastructure, data center networks, infrastructure automation, and clouds.