tag:blogger.com,1999:blog-32450996826401746522024-03-26T01:10:06.517-07:00Internet Infrastructure by Chris KlebanAll things considered. But, only things regarding tech companies with an Internet presence. Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-3245099682640174652.post-9557750732767076232016-06-09T11:32:00.002-07:002016-06-09T11:32:43.991-07:00Machine Learning + Transfer learning + tensorflow for Kaggle CompOne of the things I'm currently doing with my time is predicting the future with machine learning. One thing I'm doing is a Kaggle districted driver detection competition. I recently posted some starter code for doing transfer learning with Convnets for others to use if interested. Check out the discussion and code at the below links.<div>
<br /></div>
<div>
https://www.kaggle.com/c/state-farm-distracted-driver-detection/forums/t/21552/starter-script-for-using-the-pre-trained-inception-v3-model-tensorflow</div>
<div>
<br /></div>
<div>
https://github.com/ckleban/kaggle-distracted-drivers-inceptionv3</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Thanks</div>
<div>
Chris</div>
<div>
<br /></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com20tag:blogger.com,1999:blog-3245099682640174652.post-19954337676922778202016-05-31T15:33:00.001-07:002016-05-31T15:53:32.447-07:00Cloud Services Redone - Part 2 of 2<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Anywhere Cloud Services - Thanks to Docker & Kubernetes</span></h3>
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="font-weight: normal;">Cloud Services without the Lock In </i></span></h3>
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: normal;"><i>Part 2 of 2</i></span> </span></h3>
<div style="background-color: white; font-size: 14.85px; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="line-height: 20.79px;">In </span></span><a href="http://chriskleban-internet.blogspot.com/2015/12/cloud-services-redone-part-1-open.html" style="line-height: 20.79px;">part one</a><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="line-height: 20.79px;"> of this two part post I wrote about how open source software will be delivered as services instead of as binaries. These open source services will deliver a free, open, flexible service in which engineers can use to help build their applications without everyone all trying to solve the same problem of figuring out how to keep the software up and running. If you haven't read that post, I encourage you to read it before proceeding with this post. </span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<i style="background-color: white; line-height: 20.79px;"><span style="font-family: "arial" , "helvetica" , sans-serif;">These posts will mostly describe a vision I have but will also include some back story and some technical information. The attended audience is anyone building or using applications. </span></i><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="background-color: white; font-size: 14.85px; line-height: 20.79px;"><br /></i>
</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="background-color: white; font-size: 14.85px; line-height: 20.79px;"><br /></i></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Where are Cloud services Today? </span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="background-color: white;"><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">Today's application level cloud services like AWS's Kinesis or Google's Big Table provide engineers fully managed services in which you can build an application on top of that require no effort to maintain. These services are secure, autoscaled, monitored, logged, integrated and updated. Personally, I love these services. But...</span></span></span><br />
<ul>
<li><span style="line-height: 20.79px;"><span style="font-family: "arial" , "helvetica" , sans-serif;">You are locked into these services via your code (ie, changing database backends would need a software change)</span></span></li>
<li><span style="line-height: 20.79px;"><span style="font-family: "arial" , "helvetica" , sans-serif;">By choosing a public cloud service, you also are forced to use their hardware. What if you run on Microsoft Azure but love Big Table?</span></span></li>
<li><span style="line-height: 20.79px;"><span style="font-family: "arial" , "helvetica" , sans-serif;">What if you want to use open source software but you don't want to operate, scale and secure open source software myself?</span></span></li>
</ul>
<br />
<span style="background-color: white; line-height: 20.79px;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Enter anywhere cloud services. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="background-color: white; font-size: 14.85px; line-height: 20.79px;"><br /></i>
</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><i style="background-color: white; font-size: 14.85px; line-height: 20.79px;"><br /></i></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Model #2 - Anywhere Cloud Services</span></h3>
<div style="background-color: white; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="background-color: white; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">An anywhere cloud service is a fully managed service that can be delivered anywhere and should have these characteristics:</span></div>
<div style="background-color: white; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b style="background-color: white; line-height: 20.79px;">It is fully managed -</b><span style="background-color: white; line-height: 20.79px;"> The consumer doesn't need to worry about keeping the service online. This responsibility is owned by the service provider. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b style="background-color: white; line-height: 20.79px;"><br /></b>
<b style="background-color: white; line-height: 20.79px;">It runs anywhere </b><span style="background-color: white; line-height: 20.79px;">- The consumer chooses where the service lives. This could be any one of the major cloud providers or a private cloud with a standard API. There is a clear separation between the entity running the infrastructure and the service running on top of it. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b style="background-color: white; line-height: 20.79px;">It is consumed as a service, not as software -</b><span style="background-color: white; line-height: 20.79px;"> It provides a function to the consumer in a clearly consumable method. Ie, a NoSQL service provides a way to insert data and retrieve data. The services heals itself. It scales when required. It provides metrics, logs and alerts to the consumer. It is highly available. The service is installed as easy as running a command or clicking on a button.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="background-color: white; font-size: 14.85px; line-height: 20.79px;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
Give me some Examples...</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">To help illustrate this idea, here are a few examples that we might see in the future:</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Any of AWS's non-infrastructure services, like the NoSQL DynamoDB service, can be provisioned in all of the other cloud providers. Ie, Azure, GoogleCloud, Etc. </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Google's non-infrastructure services, like their PaaS App Engine, can be deployed on any public cloud or even in on-premise data centers via openstack</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">A startup goes after the Docker container space by building a service that will run your application/code on any public or private cloud. </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">A development team decides on a data stack of Kafka, Spark and Cassandra because they want the benefits of open source software. However, they don't want to pay consultants or do operations themselves. So, they use a new service made by a startup that specializes in delivering these software stacks in any location desired. </span></li>
<li><span style="background-color: white;"><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">Creators of open source software will package their software up so that is can be easily deployed as a service in addition to supping the source code and binaries. Companies will also offer 'supported' versions of these services at a cost. </span></span></span></li>
</ul>
</div>
<div style="background-color: white; margin: 0px; position: relative;">
<span style="font-weight: normal;"><span style="font-family: "arial" , "helvetica" , sans-serif;">All of these are possible, thanks to Anywhere Cloud Services</span></span></div>
<div style="background-color: white; margin: 0px; position: relative;">
<span style="font-weight: normal;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div>
<div style="background-color: white; margin: 0px; position: relative;">
<span style="font-weight: normal;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
How do we get to Anywhere Cloud Services? </span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Their were two big hurdles to Anywhere cloud services:</span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<ol>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;">Standard and efficient way to package and deploy software services across the multiple cloud providers and on premise hardware</span></span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;">Efficient way to package, run, deploy, scale and secure your services (NoOps)</span></span></li>
</ol>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Enter Docker and Kubernetes!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">Docker enables us to easily build, ship and run software by packaging it up in a way that will run on a wide range of systems. By using containers, cloud service providers can run their software in any environment. But, that isn't enough. We need a way to get that software installed, working and highly available. We need something like Kubernetes. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="line-height: 20.79px;"><br /></span></span></span>
<span style="background-color: white; line-height: 20.79px;">Once the software is packaged in Docker containers we can turn their software into services by using kubernetes's features. Kubernetes's goal is to take a set of infrastructure (that lives anywhere) and provide an API to create services on top of that infrastructure. Kubernetes includes features like load balancing, scaling, rolling-updates, service discovery, name spaces, APIs, scheduling and configurable infrastructure threshold limits. Using these features, we can create declarative services and give control of the service to the consumer. </span><span style="background-color: white; line-height: 20.79px;">Kubernetes provides the No in NoOps while providing developers a single API to launch into both public and private clouds.</span><span style="background-color: white; font-size: 14.85px; line-height: 20.79px;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; font-size: 14.85px; line-height: 20.79px;"><br /></span>
</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
How will we consume Anywhere Cloud Services?</span></h3>
<div style="background-color: white; font-size: 14.85px; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">First, consumers will need to have one or more kubernetes clusters. Getting kubernetes up and running can take as little as 5 minutes or as long as a few days depending on your situation. Kubernetes supports most public clouds, openstack clouds and on-premise data centers. Consumers can also choose whether to run on top of VMs or physical machines. We can run kubernetes ourselves or we can use a managed option if we are looking for increased support. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;"><br /></span>
<span style="background-color: white; line-height: 20.79px;">Notice that the consumer is in complete control here of where their applications will run as they will be freed from any lock-in. Thanks to the common API that lives in any infrastructure environment, consumers can use these clusters to deliver on a multiple cloud provider strategy or they can create a true hybrid strategy.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br style="background-color: white; line-height: 20.79px;" /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;">Second, consumers simply consume these services by having these services deploy their code into the customer's Kubernetes clusters. Either through an app-store like experience or through a developer friendly like experience, users will search for a service that meets their needs and click install. The </span><span style="background-color: white;"><span style="font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="line-height: 20.79px;">service provider would than reach out to the Kubernetes cluster in question, either in the cloud or on-premise, and install the service. Developers could than immediately start using the service. The service provider's job is to ensure the service is running, healthy and scale it as required. </span></span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;"><br /></span>
<span style="background-color: white; line-height: 20.79px;">In this new world, consumers will have more control of their application and the services that they depend on. They can move our applications between clouds or between private and public. Consumers can run their applications actively across multiple clusters or do active-standby. All this is sweetness.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; line-height: 20.79px;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
In summary, Anywhere Cloud Services</span></h3>
<div style="background-color: white; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">With anywhere cloud services, developers will be able to build their applications the way they want while not having to worry about running and operating all their back end services. Tech execs get freedom from cloud lock-in and have a path to a true hybrid solution. Public cloud providers get access to customers on other clouds that don't want to move. Open source projects get a new path to revenue.. Entrepreneur's get a brand new gold rush as their are plenty of challenges and opportunities.</span><br />
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;"><br /></span>
<span style="background-color: white; font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h3 style="background-color: white; margin: 0px; position: relative;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">
In summary - Anywhere and Open Source Services</span></h3>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In 5 years, cloud services will be just like software in that you go buy some and install it where you want. Pretty easy. </span></div>
<div style="background-color: white; line-height: 20.79px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">Thanks</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Chris Kleban</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; line-height: 20.79px;">Twitter: @kleban</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div style="font-size: 14.85px;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="font-size: 14.85px;">
<br /></div>
</div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com17tag:blogger.com,1999:blog-3245099682640174652.post-32681799553876627002015-12-02T11:30:00.002-08:002015-12-02T11:50:22.944-08:00Cloud Services Redone - Part 1 of 2<h3>
Disruption in Open Source Delivery, the Open Source Service </h3>
<div>
<br /></div>
What if we took the positives of open sourced software and merged them with the positives of cloud services like DynamoDB, BigTable, Kinesis, Lambda and S3? We would have a disruptive service delivery model.<br />
<br />
In a two part post, I'll talk about two new approaches to delivering software as a service so that engineers can build better applications. One model I will call Open Sourced Services (Part 1) and another model I will call Managed Anywhere Services (to-be-written Part 2).<br />
<br />
<i>These posts will mostly describe a vision I have but will also include some back story and some technical information. The attended audience is anyone building or using applications. </i><br />
<i><br /></i>
<br />
<h3>
Where are we today?</h3>
<div>
<br /></div>
<h2>
Today's Cloud Services </h2>
<div>
<br /></div>
Today's application level cloud services like AWS's Kinesis or Google's Big Table provide engineers fully managed services in which you can build an application on top of that require no effort to maintain. These services are secure, autoscaled, monitored, logged, integrated and updated. Engineers can focus on their particular applications versus worrying about things like mysql cluster management or file replication. Personally, I love these services. However, these services lock you into a single cloud vender and you loose control of the functionality of these services. What if we could get the benefit of services without the lock in?<br />
<br />
<h2>
Today's Open Source </h2>
<div>
<br /></div>
Open source software has many benefits. The source code is open, which gives you increased control. The software is free which helps drive adoption and potentially lowers cost. It gives back to the community by enabling the free re-use of software. And it frees you from cloud lock-in as you can easily more your solution anywhere. However, open source software is normally delivered in a software package in which someone needs to install, monitor, configure, scale and upgrade so that it can be used in production. Each company often does all this a different way and their work isn't reusable by others. Isn't that a shame? What if we could take the increased control, the re-use of code, freedom from cloud lock-in and the free price tag of open source software but deliver it as a service?<br />
<br />
<h3>
Well, what do we want? </h3>
<div>
The three main items that stick out are:</div>
<ul>
<li>services</li>
<li>control </li>
<li>portability (no lock in)</li>
</ul>
<div>
Below, I will talk about one way how we can get there. </div>
<div>
<br /></div>
<h3>
Model #1 - Open Sourced Services</h3>
<div>
<br /></div>
<div>
An open source service is the packaging and delivery of open source software such that it is delivered as a service. An open source service should have these characteristics:</div>
<div>
<br /></div>
<b>It is consumed as a service, not as software -</b> It provides a function to the consumer in a clearly consumable method. Ie, a NoSQL service provides a way to insert data and retrieve data. The services heals itself. It scales when required. It provides metrics, logs and alerts to the consumer. It is highly available. The service is installed as easy as running a command or clicking on a button.<br />
<br />
<b>It is controlled as a service, not as software </b>- The consumer takes action on the service itself versus it's subcomponents. They can initiate upgrades of the service. They can start, stop or restart the service as needed. The service should expose configuration knobs so that the consumer can control the level of redundancy, user access, scaling parameters (auto scaling, infrastructure thresholds, etc), capacity and user access.<br />
<br />
<b>It can run anywhere -</b> The service should be able to be deployed anywhere. In a public cloud or in your company's data center.<br />
<br />
<b>It is open - </b>The source code of the software and the declarative definition of how the software is wrapped as a service should both be open sourced. This will give the the consumer of the service the control they desire.<br />
<br />
<h3>
How do we get to Open Sourced Services? </h3>
<div>
<br /></div>
The creators of open source software have not had an easy and unified way to package up their software in a way that meets the above criteria until recently. Thanks to the disruptive technology of containers, such as docker, and orchestration systems (such as kubernetes, swarm and mesosphere), I believe we now have a platform in which we can build our open sourced services on top of.<br />
<br />
<i>Note: I'll be using Docker and Kubernetes in the rest of this article but you should know that there are alternatives (swarm, mesospahre, rocket, oci, etc). </i><br />
<br />
Docker enables us to easily build, ship and run software by packaging it up in a way that will run on a wide range of systems. If open source groups takes their code and packages it inside containers they will have the building blocks required for turning their software into a service (that can run anywhere).<br />
<br />
Once the software is packaged in Docker containers we can turn their software into services by using kubernetes's features. Kubernetes's goal is to take a set of infrastructure (that lives anywhere) and provide an API to create services on top of that infrastructure. Kubernetes includes features like load balancing, scaling, rolling-updates, service discovery, name spaces, APIs, scheduling and configurable infrastructure threshold limits. Using these features, we can create declarative services and give control of the service to the consumer. Kubernetes helps groups deliver on the above characteristics for what makes up an open source service.<br />
<br />
Creators of open source software now have a way to package their software up and deliver it as a service. Sweet!!!! So, how would we consume these new services?<br />
<br />
<h3>
How will we consume open sourced services?</h3>
<div>
<br /></div>
First, consumers will need to have one or more kubernetes clusters. Getting kubernetes up and running can take as little as 5 minutes or as long as a few days depending on your situation. Kubernetes supports most public clouds, openstack clouds and on-premise data centers. Consumers can also choose whether to run on top of VMs or physical machines. We can run kubernetes ourselves or we can use a managed option if we are looking for increased support. Notice that the consumer is in complete control here of where their applications will run as they will be freed from any lock-in. Thanks to the common API that lives in any infrastructure environment, consumers can use these clusters to deliver on a multiple cloud provider strategy or they can create a true hybrid strategy.<br />
<br />
Second, consumers would need to choose and install some open source services . For example, let's say they want to use a noSQL service for their application. Previously, their choices where using something like AWS's DynamoDB (with AWS lock-in) or they could have engineered their our own Cassandra cluster with a bunch of blood and sweat. With open source services, all consumers will need to do is install the cassandra service on their kubernetes cluster of choice. They might do this with an easy command line tool ('kubernetes install service cassandra') or perhaps by logging into an app store and clicking on the cassandra service. Consumers will want to control the service by setting some parameters, like capacity limits and redundancy options, so that the service meets their needs. After that, the consumer simply starts using the service and keeps an eye on all the fun metrics and logs the service provides. <br />
<br />
In this new world, consumers will have more control of their application and the services that they depend on. They can move our applications between clouds or between private and public. Consumers can run their applications actively across multiple clusters or do active-standby. All this is sweetness.<br />
<br />
<h3>
In summary</h3>
<div>
<br /></div>
With open source services there are lots of winners: Developers will be able to build their applications the way they want to while getting the control they need. Tech execs get freedom from cloud lock-in and have a path to a true hybrid solution. Public cloud providers get to continue to sell infrastructure. Open source projects get a way to easily package their software as a service which increases their user's perceived value. Entrepreneur's get a brand new gold rush as their are plenty of challenges and opportunities.<br />
<br />
<h3>
Up next - Marriage #2 - Managed Anywhere Services</h3>
<div>
<br /></div>
<div>
One of the down sides to open sourced services is that sometimes there is no one to call in case things go wrong. In my next post I'll write about an alternative approach that aims at solving that problem and opens up a new way for companies to make $$$ by delivering these services.<br />
<br />
End<br />
<br />
Thanks<br />
Chris<br />
<br />
--<br />
<br />
<b>Some useful links:</b><br />
<ul>
<li><b>Learn more about containers and kubernetes:</b> http://kubernetes.io/v1.0/docs/whatisk8s.html</li>
<li><b>A list of some early versions of open source software packaged up as services: </b>https://github.com/kubernetes/kubernetes/tree/master/examples</li>
</ul>
<br /></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com8tag:blogger.com,1999:blog-3245099682640174652.post-36678672145111493322015-08-26T09:51:00.002-07:002016-11-28T14:47:32.808-08:00Real-time Data Distribution with Apache Kafka<br />
<br />
I wrote a blog post about CenturyLink Cloud's usage of Kafka for CenturyLink Cloud's website. That post, in it's full content, can be found below. Thanks. Chris<br />
<br />
<div style="border-bottom: solid silver 1.0pt; border: none; mso-element: para-border-div; padding: 0in 0in 4.0pt 0in;">
<h1>
Real Time Data Distribution with Apache Kafka<o:p></o:p></h1>
</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Like most enterprises and service providers, here at
CenturyLink Cloud we aim to please our customers by making data based decisions.
As we grew our business, so does the amount of data we collect. The effort
required to distill, distribute and analyze the data in meaningful ways was
increasingly strenuous. In short, we needed to become faster with our
analytics. If your organization is becoming overwhelmed with data, read on, as
I’ll share with you how we used Apache Kafka to solve the following challenges
with collecting and distributing data at scale.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Efficient access to data <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Speed<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Scalability <o:p></o:p></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;">·<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Distributed, Fault Tolerant and Highly Available<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<h2>
<a href="https://www.blogger.com/null" name="challenge-1-efficiency-with-data-distrib"></a>Challenge #1: Efficient
access to data <o:p></o:p></h2>
<div class="MsoNormal">
Getting access to data is sometimes the hardest part. Sometimes
the data that you need lives in some isolated software service and a one off
data export is required. Perhaps your data integration flow is overly complex
due to your adoption of a service-oriented-architecture. Maybe you had a process
that required multiple teams to due work before data moved around. We saw all
of this and more. We wanted to make it easier to move data around which would
increase the overall velocity of our organization. Our data distribution flow
between services looked something like this:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Previous Data
Distribution Model</b><o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2s78Z7GYs-B7FlWRklNoPFfLPptMoLtcswt39rNnmk-DeXNzDc6gOYk-ApMDMPCTW6fiuqAxlPH5nocaDdyQttqdLbHmK7T-W2J42jHaLnhcH6O3LkXmtYplnC1HzbDLXmGDGDKdR3H0/s1600/1.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="104" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2s78Z7GYs-B7FlWRklNoPFfLPptMoLtcswt39rNnmk-DeXNzDc6gOYk-ApMDMPCTW6fiuqAxlPH5nocaDdyQttqdLbHmK7T-W2J42jHaLnhcH6O3LkXmtYplnC1HzbDLXmGDGDKdR3H0/s320/1.png" width="320" /></a></div>
<div>
<br /></div>
<br /><div class="MsoNormal">
To fix this, we centralized our data flow with Apache Kafka.
Kafka is commonly referred to as a pub/sub (publish and subscribe) messaging
system. Simply put, systems publish their data into a Kafka ‘topic’. Other
systems subscribe to the desired ‘topic’ within Kafka and extract the data.
Kafka acts as a messenger, moving data to where it needs to go. Kafka, or any
pub/sub messaging system, decouples the publishing of data from the
subscription of data. Publishers only need to worry about sending the message.
Subscribers only need to worry about receiving the message. It’s important to
know that Kafka is not intended to be used a permanent data store. Rather, it
provides a reliable, fast and scalable method to distribute data from data
publishers to subscribers. Data storage is left to the consumers. By using
Kafka we had an asynchronous, decoupled, centralized communication mechanism
that looked like this:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Data Distribution
Model with Kafka</b><o:p></o:p></div>
<br /><div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKLzJor6dGIRkulcI0qEDMBLUOtMru4g53UfP0Wfw1QEyWnyn8q5Ebl2x7f7Aw9vlpWf-V0_Z_lE5azdla849RyHzD6o56nx6I4w01CDo1U5SsVTX1hoaPGqA7z4HaQQOACAYkM5lwNc/s1600/2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijKLzJor6dGIRkulcI0qEDMBLUOtMru4g53UfP0Wfw1QEyWnyn8q5Ebl2x7f7Aw9vlpWf-V0_Z_lE5azdla849RyHzD6o56nx6I4w01CDo1U5SsVTX1hoaPGqA7z4HaQQOACAYkM5lwNc/s320/2.png" width="320" /></a>We moved away from the many-to-many data distribution model
to a centralized model. All of our services only need to send their data once,
regardless to the number of destinations for the data. This also means that our
services only need to pull from one source versus worrying about implementing a
variety of different data integrations technologies. This centralization design
reduced the overall effort spent by our engineers distributing data. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<i style="mso-bidi-font-style: normal;">Note: There are
alternative messaging software systems out there. RabbitMQ is similar in
capabilities and may suite your needs better depending on what you need. <o:p></o:p></i></div>
<h2>
<a href="https://www.blogger.com/null" name="challenge-2-the-need-for-speed"></a>Challenge #2: The Need for Speed<o:p></o:p></h2>
<div class="MsoNormal">
Time is money. The quicker we can collect, detect, process,
predict, and take action on our data, the faster we can act on behalf of our
customers. We needed a data distribution model that enabled both near real time
(streaming) analytics as well as the more traditional batch analytics. Enter
the Lambda Architecture framework. The <a href="http://lambda-architecture.net/"><span class="Link"><span style="text-decoration: none; text-underline: none;">Lambda
framework </span></span></a>calls for all new data to be distributed
simultaneously to both stream and batch processing pipelines, as such: <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Lambda’s view of data
distribution</b><o:p></o:p></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuQrBlZqZKb_9P4TeT0uGsIq3XVSsA87ZSLX2p48Q4qjejjg0Fyt-Kdj0UGMLcaz1gTqLZAsG6e_934b7uNIE2JY_dk2lWDHOxxEjGuKlckSShvZ60PMy3WhaU2feB4RvOMIw8THBI1uM/s1600/3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuQrBlZqZKb_9P4TeT0uGsIq3XVSsA87ZSLX2p48Q4qjejjg0Fyt-Kdj0UGMLcaz1gTqLZAsG6e_934b7uNIE2JY_dk2lWDHOxxEjGuKlckSShvZ60PMy3WhaU2feB4RvOMIw8THBI1uM/s320/3.png" width="320" /></a></div>
<div class="MsoNormal">
<br /><i><span style="font-size: xx-small;">
Source: <a href="http://en.wikipedia.org/wiki/Lambda_architecture#/media/File:Diagram_of_Lambda_Architecture_(generic).png"><span class="Link"><span style="text-decoration: none; text-underline: none;">http://en.wikipedia.org/wiki/Lambda_architecture#/media/File:Diagram_of_Lambda_Architecture_(generic).png</span></span></a></span></i><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Using the Lambda framework and Kafka for the messaging
queue, we gained the ability to seamlessly add streaming analytics when needed (we
use Apache Spark’s Streaming Module). In addition, it helps that Kafka itself
is fast. Kafka’s data structure is essentially log files. As data comes into
Kafka, Kafka appends the data to a log file. If subscribers are pulling data
out in real-time they are also reading from the end of the file. This allows
Linux’s page caching to store the needed data in memory for reads while using
disk drives for writes. Lambda, Spark and Kafka’s low latency messaging allows
us to make near real time decisions for our customers.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<i style="mso-bidi-font-style: normal;">Note: An alternative framework
to Lambda that is also worth looking into, as described in </i><a href="http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html"><span class="Link"><i style="mso-bidi-font-style: normal;"><span style="text-decoration: none; text-underline: none;">this article</span></i></span></a><i style="mso-bidi-font-style: normal;">, suggests that users can feed data into a
stream processing pipeline which in turn feeds the data into the batch
pipeline. <o:p></o:p></i></div>
<div class="MsoNormal">
<br /></div>
<h2>
<a href="https://www.blogger.com/null" name="challenge-3-scalability-and-performance"></a>Challenge #3:
Scalability <o:p></o:p></h2>
<div class="MsoNormal">
The amount of data we collect and process is large and growing.
We needed a data distribution model that will scale horizontally, use cloud
infrastructure, support large workloads and auto-scale without impacting the
data distribution flow. Kafka was built in order to scale and is has proven to
work well under heavy load, as described in this <a href="https://engineering.linkedin.com/kafka/running-kafka-scale"><span class="Link"><span style="text-decoration: none; text-underline: none;">LinkedIn</span></span></a><span class="Link"> post</span>. Kafka’s ability to scale is achieved through it’s clustering
technology. It relies on the concept of data partitioning in order to
distribute the load across members in a cluster. If you need more data
throughput in a particular location, you simply need to add more nodes to the
Kafka cluster. Many cloud providers, like ours, and infrastructure automation
frameworks provide the mechanisms to automate the creation and scaling of Kafka
clusters. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<h2>
<a href="https://www.blogger.com/null" name="challenge-4-distributed-fault-tolerant-a"></a>Challenge #4:
Distributed, Fault Tolerant and Highly Available<o:p></o:p></h2>
<div class="MsoNormal">
Losing data is bad. Having data stop flowing is bad. Having
data only available in one datacenter is bad. We needed a solution that wasn’t
bad and Kafka had us covered. Kafka supports data replication inside a cluster.
This means that even if a server in a Kafka cluster crashes, data in the Kafka
message bus would not be lost. Great. In addition to replication, Kafka’s uses
sequential IDs to enable the graceful restarting of a data flow. If a client
stops for any reason, another client can use the sequential IDs to pick up where
the other client left off. We process the data in aggregate or to have high
availability on the aggregate data, there is a need to replicate data to
multiple data centers. Kafka provides an easy mechanism to have a Kafka
clusters pull data from other Kafka clusters in other data centers. We use this
to pull data from all our data centers to a single data center in order to
preform analytics in aggregate. Kafka is reliable, distributed and fault
tolerate. It just works. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<i style="mso-bidi-font-style: normal;">Note: The delta in
sequential IDs between the publisher and subscriber is referred to as the
offset. The offset often used as a KPI metric on data flow through Kafka. <o:p></o:p></i></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<h2>
<a href="https://www.blogger.com/null" name="whats-wrong-with-kafka"></a>What’s wrong with Kafka?<o:p></o:p></h2>
<div class="MsoNormal">
So what’s wrong with Kafka? Not much. It solves our
challenges and the price is right, free. The one drawback with Kafka that I’d
like to call out is the complexity is puts on it’s clients. For example, each
client needs to connect to every Kafka server in a particular cluster versus
making only a single connection to the cluster. It needs to do this because the
data being put into a Kafka cluster is split across members of the cluster. Thankfully
there are SDKs, APIs and tools available that help engineers over come these
challenges if a little time is spent researching and testing these items. In
the end, we gladly accepted burden on the client in exchange for Kafka’s
reliability, performance and scalability.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Summary<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Revision>0</o:Revision>
<o:TotalTime>0</o:TotalTime>
<o:Pages>1</o:Pages>
<o:Words>1267</o:Words>
<o:Characters>7225</o:Characters>
<o:Company>CenturyLink Cloud</o:Company>
<o:Lines>60</o:Lines>
<o:Paragraphs>16</o:Paragraphs>
<o:CharactersWithSpaces>8476</o:CharactersWithSpaces>
<o:Version>14.0</o:Version>
</o:DocumentProperties>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves>false</w:TrackMoves>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>JA</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073711037 9 0 511 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;
mso-font-charset:2;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:0 268435456 0 0 -2147483648 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;
mso-font-charset:2;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:0 268435456 0 0 -2147483648 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
h1
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-link:"Heading 1 Char";
mso-style-next:"Body Text";
margin-top:9.0pt;
margin-right:0in;
margin-bottom:6.0pt;
margin-left:0in;
mso-pagination:widow-orphan;
page-break-after:avoid;
mso-outline-level:1;
tab-stops:right 5.0in;
border:none;
mso-border-bottom-alt:solid silver 1.0pt;
padding:0in;
mso-padding-alt:0in 0in 4.0pt 0in;
font-size:14.0pt;
mso-bidi-font-size:10.0pt;
font-family:Arial;
mso-bidi-font-family:"Times New Roman";
color:red;
mso-font-kerning:16.0pt;
mso-bidi-font-weight:normal;}
h2
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-link:"Heading 2 Char";
mso-style-next:"Body Text";
margin-top:12.0pt;
margin-right:0in;
margin-bottom:6.0pt;
margin-left:0in;
mso-pagination:widow-orphan;
page-break-after:avoid;
mso-outline-level:2;
font-size:11.0pt;
mso-bidi-font-size:10.0pt;
font-family:Arial;
mso-bidi-font-family:"Times New Roman";
mso-bidi-font-weight:normal;}
p.MsoBodyText, li.MsoBodyText, div.MsoBodyText
{mso-style-noshow:yes;
mso-style-priority:99;
mso-style-link:"Body Text Char";
margin-top:0in;
margin-right:0in;
margin-bottom:6.0pt;
margin-left:0in;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:"Times New Roman";}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Heading 1";
mso-ansi-font-size:14.0pt;
font-family:Arial;
mso-ascii-font-family:Arial;
mso-hansi-font-family:Arial;
color:red;
mso-font-kerning:16.0pt;
font-weight:bold;
mso-bidi-font-weight:normal;}
span.Heading2Char
{mso-style-name:"Heading 2 Char";
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Heading 2";
mso-ansi-font-size:11.0pt;
font-family:Arial;
mso-ascii-font-family:Arial;
mso-hansi-font-family:Arial;
font-weight:bold;
mso-bidi-font-weight:normal;}
span.Link
{mso-style-name:Link;
mso-style-unhide:no;
color:#4F81BD;
mso-themecolor:accent1;}
span.BodyTextChar
{mso-style-name:"Body Text Char";
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-unhide:no;
mso-style-locked:yes;
mso-style-link:"Body Text";
mso-ansi-font-size:12.0pt;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-size:10.0pt;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:425420329;
mso-list-type:hybrid;
mso-list-template-ids:583575394 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><br />
<div class="MsoNormal">
Kafka is the heart of our data stack. It pumps our data
around so we can quickly and accurately act on behalf of our customers. Before
Kafka, our data was moving around all willy-nilly. Post Kafka, we have
simplified, scaled and structured dataflow that enables our business to move
faster than was previously possible. Today, we are publishing operational
metrics, logs, usage and performance information into Kafka and we are planning
on importing additional types of data. We have a variety of software systems
consuming the data from Kafka: Apache Spark for real-time and batch analytics,
Cassandra for data storage, a SQL based data warehouse for reporting purposes,
custom written applications and ELK (Elasticsearch, logstash and kibana) for
operational visualization and search. Kafka is pumping here at Centurylink
Cloud.<o:p></o:p></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com1tag:blogger.com,1999:blog-3245099682640174652.post-11244148912466200682015-08-20T21:37:00.003-07:002015-08-26T09:47:56.030-07:00Minecraft on KubernetesThis is a post about launching a minecraft server, inside docker, on a kubernetes cluster.<br />
<h4>
Why do this?</h4>
<ul>
<li>I have a 5 year old son </li>
<li>we like to play games and build stuff with blocks</li>
<li>I'm interested in kubernetes</li>
<li>I saw this <a href="https://blog.docker.com/2015/06/minecraft-server-docker-1/">post</a> on minecraft in docker </li>
<li>I wanted to get to know minecraft and kubernetes better</li>
</ul>
<br />
<i>Note: After I got this working, but before I posted this, I came across <a href="http://www.blog.juliaferraioli.com/">Julia Ferraioli's series of blog posts about minecraft, docker and kubernetes.</a> These posts are good and you may want to check them out too.</i><br />
<i><br /></i>
<br />
<h4>
Docker</h4>
<br />
One of the advantages of Docker is sharing. In the public docker hub, there were a few different minecraft builds available. I choose to reuse Geoff Bourne's (@itzg) dockerfile listed <a href="https://hub.docker.com/r/itzg/minecraft-server/">here</a>. Its a great build that lets you specificy many server configuration options via environment variables so you don't have to worry about putting together a minecraft configuration file. With my container already created for me, getting minecraft up in kubernetes was easy.<br />
<h4>
</h4>
<h4>
</h4>
<h4>
Game on Kubernetes</h4>
<br />
First, you will need kubernetes cluster. Doesn't matter where your cluster is as the bueaty of kubernetes is this will work regardless of where your cluster is. Since I work at Centurylink Cloud, I used the scripts I created <a href="https://github.com/ckleban/k8-on-clc">here</a> to create my kubernetes cluster on CL cloud.<br />
<br />
<br />
Kubernetes has the concept of pods. A pod in k8 is a logical grouping of one or more containers, zero or more storage volumes and an IP address. We will launch the above minecraft docker container in a pod in kubernetes. And, since minecraft isn't a clustered server, we only need to run one pod per minecraft server.<br />
<br />
<br />
Here is the yaml file I used to create this pod.<br />
<i>(Also located here: https://github.com/ckleban/minecraft-k8)</i><br />
<br />
<br />
apiVersion: v1<br />
kind: Pod<br />
metadata:<br />
name: minecraft-server<br />
labels:<br />
app: minecraft-server<br />
spec:<br />
containers:<br />
- name: minecraft-server<br />
image: itzg/minecraft-server<br />
env:<br />
- name: OPS<br />
value: madsurfer79<br />
- name: EULA<br />
value: "TRUE"<br />
ports:<br />
- containerPort: 25565<br />
<br />
<br />
You will want to at least change the username in the OPS ENV variable to be your minecraft username. This let's the server know who has admin privileges on the server. You can add more settings, like game type, minecraft version, etc as you wish. To see a more detailed list of what options are available, go check out <a href="https://blog.docker.com/2015/06/minecraft-server-docker-1/">here.</a> Now, go create the minecraft server:<br />
<br />
-------------------------------------<br />
<br />
<br />
#Create pod<br />
<div class="p1">
<span class="s1">> kubectl create -f pod.yml</span></div>
<div class="p1">
<br />
#See it up and running:</div>
> kubectl get pods<br />
<div class="p1">
NAME READY STATUS RESTARTS AGE</div>
<div class="p1">
<span class="s1">minecraft-dfhse 1/1 Running 0 2h</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">#If you want this server to be reachable on the internet, you probably need to do some more things. If you are on a public cloud, like Centurylink, amazon or google, this is easy to do. Simply run:</span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1">> </span>kubectl expose pod minecraft-server --port=25565 --type=<span class="s1">LoadBalancer</span></div>
<div class="p1">
<br /></div>
<div class="p1">
You will be given a public IP address, which you can see by running:</div>
<div class="p1">
<br /></div>
<div class="p1">
> kubectl get services</div>
<div class="p1">
<br />
<br />
-------------------------------------<br />
<div>
<br /></div>
<div>
<br /></div>
<div class="p1" style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
<div style="margin: 0px;">
That is it. You and your friends can now play minecraft, inside docker, on kubernetes. </div>
</div>
</div>
<div class="p1">
<br />
A few things to note:<br />
<ul>
<li>you may need to wait a few minutes while you wait for the external IP address to be published. </li>
<li>you may also need to open up a firewall rule to this IP address and port. </li>
<li><b>this isn't a durable configuration.</b> If the pod dies, it won't come back online and the save games and world will be gone for ever. These problems are easily solved in kubernetes. We just need to add a replication controller and persistent storage. Perhaps, I'll work on that next and post it sometime. </li>
</ul>
</div>
<div class="p1">
<br />
Thanks</div>
<div class="p1">
Chris</div>
<div class="p1">
<br /></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com5tag:blogger.com,1999:blog-3245099682640174652.post-17200131165732283662015-07-27T23:30:00.002-07:002015-07-27T23:32:19.143-07:00Kubernetes + Boinc for a better world. Kubernetes, Docker and Boinc make it ridiculously easy to donate cpu cycles to solve earth's scientific challenges.<br />
<br />
<b>tl;dr:</b> <a href="https://github.com/ckleban/boinc-on-k8">https://github.com/ckleban/boinc-on-k8</a><br />
<br />
I figure, if we are deploying example applications to clusters to learn and test, why not deploy work loads that might add some benefit. So that is what I did. To get this all going, it was real simple. I reused a docker image that Ozzy Johnson <a class="ProfileHeaderCard-screennameLink u-linkComplex js-nav" href="https://twitter.com/ozzydidact" style="background: rgb(245, 248, 250); color: #8899a6; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 11.1999998092651px; text-decoration: none !important;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #8899a6; font-family: Helvetica Neue, Helvetica, Arial, sans-serif; line-height: 11.1999998092651px;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; line-height: 11.1999998092651px;">@</span></span><span class="u-linkComplex-target" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; line-height: 11.1999998092651px;">ozzydidact</span></a> put together and I created a replication controller yaml file to spin up a group of containers on a cluster. Check out the above github link for the source code and some further instructions.<br />
<br />
Here is me using all this:<br />
<br />
<span style="font-size: x-small;">MacBook-Pro:boinc ChrisKleban$ <b>kubectl create -f ~/GitHub/docker-boinc/boinc-rc.yml </b></span><br />
<span style="font-size: x-small;">replicationcontrollers/boinc-workers</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">MacBook-Pro:boinc ChrisKleban$ <b>kubectl get rc,pods</b></span><br />
<span style="font-size: x-small;">CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS</span><br />
<span style="font-size: x-small;">boinc-workers boinc-workers ckleban/boinc-on-k8 name=boinc-workers,version=v1 1</span><br />
<span style="font-size: x-small;">NAME READY STATUS RESTARTS AGE</span><br />
<span style="font-size: x-small;">boinc-workers-oprmg 1/1 Running 0 24s</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">MacBook-Pro:boinc ChrisKleban$ </span><b style="font-size: small;">kubectl scale rc boinc-workers --replicas=20</b><br />
<span style="font-size: x-small;">scaled</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">MacBook-Pro:boinc ChrisKleban$ <b>kubectl get rc,pods</b></span><br />
<span style="font-size: x-small;">CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS</span><br />
<span style="font-size: x-small;">boinc-workers boinc-workers ckleban/boinc-on-k8 name=boinc-workers,version=v1 20</span><br />
<span style="font-size: x-small;">NAME READY STATUS RESTARTS AGE</span><br />
<span style="font-size: x-small;">boinc-workers-036t8 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-5yjas 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-6jogy 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-frp3w 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-giob9 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-h55wg 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-hlh9k 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-idcds 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-j7uln 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-kg6nb 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-lgzkd 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-ngbz5 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-nvdi9 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-oprmg 1/1 Running 0 1m</span><br />
<span style="font-size: x-small;">boinc-workers-s1m5t 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-twnoj 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-wixrg 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-xge9n 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-xgy33 1/1 Running 0 17s</span><br />
<span style="font-size: x-small;">boinc-workers-yo0du 1/1 Running 0 17s</span><br />
<br />
<br class="Apple-interchange-newline" />
Sometime soon I'll be using this to test out kubernetes's current resource quota features to ensure a group of containers do not use more than a specified amount of CPU resources in a cluster.<br />
<br />
In the future, kubernetes will be releasing priority (QOS) feature which will allow users to specify which groups of containers should take priority when resource starvation occurs. Once this is realized, users will be able to run work loads like this without impacting the higher priority business functions occurring in the cluster. This end goal would be similar to how priority would be used to ensure batch jobs don't cause real time customer facing workloads due to resource constraints.<br />
<div>
<br /></div>
Enjoy and thanks<br />
Chris<br />
<br />
<br />
<br />Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com7tag:blogger.com,1999:blog-3245099682640174652.post-11723229257722634652015-07-09T14:48:00.000-07:002015-07-09T15:00:53.090-07:00Containers and Clusters - Disrupting how cloud services are delivered<div class="p1">
<h4>
The Problem</h4>
</div>
<div>
<br /></div>
<div class="p1">
Cloud computing and the higher level cloud services that public cloud companies offer have changed our industry. They allow developers to focus on solving customer needs versus worrying about servers, databases, asynchronous messaging, analytics, storage, media encoding, graphic rendering, content delivery and so on. These cloud services enable engineers to create applications quicker and reduce their costs. However, one of the draw backs of using the higher level cloud services is cloud lock in. The SDKs and APIs developers need to use to interact with these services are, for the most part, not standardized. If you use these services and you want to run the same application on another cloud or on premise, you need to rewrite some code. I don't think that this lock in should exclude people from using these services. However, we need a model that allows for these companies to provide innovative high level cloud services while also allowing us the freedom and flexibility of true portability. </div>
<div class="p1">
<br /></div>
<h3>
</h3>
<h4>
Disruption is needed </h4>
<div class="p1">
<br />
One approach could be to get all the cloud companies to standardize their services. Ha. Anyone see pigs flying? Another approach, and the one I believe in, would be for mainstream adoption of containers and clusters to provide a path for cloud services to run anywhere. While I don't think cloud providers will jump at this notion, I do think there is an ongoing technology change that might force their hands. If they don't jump on board, startups will create similar services and take away their market share. Heres how.</div>
<div class="p1">
<br /></div>
<h4>
Containers</h4>
<div>
<br /></div>
<div class="p1">
Containers will be the foundation of this revolution. Docker and the rest of the container technologies offer a way for an engineer to package their software, deliver it to a server and run the package anywhere. If you want to learn more about containers and docker you should google it or read this <a href="http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/">webpage</a>. Docker Docker Docker. It's all you here about these days. In time, containers will be main stream.</div>
<div class="p1">
<br /></div>
<h4>
Clusters</h4>
<div>
<br /></div>
<div class="p1">
In order to run a production service or application, you will need to build, run and manage many different containers. For redundancy, performance and scalability you will need to spread these containers across multiple servers and data centers. This is starting to sound complex. To make this easier, people have created clusters. </div>
<div class="p1">
<br /></div>
<div class="p1">
Kubernetes, Apache Mesos and Docker Swarm are all clustering and scheduling software frameworks that allow organizations to create a logical collection of compute power called clusters. These clusters are made up of servers or VMs and enable engineers to deploy their containers across the infrastructure. In addition, the cluster software also provides container replication, auto-scaling, load balancing, monitoring, logging, scheduling, resource management and so on. The end result is that we can create clusters on the hardware of our choice: in the cloud or on premise. We package up our code and deploy it wherever we want. True utility compute. </div>
<div class="p1">
<br /></div>
<h4>
Operating systems for Clusters</h4>
<div>
<br /></div>
<div class="p1">
To make things easier to manage, clustering software allows engineers to logically define groups of containers and various cluster attributes, like load balancing and security, into logically defined services. This allows engineers to manage services on their clusters, instead of a collection of containers. To make things easier for engineers, enter the concept of cluster operation systems. Two exist today: Mesosphere's data center operating system (DCOS) which offers a web-portal and CLI and some might consider kubernetes CLI tool a cluster operating system as well. These cluster operating systems make it easier for engineers to deploy and manage services on one or more clusters. They allow us to see the status of the cluster, the services, the containers and so on. All the things you need to run and maintain your service in production. Cluster operating systems will make deploying your services across multiple clusters a breeze. </div>
<div class="p1">
<br /></div>
<h4>
The app store for clusters </h4>
<div>
<br /></div>
<div class="p1">
It pretty obvious how this will enable us to do great things. But how will this change how cloud services are delivered? The answer will be a cluster service repository. Like an app-store for clusters. Want to run a HA database with your application? You will simply download the DB service you wish to integrate to your application. Want a web server? Go choose a web stack. Need caching? Need messaging? Simply pick your service. Then, you get to write code to utilize it, package it up and deploy it to whatever cluster you have running. </div>
<div class="p1">
<br /></div>
<h4>
Disruption in how services are delivered</h4>
<div>
<br /></div>
<div class="p1">
Folks in the community will create and manage services in the services repo based on open source software packages. I see traditional software companies packaging their software into cluster services and perhaps charing for licensing. I see startups jumping at the chance to be first to offer new services that run on these clusters. And the kicker, I see cloud providers packaging their existing services so that you can run a copy of their service in your cluster, perhaps for a fee. Imagine running services like AWS Kinesis, Google machine learning, Oracle DB, azure's data warehouse or whatever, anywhere you want. Awesome. </div>
<div class="p1">
<br /></div>
<h4>
The end state unicorn</h4>
<div>
<br /></div>
<div class="p1">
Containers, clusters, cluster operating systems, cluster services and a service repository will change the way we use data centers. It will change the way cloud services and open source software is packaged and delivered. This will have all the benefits of Platform as a Service. The control of doing things yourself. The dream of utility compute will be realized. Cloud provider lock in will be a thing of the past. Clustering and the service appstore will rock the cloud industry. </div>
<div class="p1">
<br /></div>
<h4>
What now</h4>
<div>
<br /></div>
<div class="p1">
There is still a lot of work to do. Some of the things I described are here today and some are just visions that are being worked on. These technologies have huge momentum and I'm personally very interested in all this. I believe in this disruption and I'll be investing in it one way or another. If you are in the cloud business you should consider this idea and decide if it's worth embracing. If you are an engineer or developer, keep an eye on these developments. </div>
<div class="p1">
<br />
<br /></div>
<div class="p1">
Excited </div>
<div class="p1">
--Chris</div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com1tag:blogger.com,1999:blog-3245099682640174652.post-26457066359922625442015-05-15T14:15:00.001-07:002015-05-15T14:26:36.771-07:00Global Internet Access with Lightbulbs and Mesh NetworkingWhat if the world was connected together by wireless enabled Lightbulbs and mesh networking software?<br />
<br />
<h3>
A problem worth solving</h3>
Many amazing people, groups and companies are working on how to better provide Internet and network access to the masses. Some ideas currently in development by the likes of Google, Facebook and SpaceX (to name a few) are: low orbit satellites, solar powered plains, hot air balloons and fiber to the home. All of these have merit and I applaud them for what they are trying to do and ultimately will do. But I think there is another idea to explore.<br />
<br />
<h3>
The internet isn't everywhere</h3>
<div>
One way to extend the Internet's reach is through distributed mesh networks. According to wikipedia, mesh networking is defined as ".. a network topology in which each node relays data for the network. All mesh nodes cooperate in the distribution of data in the network. Mesh networks can relay messages using either a flooding technique or a routing technique." </div>
<div>
<br /></div>
<div>
Let me explain how mesh networks can help. Take my home Internet connection that is provided by my cable company. Anything within wifi range of my router (100 feet?) has internet access. However, if I leave my house, I no longer have wifi access. If there was were hundreds or thousands of devices in the city that formed a mesh network, I would be able to use the mesh network to reach my home's Internet connection no matter where I went in the city. Or, if other people around the city offered to connect their Internet connections to the mesh network, I would be able to use the mesh network to reach the closet or best Internet connection based on where I was. </div>
<div>
<br /></div>
<h3>
Lightbulbs!</h3>
<div>
Let's look at some simple facts. Lightbulb sockets are everywhere. Some lightbulbs today have wifi. Some have computers in them creating 'Smart' lightbulbs. Some are energy efficient. I've recently read about a lightbulb product that has a built in speaker and bluetooth. This allows people to use lightbulbs in their house as a house wide speaker system. </div>
<div>
<br /></div>
<div>
What if we built lightbulbs for the purpose of acting as nodes in mesh networks? What if we put software code in these lightbulbs that join and create mesh networks automatically so that all someone needs to do is screw it in a socket? What if people, laptops, phones and IoT devices could freely connect to this light bulb enabled mesh network to communicate with each other and the internet? What if we put these lightbulbs all over the world? What if people, organizations, and Internet service providers connected their Internet connections to these mesh networks so that the mesh networks has gateways to the Internet? </div>
<div>
<br /></div>
<div>
We would have a series of mesh networks throughout the world that would together bring the Internet to the masses and to the billions of IoT devices that will be coming in the near future. </div>
<div>
<br /></div>
<h3>
Beyond Lightbulbs</h3>
<div>
Lightbulbs are just one way to create mesh networks. What if a bunch of other things do the same things: Cars, drones, consumer devices (phones, watches, laptops), home routers, artificial birds, etc. Some people are already working on these things which is amazing to see. We just need more of them and for all of the efforts to integrate, versus creating a series of isolated mesh networks.<br />
<br /></div>
<h3>
Privacy and Security</h3>
<div>
Besides basic Internet connectivity, others have privacy and security concerns. Software and protocols exists today on mesh networks that provide encryption, protection and anonymous network access services to users. These features can easily be enabled by the mesh provider or by the end user through overlay networks. <br />
<br /></div>
<h3>
Path forward</h3>
<div>
Communities are popping up that are organizing the creation and expansion of these mesh networks. (My local organization is https://seattlemesh.net/). It would be great to speed up this process with major investment in the hardware rollout and node creation (lightbulbs, cars etc). In the ideal world, governments, companies, communities, individuals and organizations all work together in order to roll out mesh networks that interact freely with one another. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com0tag:blogger.com,1999:blog-3245099682640174652.post-85251594040436637922015-03-18T12:33:00.003-07:002015-03-24T11:54:25.211-07:008 Tips - Build a Highly Available Service<span style="font-family: inherit;">Working at AWS, Citrix, Register.com, Above.net and CenturyLink has taught me a lot about availability and scale. These are the lessons I've learned over time. From infrastructure as a service to web applications, these themes will apply.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Build for failures</span></h3>
<span style="font-family: inherit;">Failures happen all the time. Hard drives fail, systems crash, performance decreases due to congestion, power outages, etc. Build a service that can handled failures at any level in the stack. No one server, app, database or employee should be able to cause your service to go offline. Test your failure modes and see how your system recovers. Better yet, use a chaos monkey to continuously test for failures.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Build across multiple failure domains</span></h3>
<span style="font-family: inherit;">A failure domain consists of a set of infrastructure that can all go down due a single event. Data centers and public cloud availability zones are examples of failure domains as they can go down due to one event (fire, power, network, etc). Build your service so that it actively serves customers in multiple failure domains. Test it. A simple example is to use global load balancing to route customers to multiple failure domains.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Don't have real time dependencies across failure domains</span></h3>
<span style="font-family: inherit; font-weight: normal;">Don't build a distributed system that relies on synchronous communication across failure domains to serve your customers. Instead, build systems that can independently service your customers completely within a failure domain and make any communication between failure domains asynchronous. Having inter-failure domain dependencies will increase the blast radius for any single outage and increases the overall likelihood of service impacting issues. Also, there are often network instabilities between failure domains that can cause variable performance and periods of slowness to your systems and your customers. One example of this is data replication. Don't require storage writes to be replicated accross failure domains before the client considers the data 'stored'. Rather, store it inside the failure domain and consider it committed. Handle any cross failure domain replication requirements asynchronously, IE, after the fact.</span><br />
<span style="font-family: inherit; font-size: 17px; font-weight: normal;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Reduce your blast radius</span></h3>
<span style="font-family: inherit;">If a single change or failure can impact 100% of your customers, your blast radius is too large. Break your system up in some way so that any single issue only impacts a portion of your customers. User partitioning, using multiple failure domains (global load balancing), rolling deployments, separate control planes, SOA and A/B testing are a few ways to accomplish this. One example is using partitioning for an email sending service. Assign groups of customers to different groups of email sending servers. If any group of servers has an issue, only a portion of your customers are impacted versus all of them.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Reduce the level of impact</span></h3>
<span style="font-family: inherit;">Having your service go completely down for a portion of your customers is much worse than only having a part of your service unavailable for a portion of your users. Break apart your system into smaller units. An example is user authentication. Consider having a scalable, read only, easily replicated system for user logins but have another system for account changes. If you need to bring down the account change system for whatever reason, your users will still be able to login to the service.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Humans make mistakes</span></h3>
<span style="font-family: inherit;">Humans are the reason for most service impacts. Bad code deployments, network changes with unintended consciousness, copy/paste errors, unknown dependencies, typos and skill-set deficiencies are just a few examples. As the owner of a service it is critical that you apply the appropriate level of checks, balances, tools and frameworks for the people working on your system. Prevent the inevitable lessons learned of one individual from impacting the service. Peer reviews, change reviews, system health checks and tools that reduce the manual inputs required for 'making changes' can all help reduce service impacts due to human error. The important thing here is that the sum of the things you put in place to prevent human error can not make the overhead of making change so high that your velocity falls to unacceptable levels. Find the right balance.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Reduce complexity</span></h3>
<span style="font-family: inherit;">I am not a fan of the band Kiss, but I do like to keep it simple stupid. A system that is too complex is hard to maintain. Dependency tracking becomes impossible. The human mind can only grasp so much. Don't require geniuses to make successful changes on your system.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">
Use the ownership model</span></h3>
<span style="font-family: inherit;">IE, use the devops model. If you build it you should also own it end to end (uptime, operations, performance, etc). If a person feels the pain of of a broken system, that person will do the needed to stop the pain. This has the result of making uptime and system serviceability a priority versus an after thought.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Good luck</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">--chris</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com0tag:blogger.com,1999:blog-3245099682640174652.post-43900776890690763942015-01-02T17:08:00.001-08:002015-01-02T17:08:34.116-08:005 Tips - Don't Screw Up the On-Site InterviewWho is the only one that can screw up an on-site interview? You are.<b> </b><br />
<br />
If you have successfully navigated past the phone screens and find yourself about to go on the on-site interview, chances are you are at least somewhat qualified for the job. Tech companies want to hire you. All you need to do is not screw it up by giving them a reason to not hire you. Pretty easy, eh?<br />
<br />
Your interviewers are trying to determine whether you can do the job and whether you can succeed given their environment. Your goal is to convince them that you can by leaving them with a positive impression of you. Enlighten them by following the below tips.<br />
<br />
<i>Note: This post is written as you are the interviewee. If you are in the interviewer seat, the below tips can be thought of as 'What to look for in a candidate'. </i><br />
<i><br /></i>
<h3>
Be Passionate</h3>
Love what you do. I do. You should too. If you aren't doing something you love, stop what your doing and do something else. People want to work with passionate people. No one wants a downer. No one wants someone just collecting a pay check. Passion is what drives action, improvement and ownership. It pushes people to do what others might think as unattainable. Be passionate. It's contagious. It will lead to amazing things (like a job offer at a great tech company).<br />
<br />
When I was interviewing for an engineer role many years ago, the interviewer asked me what I was passionate about. Being young and foolish, I answered, "Solving problems and girls". They asked me to elaborate. I told him how I've used technology's infinite source of problems to feed my problem solving addiction. In doing so, I laid out a few of my accomplishments and how the motivation behind them came from my passion within. I also described that I liked girls, like most boys. At the time, I was somewhat shy around women. I realized that in order to get a girlfriend I needed to change that about myself. I described how I thought of it has just another problem that needed to be solved. I then described my approach at overcoming this shyness by forcing myself to talk with girls. The passion I had drove desirable outcomes in both cases and I made it clear to the interviewer. I ended up getting the job, which I ended up loving. More importantly, my passion resulting in me meeting and marrying my wonderful wife.<br />
<br />
Be passionate, or go collect a pay check somewhere else.<br />
<br />
<h3>
<b>Be yourself</b> </h3>
Seriously. Be yourself. Proudly be yourself and show your potential new team that you are a human being. No one wants to hire a rock with good coding skills. Note: if you know of, or if you are, a rock with good coding skills, please contact me as we are hiring. Embrace what makes you unique, your strengths and your weaknesses.<br />
<br />
Own the message by connecting 'who you are' with 'how you will succeed in this role'. Some examples: you're desire to work alone allows you to go deep into complex problems, your interpersonal skills helps have helped those around you by creating a strong sense of teamwork, etc.<br />
<br />
If you are a jerk, or there is something about yourself that you know to be truly bad in some way, my advise is to solve that problem and improve yourself. Your life will be better off for it. And, it will help you nail the on-site interview.<br />
<br />
Be yourself so that they know you aren't a rock (with coding skills). Own the message of who you are so that you. Don't screw it up by pretending.<br />
<br />
<h3>
Have Confidence</h3>
If you doubt yourself, others will too, leading to no job for you. The hiring team needs to decide, with a limited amount of data, whether or not to offer you the job. Believing in yourself will help them believe in you. Don't give them a reason to doubt you by doubting yourself.<br />
<br />
Again, the key is to own the message. Don't let them paint the picture for you. If you have successes in your past, use them to show that your past success will help you succeed in this role. Don't let them assume it, connect the dots for them. Believe in yourself. Others will too.<br />
<br />
If you are lacking in some area for the role, be confident that you can succeed then prove to them why you will succeed. This will earn their trust and make them believe that you can really do it. Be up front with where you are weak but provide thoughts on how you will overcome this issue. If you haven't thought about this and you don't have a method to overcome your weakness, you are screwing up. This shows them that you can't solve your own problems. A candidate once said to me, "Currently, I don't know how to do this job. But, I believe I can overcome this by ... " We ended up hiring him because he owned the message, demonstrated his ability to learn new things, was passionate and addressed our concerns head on. He went on to become a super star and one of our best hires.<br />
<br />
Believe in yourself and others will too.<br />
<br />
<h3>
Be honest </h3>
Being caught in a lie will raise a red flag quicker than road runner can say, "Beep Beep!" You want to be genuine and come off as trustworthy. This sounds simple, but be honest throughout the interview. If you don't know an answer, simply say "I don't know". Even better, admit you don't know but follow it up with your thoughts on what the answer or solution could be and why. Doing that will show your integrity while also showing off your ability to think through problems.<br />
<br />
A very direct and common interview question that tests honesty and trust are questions about your past mistakes or your current areas of improvement. This is them lobbing up a softball pitch for you to nail out of the park. Be honest! Admit to bringing down the site. Admit that you aren't the best speller. This will prove to them that you can openly discuss when things go wrong and that you aren't full of yourself. Going deeper, if you can demonstrate that you have learned from your previous mistakes and that you took action to improve things, you will show them the passion and drive you have to improve yourself and those around you.<br />
<br />
<i>Note: I'd like to point out that some things are better left unsaid. Discretion and situational awareness are key everywhere, but especially key during interviews. I sometimes have problems with this myself and I over share information (like how I answered one of my passions was girls). Make your own decisions up on what to say or not say. However, error on the side of honestly an openness. </i><br />
<i><br /></i>
<h3>
Be able to get stuff done</h3>
Be able to get stuff done. To do that, you need to have the required skill-sets, both technical and non-technical. Either you are an expert at something or you are on your way to becoming one. There is no other option in which you will get an offer. If you are an expert, awesome. The only way you can screw it up is if you are in a passionless death spiral to the land of irrelevance.If you aren't an expert (few of us are), or if you are a newbie, you should in learning mode. Most people have the opportunity to learn regardless of where they are in life. Seek out new assignments at work, use your free time, take a class or read a book. Don't solely focus on your primary skill-set (like coding). Soft skills are equaling important to be good at and to improve. Always be learning. Be able to articulate what you have learned recently.<br />
<br />
Note, ability isn't having knowledge. Rather, its having the hard and soft skill-sets required to accomplish something. This is what makes or breaks a lot of interviews. Make sure you can, and have demonstrated in the past, the ability to accomplish things that are relevant to your new role. This is super important. I've seen countless college graduates come out of school without the ability to achieve something. Or, I've seen people with years of work experience require someone to hold their hand the entire way. But be like these people. Know how to get stuff done.<br />
<br />
Let me share with you a story about a candidate who knew nothing, but could do anything. I once had a network engineer candidate bring a ridiculously large notebook to his on-site interview. This thing contained every bit of reference data a network engineer could ever possibly need. During the interview, for almost ever question, he referenced the notebook in order to answer the questions that could be solved directly with knowledge. We didn't know if he did this out of habit or necessity. Given this fact, the surprising thing to us was that he was great at applying the technical information he had in order to solve problems and delight customers. We discussed these traits, which at the surface seem contradictory, at great length in the debrief. The outcome was that we gave him a very strong offer, but he unfortunately turned it down to go work for Google. Goes to show, if you can get stuff done, that's all that matters.<br />
<br />
Be able to demonstrate that you can 'get things done' given the knowledge that is available to you.<br />
<br />
<h3>
Have Common Sense</h3>
Don't be a jerk. Some people have common sense, others don't. I myself sometimes lack in this department from time to time.<br />
<br />
Before, during and after an interview, be on your best behavior. Do the little things well. Be polite, on time, respectful and show gratitude. Research the company, the people, the product and the market. Show a sense of ownership for yourself and everything around you. Don't be a know-it-all, too aggressive or smelly. The sum of all these little, common sense things, can tip the decision one way or another.<br />
<br />
I once saw someone show up 15 minutes late to both of his two on-site interviews. He didn't get the job. Another time, I saw how a candidate's simple followup email (which showed gratitude and passion) resulted in pushing A hiring manager's vote from no-hire to hire.<br />
<br />
Do the little things well.<br />
<br />
<h3>
In summary</h3>
Don't screw it up.<br />
<br />
<br />
Good luck,<br />
Chris<br />
<br />
PS, my company is hiring for lots of roles. Go check out our current openings at http://www.centurylinkcloud.com/careers<br />
<br />
PPS, yes, there were 6 tips. Cliche, I know.<br />
<br />Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com0tag:blogger.com,1999:blog-3245099682640174652.post-72670600174756251662012-04-02T14:00:00.001-07:002012-06-04T19:11:54.916-07:00Infrastructure Internet Markets - Then, now and next.<h2>
The garden of markets</h2>
The internet is like a garden. Things grow, as long as you water them. In fact, the internet is really like a continuously expanding cluster of gardens, each with a different environment (aka market). Sales, marketing, finance, communication, and everything in between.<br />
<br />
What I'll be talking about here are the infrastructure markets that grew to meet the needs of the internet.<br />
<br />
<br />
<h2>
Some new markets (gardens) of the past ... </h2>
<br />
<h4>
Connectivity, Dialup and beyond </h4>
First, there was the need for people and companies to be connected to the Internet. An untold amount of ISPs (internet service providers) were created to meet this need. They ranged from the size of your local mom and pop shop all the way to the size of AOL. Hundreds, thousands, hundreds of thousands of these companies were created world wide in order to meet the demand of humans wanting to communicate with one another. The ISP market will always exist and it will continue to grow and evolve. The speed of our connections will increase and along with it, what we do on the internet will change as well. <br />
<br />
<h4>
Web Hosting. Anyone remember geocities?</h4>
How many web hosting companies have existed in the last 15 years? A lot. If companies wanted to get on the internet, the most common way to do that at the time, was to pay a web hosting company to manage the server and internet connectivity for you. In this method, all you had to do was worry about creating and maintaining the content (which was a whole other market).<br />
<br />
<br />
<h2>
Today's new garden is... IaaS and PaaS</h2>
<h4>
Infrastructure as a Service (Iaas)</h4>
Remember the ISP and web hosting markets exploding? Well, IaaS and PaaS is next! It is now! <br />
<br />
<br />
When Amazon web services came out, it caused somewhat of a distribution in the industry. Why was this such a big event? Well, it was the first time internet infrastructure could be programatically controlled (can anyone say api). So simple, yet so game changing. Infrastructure as a service (IaaS) is really just web operations (data center, servers, storage, network) with an api in front of it. The benefits are huge to both internet based companies and tradition enterprise IT organizations. <br />
<br />
We are now in a gold rush. Everyone under the sun wants to offer an IaaS solution. No one wants to be left behind. ISPs, Data center providers, enterprises, web hosting companies, governments, education, and even individuals are all getting on the boat. All of these people are interesting in providing IaaS features to their clients. Both publicly and privately. <br />
<br />
There are now CMS (cloud management solutions) that allows these organizations to create an IaaS solution without the investment in writting their own code. Open Stack, Cloud stack, and eucalyptus seem to be the leading options available today. An organization can have an inhouse proof of concept up and running in a matter of hours. A production ready system can be ready in a few months.<br />
<br />
<h4>
Platform as a Service (Paas)</h4>
This is up a layer from IaaS. Instead of offering raw infrastructure, it offers a more of an application platform to its users. Someone only needs to upload their application, ie source code, and the PaaS will run and scale their solution as required.<br />
<br />
Salesforce, Engine Yard, AppEngine, and Azure are a few of the leading public PaaS providers. Now, there are also PaaS software solutions that will allow people to create their own PaaS. Openshift and cloudfoundry are a couple in this category.<br />
<br />
<h4>
If I where you</h4>
I would get into this scene today. In 5 years from now, no one will be provisioning infrastructure manually anymore. <br />
<h2>
</h2>
<h2>
</h2>
<h2>
What Next?</h2>
Obviously, more growth in IaaS and PaaS is coming. Software and solutions to help secure, monitor, provision, deploy all this stuff is currently being developed. But, what is the next garden? What is the next culture shift? <br />
<br />
If I knew the answer to this, I probably wouldn't be talking publicly about it. The point is, some new internet garden will arise in which we will all see ourselves shifting onto. <br />
<br />
Do you have a guess? Do you know?<br />
<br />
Thanks<br />
Chris <br />
<br />
<br />
<br />
<br />Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com1tag:blogger.com,1999:blog-3245099682640174652.post-11721857650184146992012-03-27T09:39:00.000-07:002012-03-27T09:51:52.456-07:00Cloudstack Integration - Detecting instance creation and deletion<h2>
So, you want to integrate your cloud?</h2>
<br />
Let us assume you have a cloud built on cloudstack. Now that you have this cloud, you may need integrate some of your other systems with cloudstack. Why? Well, for all those compute instances that you will run in the cloud, perhaps you will need to tell your other systems about them?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcSYQJUg5yxjsXGCvvM56Bi6EVX1d8LtVVTBdVqQ6gtdPTuVZOUZsg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://encrypted-tbn3.google.com/images?q=tbn:ANd9GcSYQJUg5yxjsXGCvvM56Bi6EVX1d8LtVVTBdVqQ6gtdPTuVZOUZsg" /></a></div>
Perhaps you have a monitoring system that you need to configure so that it knows about your new instances?<br />
<br />
Perhaps you need to register a dns entry?<br />
<br />
Perhaps you need to tell puppet or chef about your new instance?<br />
<br />
Perhaps you need to send an email to your boss asking for a raise every time a cloud instance is created?<br />
<br />
I had some of these issues and I found out that cloudstack didn't have a great way, out of the box, to run 'custom jobs' when an instance was created or destroyed. However, cloudstack does do a great job giving us the information we need to make this happen. So, with a little help from the folks @ cloudstack, here is what I came up with.<br />
<br />
<br />
<br />
<h4>
Create and destroy?</h4>
<br />
Let us assume we want to configure our monitoring system and tell it to start monitoring instances the moment they get created. And, since we care about cleaning up our monitoring system, we also care about removing the instance from the system when the instance is destroyed.<br />
<br />
So, how can we find out from cloudstack when these things occur?<br />
<br />
At first, I thought about putting something in the OS template itself so that the instance itself would do these things. However, I realized that this might not be the best thing for me a few reasons. Those reasons were:<br />
<br />
<ul>
<li>The instance will have no way of knowing if it is being destroyed. (versus just stopped). This would prevent the instance from calling the 'remove me' jobs. </li>
<li>I didn't want to manage and troubleshoot things on a lot of instances (versus a central location). </li>
<li>It might not be secure (assuming a lot of people would have access to the instances). </li>
</ul>
<br />
So, I decided to query cloudstack itself for this information. Here are some of the options that I came up with:<br />
<br />
<ul>
<li>Query the DB directly</li>
<li>Parse the log file for output</li>
<li>Access the cloud stack events via the api</li>
<li>Access the cloud stack instance information via the api</li>
</ul>
I ended up choosing to use the event information via the api. It had all the inforamtion I needed and using the api is always the best coarse if it is available to you. <br />
<br />
So, I wrote a script that polls for event info every X seconds. The script determines if there has been an instance that was created or destroyed successfully. And, if so, the script runs a bunch of integration tasks (like configure my monitoring system).<br />
<br />
I wrote the script in python using the python cloudstack library found here:<br />
<br />
<a href="https://github.com/jasonhancock/cloudstack-python-client">https://github.com/jasonhancock/cloudstack-python-client</a><br />
<br />
<br />
<h4>
How to get the events?</h4>
In order to get the list of events, I decided to use the api method 'listAsyncJobs to get the events. If we want to get the events for all users (not just the account's events), we need to pass the isrecursive and listall parameters correctly).<br />
<br />
<br />
<br />
Code snipnet: <br />
<br />
<br />
#!/usr/bin/python<br />
import CloudStack<br />
<br />
api = 'http://10.1.2.3:8080/client/api'<br />
apikey = 'xxxxxxxxxxxxxxxx'<br />
secret = 'xxxxxxxxxxxxxx'<br />
<br />
cloudstack = CloudStack.Client(api, apikey, secret)<br />
<br />
jobs = cloudstack.listAsyncJobs({'isrecursive': 'true', 'listall': 'true'})<br />
for job in jobs:<br />
<br />
<br />
<br />
<h4>
How to detect new instances?</h4>
Basically, you need to check for these things:<br />
<br />
- event cmd = ' com.cloud.api.commands.DeployVMCmd' <br />
- event jobstatus = 1 (this means the event was a success)<br />
<br />
If these two items are true, we know we have a new instance in our cloud. <br />
<br />
<br />
Code snipnet:<br />
<br />
<br />
# check for new vm jobs<br />
if ((job['cmd'])=="com.cloud.api.commands.DeployVMCmd"):<br />
if ((job['jobstatus'])>1):<br />
#print "SOMETHING WENT WRONG WITH NEW INSTANCE"<br />
if ((job['jobstatus'])<1):<br />
#print "VM being created now............"<br />
if ((job['jobstatus'])==1):<br />
#print "NEW VM was created successfully........."<br />
# Here is where we would add the part to configure our monitoring system<br />
<h4>
How to detect destroyed instances?</h4>
Basically, you need to check for these things:<br />
<br />
- event cmd = 'com.cloud.api.commands.DestroyVMCmd' <br />
<br />
- event jobstatus = 1 (this means the event was a success)<br />
<br />
<br />
If these two items are true, we know that a cloud instance was destroyed in our cloud. <br />
<br />
<br />
<br />
<h4>
How to get information about the instance in question? </h4>
Now that we know an instance was created or destroyed, our script will need to know some information about the instance in question. No problem!<br />
<br />
The data returned in the event itself will have this information!<br />
<br />
Below, you will see how I get the instance ID, hostname, and IP address. (other information is available if you need it)<br />
<br />
Now that your have that information, you call go create that dns entry or setup monitoring or whatever you need to!<br />
<br />
<br />
Code snipet:<br />
<br />
vmname=job['jobresult']['virtualmachine']['name']<br />
vmzoneid=job['jobresult']['virtualmachine']['zoneid']<br />
for nic in job['jobresult']['virtualmachine']['nic']:<br />
vmip=nic['ipaddress']<br />
zones = cloudstack.listZones({'id': vmzoneid})<br />
for zone in zones:<br />
vmdomain=zone['domain']<br />
vmhostname=vmname+"."+vmdomain<br />
<br />
print vmid, vmhostname, vmip<br />
<br />
<h4>
To sum it up </h4>
Now, if you want, you can take your cloudstack cloud and integrate it so that your instances in your cloud are fully and automatically integrated to all your other systems too.<br />
<br />
If someone is interesting in the full code I used, let me know and perhaps I can post it up somewhere. <br />
<br />
Thanks and happy hunting<br />
Chris<br />
<br />
<br />Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com2tag:blogger.com,1999:blog-3245099682640174652.post-2841222031793876672012-03-26T13:43:00.000-07:002012-03-26T14:09:57.800-07:00Cloudstack instance authentication using ssh keys<h2>
<span style="font-size: large;">SSH Keys... </span></h2>
Background: I am currently building a private IaaS cloud using the open source cloudstack software. <br />
<br />
Did you know that cloudstack comes with the ability to offer cloud clients the ability to authenticate into their instance's using ssh keys? I didn't.<br />
<br />
So, I'm reading the cloudstack api documentation and ... Before you say anything, yes, I was actually reading an api doc. So, I see an api method entitled createSSHKeyPair and I wondered if it did what I think it did. It did.<br />
<br />
Unfortunately, I couldn't find any documentation on this, so I thought I'd help the community out and share what I learned. <br />
<br />
I should also mention that Cloudstack also supports another form of user authentication. That is random root/admin password generation. A user is given a random password for use on their instance. I've tested this an it seems to work fine. This works on linux, unix, and windows. This is not covered in this article however.<br />
<br />
<h3>
<span style="font-size: large;">A quick primer on SSH keys and why this would be useful:</span></h3>
<br />
If you know all about ssh keypairs, then please feel free to skip to the next section.<br />
<br />
If you don't know, an ssh key pair can be used to login into a system with a 'ssh key file' versus using a known password. Please feel free to reference other material to learn about ssh keys.<br />
<br />
Using ssh keys and infrastructure as a service (iaas) clouds is useful for a couple reasons. First, every cloud user would be using their own ssh key. This, in itself, offers some additional security to your cloud solution. This ensures that one cloud user can not access another cloud user's instances (unless they shared their ssh key files). Secondly, when people choose to use clouds, they often are doing it because they know the value of programmatically being able to access and modifuy thier infrastructure (instances). And, one way to do that is to use ssh keys. This would allow to easily manage many instances with a simple, secure method.<br />
<br />
By the way, Amazon's EC2 uses SSH KeyPairs pretty heavily. <br />
<br />
<br />
<h3>
<span style="font-size: large;">Now, what work is needed to make this happen on a cloudstack cloud?</span></h3>
<br />
First, some restrictions, notes and disclaimers. This should work on most unix/linux variants. I don't think this works with Windows atm (since I don't think is has a built in SSH server =]). I've personally tested it on CentOs linux. Also, SSH Key management is not available via the cloudstack GUI (currently). That means that the only way to create instances that use ssh keys is via the API (for now). My work was done on cloudstack 3.0.0. However, I think it also works on older versions as well. <br />
<br />
The only thing a cloudstack administrator needs to do is: <br />
<br />
<ol>
<li>Create a instance template that supports SSH Keys</li>
</ol>
<br />
The process for the cloud user would be:<br />
<br />
<ol>
<li>Create an SSH Key in cloudstack</li>
<li>Create an instance using the SSHKey enabled temaplate and specifying their personal ssh key</li>
<li>Login to the instance using their ssh key (not their password)</li>
</ol>
<br />
<h3>
<span style="font-size: large;"> Create a instance template that supports SSH Keys</span></h3>
<br />
<br />
The folks @ Cloudstack published a script that enables SSH Key support on a linux instance. What does this script do? Well, the script will run on instance startup. It will communicate to the instance's virtual router and ask cloudstack for the SSH key private key. The script will then download the ssh key and install it on the instance for the cloud user to authenticate against. What we, as the cloud admin, will need to do is install this script on an instance and save the instance as a template. This way, cloud users will be able to use this feature.<br />
<br />
Here is what I did on a CentOs instance in order to get this to work:<br />
<br />
<ul>
<li>Create a new instance using the CentOs template provided by cloudstack.</li>
<li>Download the cloudstack script from here to the centos instance: http://sourceforge.net/projects/cloudstack/files/SSH%20Key%20Gen%20Script/ </li>
<li>Copy the file to /etc/init.d and have it run on startup.</li>
<li>Stop the instance (via the cloudstack GUI)</li>
<li>Create a template (via the cloudstack GUI)</li>
</ul>
<br />
Here are the commands I ran on the CentOs host before I created a template. <br />
<br />
# download the script <br />
wget http://downloads.sourceforge.net/project/cloudstack/SSH%20Key%20Gen%20Script/cloud-set-guest-sshkey.in?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcloudstack%2Ffiles%2FSSH%2520Key%2520Gen%2520Script%2F&ts=1331225219&use_mirror=iweb<br />
<br />
# copy the script to the proper place <br />
cp cloud-set-guest-sshkey.in /etc/init.d/<br />
<br />
# set permissions on the script <br />
chmod +x /etc/init.d/cloud-set-guest-sshkey.in<br />
<br />
# tell the OS to start the script on start up <br />
chkconfig --add cloud-set-guest-sshkey.in <br />
<br />
<br />
<ul>
</ul>
<h3>
<span style="font-size: large;">The Cloud User work flow</span></h3>
<br />
<ul>
</ul>
<h4>
1. Create an ssh key:</h4>
<br />
We will need to make a call to the createSSHKeyPair api method. I personally use the cloudstack python api library that jason hancock published. But, for a better example of how to do this via the api, I'll show example curl commands to the cloudstack api.<br />
<br />
Note: I am making the below call from my cloudstack server itself. If you are making the api call from a different server, your URL/PORT will be different and you will need to use api keys.<br />
<br />
Here, I am creating a ssh keypair called "keypair-test1" for the admin account in the root domain. (Please adjust these values to meet yoru needs)<br />
<br />
My curl command:<br />
<br />
curl --globoff "http://localhost:8096/?command=createSSHKeyPair&name=keypair-test1&account=admin&domainid=5163440e-c44b-42b5-9109-ad75cae8e8a2"<br />
<br />
The output will look something like this:<br />
<br />
<?xml version="1.0" encoding="ISO-8859-1"?><createsshkeypairresponse cloud-stack-version="3.0.0.20120228045507"><keypair><name>keypair-test1</name><fingerprint>f6:77:39:d5:5e:77:02:22:6a:d8:7f:ce:ab:cd:b3:56</fingerprint><privatekey>-----BEGIN RSA PRIVATE KEY-----<br />
MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci<br />
dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB<br />
AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu<br />
mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy<br />
QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7<br />
VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK<br />
lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm<br />
nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14<br />
4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd<br />
KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3<br />
-----END RSA PRIVATE KEY-----<br />
</privatekey></keypair></createsshkeypairresponse><br />
<br />
What you will want to do, is copy the key data into a file so that file looks like this:<br />
<br />
-----BEGIN RSA PRIVATE KEY-----<br />
MIICXQIBAAKBgQCSydmnQ67jP6lNoXdX3noZjQdrMAWNQZ7y5SrEu4wDxplvhYci<br />
dXYBeZVwakDVsU2MLGl/K+wefwefwefwefwefJyKJaogMKn7BperPD6n1wIDAQAB<br />
AoGAdXaJ7uyZKeRDoy6wA0UmF0kSPbMZCR+UTIHNkS/E0/4U+6lhMokmFSHtu<br />
mfDZ1kGGDYhMsdytjDBztljawfawfeawefawfawfawQQDCjEsoRdgkduTy<br />
QpbSGDIa11Jsc+XNDx2fgRinDsxXI/zJYXTKRhSl/LIPHBw/brW8vzxhOlSOrwm7<br />
VvemkkgpAkEAwSeEw394LYZiEVv395ar9MLRVTVLwpo54jC4tsOxQCBlloocK<br />
lYaocpk0yBqqOUSBawfIiDCuLXSdvBo1Xz5ICTM19vgvEp/+kMuECQBzm<br />
nVo8b2Gvyagqt/KEQo8wzH2THghZ1qQ1QRhIeJG2aissEacF6bGB2oZ7Igim5L14<br />
4KR7OeEToyCLC2k+02UCQQCrniSnWKtDVoVqeK/zbB32JhW3Wullv5p5zUEcd<br />
KfEEuzcCUIxtJYTahJ1pvlFkQ8anpuxjSEDp8x/18bq3<br />
-----END RSA PRIVATE KEY-----<br />
<br />
Save the file. I called my file keypair-test1<br />
<br />
<br />
<br />
<h4>
2. Create an instance</h4>
<br />
Now, we need to create a new instance in cloudstack. We have to use the template we created above. We also will have to use the ssh key name from above. Note: you can not create the instance via the gui at this time and associate the instance with the newly created ssh keypair. A sample curl command to create a new instance would be:<br />
<br />
Note: Please substitute the template, service offering and security group IDs (you may not need to specify a security group if you are not using that feature) that are in your cloudstack environment:<br />
<br />
curl --globoff http://localhost:8096/?command=deployVirtualMachine\&zoneId=1\&serviceOfferingId=18727021-7556-4110-9322-d625b52e0813\&templateId=e899c18a-ce13-4bbf-98a9-625c5026e0b5\&securitygroupids=ff03f02f-9e3b-48f8-834d-91b822da40c5\&account=admin\&domainid=1\&keypair=keypair-test1<br />
<br />
<br />
<br />
<h4>
3. Login using the ssh key</h4>
<br />
Assuming everything above went well, you should be able to login into you new instance without using a password, but instead using your keypair:<br />
<br />
From a linux cli, I ran this:<br />
<br />
ssh -i ~/.ssh/keypair-test1 192.168.1.100<br />
<br />
The -i parameter tells the ssh client to use a ssh key found at ~/.ssh/keypair-test1.<br />
<br />
Notice it just works and doesn't ask for a password: <br />
<br />
[root@cloudstack1]# ssh -i ~/.ssh/keypair-test1 192.168.1.100<br />
Last login: Mon Mar 26 19:58:30 2012 from 1.2.3.4<br />
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# hostname<br />
faa9ccca-30fe-4e4b-9645-d878514f0966<br />
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# cat /etc/redhat-release<br />
CentOS release 5.6 (Final)<br />
[root@faa9ccca-30fe-4e4b-9645-d878514f0966 ~]# uname -a<br />
Linux faa9ccca-30fe-4e4b-9645-d878514f0966 2.6.18-238.el5xen #1 SMP Thu Jan 13 16:41:45 EST 2011 x86_64 x86_64 x86_64 GNU/Linux<br />
<br />
<br />
<br />
<br />
Good luck and happy hunting.<br />
Chris<br />
<br />
<br />
<br />
<br />Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com0tag:blogger.com,1999:blog-3245099682640174652.post-78100003169482016832012-03-26T09:17:00.003-07:002012-03-26T13:55:48.149-07:00Welcome to my new blog. I'll be writing about internet infrastructure, data center networks, infrastructure automation, and clouds. <br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-4nc1g0-XT9I/TiYbh706J-I/AAAAAAAAC2Q/C7RlmnK0Gug/s1600/our+wedding+080.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="319" src="http://4.bp.blogspot.com/-4nc1g0-XT9I/TiYbh706J-I/AAAAAAAAC2Q/C7RlmnK0Gug/s320/our+wedding+080.jpg" width="320" /></a></div>
<br />
<div style="text-align: justify;">
<br /></div>Chris Klebanhttp://www.blogger.com/profile/09079353717313079698noreply@blogger.com3