Cloud Services Redone - Part 1 of 2

Disruption in Open Source Delivery, the Open Source Service 

What if we took the positives of open sourced software and merged them with the positives of cloud services like DynamoDB, BigTable, Kinesis, Lambda and S3? We would have a disruptive service delivery model.

In a two part post, I'll talk about two new approaches to delivering software as a service so that engineers can build better applications. One model I will call Open Sourced Services (Part 1) and another model I will call Managed Anywhere Services (to-be-written Part 2).

These posts will mostly describe a vision I have but will also include some back story and some technical information. The attended audience is anyone building or using applications. 

Where are we today?

Today's Cloud Services 

Today's application level cloud services like AWS's Kinesis or Google's Big Table provide engineers fully managed services in which you can build an application on top of that require no effort to maintain. These services are secure, autoscaled, monitored, logged, integrated and updated. Engineers can focus on their particular applications versus worrying about things like mysql cluster management or file replication. Personally, I love these services. However, these services lock you into a single cloud vender and you loose control of the functionality of these services. What if we could get the benefit of services without the lock in?

Today's Open Source 

Open source software has many benefits. The source code is open, which gives you increased control. The software is free which helps drive adoption and potentially lowers cost. It gives back to the community by enabling the free re-use of software. And it frees you from cloud lock-in as you can easily more your solution anywhere. However, open source software is normally delivered in a software package in which someone needs to install, monitor, configure, scale and upgrade so that it can be used in production. Each company often does all this a different way and their work isn't reusable by others. Isn't that a shame? What if we could take the increased control, the re-use of code, freedom from cloud lock-in and the free price tag of open source software but deliver it as a service?

Well, what do we want? 

The three main items that stick out are:

• services
• control 
• portability (no lock in)

Below, I will talk about one way how we can get there. 

Model #1 - Open Sourced Services

An open source service is the packaging and delivery of open source software such that it is delivered as a service. An open source service should have these characteristics:

It is consumed as a service, not as software - It provides a function to the consumer in a clearly consumable method. Ie, a NoSQL service provides a way to insert data and retrieve data. The services heals itself. It scales when required. It provides metrics, logs and alerts to the consumer. It is highly available. The service is installed as easy as running a command or clicking on a button.

It is controlled as a service, not as software - The consumer takes action on the service itself versus it's subcomponents. They can initiate upgrades of the service. They can start, stop or restart the service as needed. The service should expose configuration knobs so that the consumer can control the level of redundancy, user access, scaling parameters (auto scaling, infrastructure thresholds, etc), capacity and user access.

It can run anywhere - The service should be able to be deployed anywhere. In a public cloud or in your company's data center.

It is open - The source code of the software and the declarative definition of how the software is wrapped as a service should both be open sourced. This will give the the consumer of the service the control they desire.

How do we get to Open Sourced Services? 

The creators of open source software have not had an easy and unified way to package up their software in a way that meets the above criteria until recently. Thanks to the disruptive technology of containers, such as docker, and orchestration systems (such as kubernetes, swarm and mesosphere), I believe we now have a platform in which we can build our open sourced services on top of.

Note: I'll be using Docker and Kubernetes in the rest of this article but you should know that there are alternatives (swarm, mesospahre, rocket, oci, etc). 

Docker enables us to easily build, ship and run software by packaging it up in a way that will run on a wide range of systems. If open source groups takes their code and packages it inside containers they will have the building blocks required for turning their software into a service (that can run anywhere).

Once the software is packaged in Docker containers we can turn their software into services by using kubernetes's features. Kubernetes's goal is to take a set of infrastructure (that lives anywhere) and provide an API to create services on top of that infrastructure. Kubernetes includes features like load balancing, scaling, rolling-updates, service discovery, name spaces, APIs, scheduling and configurable infrastructure threshold limits. Using these features, we can create declarative services and give control of the service to the consumer. Kubernetes helps groups deliver on the above characteristics for what makes up an open source service.

Creators of open source software now have a way to package their software up and deliver it as a service.  Sweet!!!! So, how would we consume these new services?

How will we consume open sourced services?

First, consumers will need to have one or more kubernetes clusters. Getting kubernetes up and running can take as little as 5 minutes or as long as a few days depending on your situation. Kubernetes supports most public clouds, openstack clouds and on-premise data centers. Consumers can also choose whether to run on top of VMs or physical machines. We can run kubernetes ourselves or we can use a managed option if we are looking for increased support. Notice that the consumer is in complete control here of where their applications will run as they will be freed from any lock-in. Thanks to the common API that lives in any infrastructure environment, consumers can use these clusters to deliver on a multiple cloud provider strategy or they can create a true hybrid strategy.

Second, consumers would need to choose and install some open source services . For example, let's say they want to use a noSQL service for their application. Previously, their choices where using something like AWS's DynamoDB (with AWS lock-in) or they could have engineered their our own Cassandra cluster with a bunch of blood and sweat. With open source services, all consumers will need to do is install the cassandra service on their kubernetes cluster of choice. They might do this with an easy command line tool ('kubernetes install service cassandra') or perhaps by logging into an app store and clicking on the cassandra service. Consumers will want to control the service by setting some parameters, like capacity limits and redundancy options, so that the service meets their needs. After that, the consumer simply starts using the service and keeps an eye on all the fun metrics and logs the service provides.

In this new world, consumers will have more control of their application and the services that they depend on. They can move our applications between clouds or between private and public. Consumers can run their applications actively across multiple clusters or do active-standby. All this is sweetness.

In summary

With open source services there are lots of winners: Developers will be able to build their applications the way they want to while getting the control they need. Tech execs get freedom from cloud lock-in and have a path to a true hybrid solution. Public cloud providers get to continue to sell infrastructure. Open source projects get a way to easily package their software as a service which increases their user's perceived value. Entrepreneur's get a brand new gold rush as their are plenty of challenges and opportunities.

Up next - Marriage #2 - Managed Anywhere Services

One of the down sides to open sourced services is that sometimes there is no one to call in case things go wrong. In my next post I'll write about an alternative approach that aims at solving that problem and opens up a new way for companies to make $$$ by delivering these services.

End

Thanks
Chris

--

Some useful links:

• Learn more about containers and kubernetes: http://kubernetes.io/v1.0/docs/whatisk8s.html
• A list of some early versions of open source software packaged up as services: https://github.com/kubernetes/kubernetes/tree/master/examples

I wrote a blog post about CenturyLink Cloud's usage of Kafka for CenturyLink Cloud's website. That post, in it's full content, can be found below. Thanks. Chris

Real Time Data Distribution with Apache Kafka

Like most enterprises and service providers, here at CenturyLink Cloud we aim to please our customers by making data based decisions. As we grew our business, so does the amount of data we collect. The effort required to distill, distribute and analyze the data in meaningful ways was increasingly strenuous. In short, we needed to become faster with our analytics. If your organization is becoming overwhelmed with data, read on, as I’ll share with you how we used Apache Kafka to solve the following challenges with collecting and distributing data at scale.

·      Efficient access to data

·      Speed

·      Scalability

·      Distributed, Fault Tolerant and Highly Available

Challenge #1: Efficient access to data

Getting access to data is sometimes the hardest part. Sometimes the data that you need lives in some isolated software service and a one off data export is required. Perhaps your data integration flow is overly complex due to your adoption of a service-oriented-architecture. Maybe you had a process that required multiple teams to due work before data moved around. We saw all of this and more. We wanted to make it easier to move data around which would increase the overall velocity of our organization. Our data distribution flow between services looked something like this:

Previous Data Distribution Model

To fix this, we centralized our data flow with Apache Kafka. Kafka is commonly referred to as a pub/sub (publish and subscribe) messaging system. Simply put, systems publish their data into a Kafka ‘topic’. Other systems subscribe to the desired ‘topic’ within Kafka and extract the data. Kafka acts as a messenger, moving data to where it needs to go. Kafka, or any pub/sub messaging system, decouples the publishing of data from the subscription of data. Publishers only need to worry about sending the message. Subscribers only need to worry about receiving the message. It’s important to know that Kafka is not intended to be used a permanent data store. Rather, it provides a reliable, fast and scalable method to distribute data from data publishers to subscribers. Data storage is left to the consumers. By using Kafka we had an asynchronous, decoupled, centralized communication mechanism that looked like this:

Data Distribution Model with Kafka

We moved away from the many-to-many data distribution model to a centralized model. All of our services only need to send their data once, regardless to the number of destinations for the data. This also means that our services only need to pull from one source versus worrying about implementing a variety of different data integrations technologies. This centralization design reduced the overall effort spent by our engineers distributing data.

Note: There are alternative messaging software systems out there. RabbitMQ is similar in capabilities and may suite your needs better depending on what you need.

Challenge #2: The Need for Speed

Time is money. The quicker we can collect, detect, process, predict, and take action on our data, the faster we can act on behalf of our customers. We needed a data distribution model that enabled both near real time (streaming) analytics as well as the more traditional batch analytics. Enter the Lambda Architecture framework. The Lambda framework calls for all new data to be distributed simultaneously to both stream and batch processing pipelines, as such:

Lambda’s view of data distribution

Source: http://en.wikipedia.org/wiki/Lambda_architecture#/media/File:Diagram_of_Lambda_Architecture_(generic).png

Using the Lambda framework and Kafka for the messaging queue, we gained the ability to seamlessly add streaming analytics when needed (we use Apache Spark’s Streaming Module). In addition, it helps that Kafka itself is fast. Kafka’s data structure is essentially log files. As data comes into Kafka, Kafka appends the data to a log file. If subscribers are pulling data out in real-time they are also reading from the end of the file. This allows Linux’s page caching to store the needed data in memory for reads while using disk drives for writes. Lambda, Spark and Kafka’s low latency messaging allows us to make near real time decisions for our customers.

Note: An alternative framework to Lambda that is also worth looking into, as described in this article, suggests that users can feed data into a stream processing pipeline which in turn feeds the data into the batch pipeline.

Challenge #3: Scalability

The amount of data we collect and process is large and growing. We needed a data distribution model that will scale horizontally, use cloud infrastructure, support large workloads and auto-scale without impacting the data distribution flow. Kafka was built in order to scale and is has proven to work well under heavy load, as described in this LinkedIn post. Kafka’s ability to scale is achieved through it’s clustering technology. It relies on the concept of data partitioning in order to distribute the load across members in a cluster. If you need more data throughput in a particular location, you simply need to add more nodes to the Kafka cluster. Many cloud providers, like ours, and infrastructure automation frameworks provide the mechanisms to automate the creation and scaling of Kafka clusters.

Challenge #4: Distributed, Fault Tolerant and Highly Available

Losing data is bad. Having data stop flowing is bad. Having data only available in one datacenter is bad. We needed a solution that wasn’t bad and Kafka had us covered. Kafka supports data replication inside a cluster. This means that even if a server in a Kafka cluster crashes, data in the Kafka message bus would not be lost. Great. In addition to replication, Kafka’s uses sequential IDs to enable the graceful restarting of a data flow. If a client stops for any reason, another client can use the sequential IDs to pick up where the other client left off. We process the data in aggregate or to have high availability on the aggregate data, there is a need to replicate data to multiple data centers. Kafka provides an easy mechanism to have a Kafka clusters pull data from other Kafka clusters in other data centers. We use this to pull data from all our data centers to a single data center in order to preform analytics in aggregate. Kafka is reliable, distributed and fault tolerate. It just works.

Note: The delta in sequential IDs between the publisher and subscriber is referred to as the offset. The offset often used as a KPI metric on data flow through Kafka.

What’s wrong with Kafka?

So what’s wrong with Kafka? Not much. It solves our challenges and the price is right, free. The one drawback with Kafka that I’d like to call out is the complexity is puts on it’s clients. For example, each client needs to connect to every Kafka server in a particular cluster versus making only a single connection to the cluster. It needs to do this because the data being put into a Kafka cluster is split across members of the cluster. Thankfully there are SDKs, APIs and tools available that help engineers over come these challenges if a little time is spent researching and testing these items. In the end, we gladly accepted burden on the client in exchange for Kafka’s reliability, performance and scalability.

Summary

Kafka is the heart of our data stack. It pumps our data around so we can quickly and accurately act on behalf of our customers. Before Kafka, our data was moving around all willy-nilly. Post Kafka, we have simplified, scaled and structured dataflow that enables our business to move faster than was previously possible. Today, we are publishing operational metrics, logs, usage and performance information into Kafka and we are planning on importing additional types of data. We have a variety of software systems consuming the data from Kafka: Apache Spark for real-time and batch analytics, Cassandra for data storage, a SQL based data warehouse for reporting purposes, custom written applications and ELK (Elasticsearch, logstash and kibana) for operational visualization and search. Kafka is pumping here at Centurylink Cloud.

Minecraft on Kubernetes

This is a post about launching a minecraft server, inside docker, on a kubernetes cluster.

Why do this?

• I have a 5 year old son 
• we like to play games and build stuff with blocks
• I'm interested in kubernetes
• I saw this post on minecraft in docker 
• I wanted to get to know minecraft and kubernetes better

Note: After I got this working, but before I posted this, I came across Julia Ferraioli's series of blog posts about minecraft, docker and kubernetes. These posts are good and you may want to check them out too.

Docker

One of the advantages of Docker is sharing. In the public docker hub, there were a few different minecraft builds available. I choose to reuse Geoff Bourne's (@itzg) dockerfile listed here. Its a great build that lets you specificy many server configuration options via environment variables so you don't have to worry about putting together a minecraft configuration file. With my container already created for me, getting minecraft up in kubernetes was easy.

Game on Kubernetes

First, you will need kubernetes cluster. Doesn't matter where your cluster is as the bueaty of kubernetes is this will work regardless of where your cluster is. Since I work at Centurylink Cloud, I used the scripts I created here to create my kubernetes cluster on CL cloud.


Kubernetes has the concept of pods. A pod in k8 is a logical grouping of one or more containers, zero or more storage volumes and an IP address. We will launch the above minecraft docker container in a pod in kubernetes. And, since minecraft isn't a clustered server, we only need to run one pod per minecraft server.


Here is the yaml file I used to create this pod.
(Also located here: https://github.com/ckleban/minecraft-k8)


apiVersion: v1
kind: Pod
metadata:
  name: minecraft-server
  labels:
    app: minecraft-server
spec:
  containers:
  - name: minecraft-server
    image: itzg/minecraft-server
    env:
    - name: OPS
      value: madsurfer79
    - name: EULA
      value: "TRUE"
    ports:
    - containerPort: 25565
   

You will want to at least change the username in the OPS ENV variable to be your minecraft username. This let's the server know who has admin privileges on the server. You can add more settings, like game type, minecraft version, etc as you wish. To see a more detailed list of what options are available, go check out here. Now, go create the minecraft server:

-------------------------------------


#Create pod

> kubectl create -f pod.yml

#See it up and running:

> kubectl get pods

NAME                      READY     STATUS    RESTARTS   AGE

minecraft-dfhse           1/1       Running   0          2h

#If you want this server to be reachable on the internet, you probably need to do some more things. If you are on a public cloud, like Centurylink, amazon or google, this is easy to do. Simply run:

> kubectl expose pod minecraft-server --port=25565 --type=LoadBalancer

You will be given a public IP address, which you can see by running:

> kubectl get services

-------------------------------------

That is it. You and your friends can now play minecraft, inside docker, on kubernetes. 

A few things to note:

• you may need to wait a few minutes while you wait for the external IP address to be published. 
• you may also need to open up a firewall rule to this IP address and port. 
• this isn't a durable configuration. If the pod dies, it won't come back online and the save games and world will be gone for ever. These problems are easily solved in kubernetes. We just need to add a replication controller and persistent storage. Perhaps, I'll work on that next and post it sometime. 

Thanks

Chris