Breathing Data

Before starting at Joyent, I followed them and the evolution of their platform for years.  How they tackled scalability and instrumentation has always appealed to me.  I’ve realized the platform covers a lot of use cases and is extremely compelling.  I’ve talked previously about new ways to solve legacy enterprise infrastructure issues but there other ways of handling the challenges of newer more distributed workloads.

Here’s how Jason Hoffman (friend and Founder/Chief Scientist at Joyent) puts it:

The Physical Machine business is a “machine” business. The Virtual Machine business is still a “machine” business. The virtual machine is not the abstraction that is equivalent to a packet in the network. Fundamentally, being tied to a machine matters to a person, it doesn’t matter to an application and in fact causes there to be hits in performance, scale and it’s all a black box (can’t comprehend everything).

We come from networking and ask ourselves, “How can we extend the days when the cloud we drew was just the network and enable the same abstractions, economics and models into servers? Into the rest of the datacenter? So that we may have the same type of multitenancy for compute that we have for networking.”

What is Joyent’s stack?  It’s a platform first and foremost.  What’s part of the platform?

This image from a slide deck lays it out.

A SmartOS kernel – It’s based on unix and all the unimportant fat has been cut from it.  It’s lean, scalable and provides QoS *and* introspection all throughout the stack.  Yes, we’re a software development company and we have kick ass kernel hackers that are brilliant.  We also have some unbelievable UI/UX folks on board who know how to make stuff *usable*.  Bryan Cantrill of Dtrace fame and formally of Sun/Oracle leads engineering.

A platform to build on – What do you want to run?  Do you want to run a Windows/Linux/WhateverVM?  Go ahead.  We can’t provide you deep introspection but into the application but we can tell you what your VM is doing.  Use SmartMachines (container-based single kernel “virtualization) and you get introspection at ALL levels.  Why would you use the native container-based SmartMachines?  Because, what you care about is applications, not what OS is running underneath.  You want to run Apache, Nginx with CouchDB, MongoDB, MySQL, Oracle or Riak go for it.  Oh and by the way, when your app blows up because you rushed the code, we have the diagnostics and visuals that can meaningfully tell you why your app is caving.  Can you run Java on us?  You can. It’s that simple.  You can build out templates and deploy them easily. This is cloud.  It’s ITaaS. Pull the trigger and it’s yours.

Now… for the really cool stuff. Ryan Dahl at Joyent developed something (open-sourced) called node.js. What is it?  It’s server-side javascript which is a 4gl language.  It’s essentially PaaS.  You use git to push your code and it’s up and running in the cloud.  Why did Joyent do that?Because at the end of the day when you need insane bandwidth and high transaction capabilities, you’re willing to sacrifice legacy capabilities to get the performance you need.  Not only that but node.js is tiny, lightweight and can run on the server or something as small as a cell phone.

Let’s boil it down.  Is Joyent SmartDataCenter a replacement for VMware?  Yes and no.  At service providers like Verizon, Terremark and many others, it can be a compliment to an existing IaaS offering. The main target isn’t legacy enterprise use cases that need HA at the IaaS level.  But it’s a very good fit for workloads that need a lot of elasticity and dynamic scalability.  Instead of constraining resources, a user can use as much of the box that is free while still preserving QoS to the other instances.

Has anyone used this stuff? Yes, Joyent has run its own cloud for 6 years and we figured out that having other SPs deploy Joyent as a cloud solution is something we’re willing to tackle.  It’s what LinkedIn, Gilt group, Disney, Facebook developers and many others have been using for awhile now.

If you’re more curious about Joyent’s approach, you can read more here.

I had the opportunity to deliver a keynote on “Cloud Computing and the Public Sector” late last week to 225 GIS (Geographic Information Systems) professionals in Frankfort, Kentucky.  Trisha Brush was kind enough to invite me after we met at CloudCamp Cincinnati.  I tried to keep it at a high level but then drill down to some particular use cases.  I also tried to deal with some of the sticky issues like governance and compliance.

The two points I emphasized during the keynote are:

1. The Cloud is an enabler.  It lowers the cost of entry for disruptive technology.  A use case example is using Elastic Map Reduce to solve huge problems we can’t easily or inexpensively solve today with traditional enterprise methods.
2. GIS and mapping are extremely relevant these days. As admins, directors and users, they should work with developers  to create innovative mashup-style apps.  This can help disseminate information to wide masses or create revenue streams from a constituent and commercial perspective.

You can find the keynote preso here.

Trisha borrowed a lot of the CloudCamp unconference format for the KAMP Summit which worked well.  I also found out that they had an App Contest (Desktop and Web) similar to a sprint during StartupWeekend.  I spent most of the day there and this is what I walked away with:

• Cloud in the public sector is still very new, yes it’s a journey
• Unconference (a la Barcamp) continues to work well as a format
• The midwest has a lot of innovation in it
• Can’t underestimate the value of leadership and community (Trisha did an amazing job)

I had to leave during the Unpanel but I was getting a lot of questions about Cloud in general (security, resiliency, etc) and I tried to answer as much as I could.  I also met Angie Jennings of Swova who specializes in ArcGIS implementations.  Her company is also one of the first to help customers take their GIS solutions into the cloud from an IaaS perspective.

If you’ve got questions about the preso or cloud in general, feel free to comment below.

via Brenda Clarke on Flickr

Everyone talks about the consumerization of IT and how end-users are demanding enterprise support of things like iPhones, iPads and many other pieces of technology.  People want to be able to consume IT as a resource on any device or platform they have.  This is happening between enterprise hardware and software vendors and service providers.

I’ve met with quite a few them recently and they usually fall into two camps.  Ones who have invested and attempted to develop their own intellectual property and others who have leveraged economies of scale and rely on vendors to supply the IP.  There are exceptions to this rule of course.  The first camp is what I want to focus on.

Here is what they want:

• Align with my business and go-to-market strategy
• Don’t can your offering with SP-focused marketing materials if it can’t honor the promises
• Have hardware and software with open interfaces
• Have well-documented interfaces
• Be agile in the adoption of new interfaces

These conversations tend to revolve around the self-service portals many of the pure-play service providers have developed.  They don’t want a canned out of the box offering, they want to be able to provision and orchestrate the compute, network and storage layer through things like SOAP and REST protocols.  When you develop these interfaces and hand them over to the developers, strange things happen.  Nick “@lynxbat” Weaver exemplifies this.  He isn’t a developer by day but you give him some APIs and he can do crazy things on a plane like write a vSphere plugin that allows VM teleportation with our (EMC’s) VPLEX product.

Now I’m not ignoring the need for software development lifecycle management, version control.  Those are all important.  The thing is that the “neat stuff” us and our customers do can only get better if we open up with good APIs that have a happy balance between standardization and cutting-edge agility.

Why do I beat this drum?  Because it’s a win for enterprise hardware/software vendors and our customers.  What is most exciting about this is that I’ve beat this drum inside of EMC, not as a VP of strategic direction but as a joe-schmoe vSpecialist.  What has come out of it?  A lot of people have listened and it is a huge priority.  I’ve said it before on twitter but one of the best things about working at EMC is that the organization is huge but very flat.  The reality is that I’ve been able to nudge an aircraft carrier with the help of others and start to change course.  This isn’t a post about why I love working at EMC but I think it’s a darn compelling reason.  Our work has just begun…

via mafleen on flickr

“I hate everything about you, why do I love you?” — I hate everything about you by Three Days Grace.  A couple months ago I asked folks on Twitter if IT governance was helpful or a hinderance.  You’d be surprised by the reactions, at least I was.  Back in the day when I was a server hugger in my former life, I hated any form of governance that was going to slow me down. Ok, I need to level with you, I still feel that way.  The operative phrase here is “slow me down”.  There is something have come to respect over the past 5 years when I stopped focusing on what brand of processor or type of server to buy.  I started talking to folks in the business who had loftier goals than I did which was moving the ball forward.  I started to *get it*.  It’s about moving the collective agenda forward.

I realized that you can embrace governance when you have all the key stakeholders involved and this means that you should be able to set up a framework and template for classes of IT offerings.  Why does it mean so much to me?  Because I’ve never seen this offered in a package that appeals to the CxOs and server huggers at the same time.  A vblock (EMC storage, Cisco network and compute, and VMware) represent this union.  Coupled with Unified Infrastructure Manager and some open api’s, we can start to realize a service catalog built around governance that provides the business a lot of agility.  A huge benefit, albeit an uncomfortable one, is removing NRE or non-recoverable engineering from the technology cycle.  I know it’s not perfect but now we get to focus on the important stuff.

There are some really cool sessions around Vblock and VCE at VMworld.  You can find more info here.

CloudCamp Cincinnati is here.  It’s this Thursday, June 3rd, in Cincinnati from 4pm-10pm.  Registration and session info can be found here.  It’s being held at the MET Center which is supposed to be a pretty nice venue.

EMC is sponsoring and I’ll be doing a lightning talk on private cloud.  If you haven’t been to a CloudCamp before, it follows the unconference format.  CloudCamps have been extremely successful because of the user participation.

Here are some of the topics that have been covered:

• Infrastructure as a service (Amazon EC2, GoGrid, Rackspace, Nirvanix, etc)
• Platform as a service (AppEngine, Azure, etc)
• Software as a service (salesforce.com, Yahoo! Mail, etc.)
• Application / Data / Storage (development in the cloud)

Things change.  I’m not talking about my wife, job or car.  I’m talking about myself and infrastructure consisting of servers, storage and networking.  I don’t want underutilized resources that take time to manage and don’t let me get things done.

Let me explain.  I flew into Boston to give a couple of Executive Briefings on the Virtual Datacenter this week.  Some customers saw exactly where I was going and others probably thought I was insane.  I started at a high level and then went into the details but here’s the problem. When we talk about infrastructure becoming a pool of resources that you’re able to push and pull workloads into and out of, some people think it’s fairytale land.  It’s not.  I used to build my own computers but I don’t anymore.  I buy one that has the most, not all, of the best technology and is good enough.

DIY vs All-in-one - DIY image via ~keiby on flickr

What they see:

1. Complex infrastructure with many moving parts
2. Legacy applications, some virtualization adoption
3. Thick fog that’s hard to see through
4. They think they’ll have to trust something new

What I see:

1. A multi-step process that takes time
2. Virtualization allows us to focus less on the boxes, cables and spinning platters
3. Platforms (Springsource with Azure, Amazon’s AWS also qualifying) enabling quicker development
4. We can make existing infrastructure and software better

Infrastructure AND platforms are both part of the “Stack” and “Cloud” conversation.  It’s about businesses being able to let their most valuable asset (their people) work on deploying applications faster instead of provisioning servers.  Yes, the “server huggers” and “IT pros” as Brian Prince (Azure Evangelist at Microsoft) calls them, will still be needed to make sure we’re architecting and deploying apps properly on the infrastructure.

Change is ok.  Things aren’t perfect but continually improving.  Most of us drive cars instead of ride horses and we buy clothes instead of make our own.  People trust technology (network, shared storage, memory, etc) and build around its faults (RAID etc.).

Recently I was bluntly accused of spouting an EMC-centric view.  As if other technology companies didn’t see it this way too.  Guess what, it’s not just EMC, it’s many end-users and vendors both who share this view.  The view of and challenges posed to enterprise customers is much different from those faced by sometimes smaller and more public-facing web service companies.  I get it.  There are still security challenges, management challenges, and legacy application challenges but before so quickly dismiss stacks and cloud, open your mind a bit.

Here are some notes I took during the keynote at the 2009 Microsoft’s Professional Developer Conference I attended months before joining EMC.

Session: Bridging the private and public Cloud

• Move has been to get higher utilization
• It’s about the applications and working to get them into the cloud

Opportunity

• Build rich apps for the cloud while preserving app symmetry w/ the enterprise
• Frictionless deployment across the spectrum
• New breed of apps that span from on-premises to cloud

I was invited as an “influencer” to attend the 2009 Microsoft Professional Developer’s Conference by Brian Prince.  This is my first PDC and I’m absolutely stoked to be here.  The event is important because it is the official launch of Microsoft’s Azure cloud platform.

In case you don’t know what Azure is, here’s a description from Microsoft’s web site:

The Windows Azure platform offers an intuitive, reliable and powerful platform for the creation of web applications and services.

The Windows Azure platform is comprised of Windows Azure: an operating system as a service; SQL Azure: a fully relational database in the cloud; and .NET Services: consumable web-based services that provide both secure connectivity and federated access control for applications.

Currently in Community Technology Preview (CTP), the services are free to evaluate through January 2010. We will begin charging customers on February 1st, 2010.

I’m excited to be a part of such a big event for Microsoft. One thing that seems to be consistent is that this is not your dad’s Microsoft. Their cloud group started with twenty engineers and has ramped up over the last couple of years.

Like I’ve done in the past, here’s a “What I want out of PDC09″ list. If there’s something you would like out of PDC, feel free to leave a comment or e-mail me.

1. Get the latest updates on the Azure stack
2. Understand the limitations and where different services are best used
3. See how Azure addresses challenges like deployment, scaling, security and private cloud integration
4. Understand how Microsoft is making their software cloud-aware
5. Talk to more people using Azure and see examples of how they are using it

Much of it is cloud-focused because I spend a bit of my time running Indy Cloud Users and involved with CloudCamp. It has been awhile since I’ve spent time with developers so I’m sure I will be learning a lot. I’ve been impressed with the dynamic nature of the company and individuals like Brian who deliver the message about what Microsoft is doing.

Cloud Strategy - VMware’s cloud strategy is still maturing and growing.  We have been hearing from Maritz and others that technology is built into vSphere and other products to leverage it as a cloud platform.  I expect we’ll be hearing more about some tangible developments with cloud providers out there today. It will be interesting to see if VMware continues to build itself as a cloud platform or if it shifts gears and starts chasing after Amazon’s AWS and Microsoft’s Azure platforms.  Though they have invested in Teramark, without some good explanation, it would be detrimental for VMware to try to be the provider.  I suspect the folks at VMware know this and are have no desire to be the provider but instead need to seed the field.

Enhanced infrastructure awareness - VMware and its network and storage partners need to more visibility to each other.  Not only do people need to be able to see what is going on under the covers (storage and network) with things like AppSpeed but they also need to be able to make intelligent decisions on how to fix problems.  It should be easy for an admin to see what LUN on the storage side has too many VMs without having to interpret naa392dxxxxxxxxxxxxxxxx numbers.  This is continuing to happen but still has a ways to go.

Desktop Virtualization – The improvements from VDI (2.0) to View (3.1) and continuing to View 4.0 have been good but there is so much work to do.  When I meet with customers, the challenges that they face aren’t just getting applications and desktops virtualized from a technical perspective.  We need more flexibility to determine not only what desktop a user receives but what kind of desktop a physical location receives.  We need application persistence with a physical endpoint.  This is counterintuitive to what virtualizing desktops is all about but this is all going to drive back to the persona of both the person and the endpoint.  Entrigue Systems, which is being acquired by Liquidware Labs,  and other ISV’s are doing this but it needs to be seamless and well supported.

If you have anything you want to know or news to share with me about some of these things, let me know.

It’s obvious VMware and virtualization are playing a huge role in cloud computing from the perspective of Infrastructure as a Service (IaaS).  VMware’s lead Cloud Architect, Mike Dipetrillo, was gracious enough to provid some great insight into VMware’s strategy.

IaaS is where most managed service providers focusing on today for their cloud offerings.  We discussed that developing self-service infrastructure as a service provider offering is tricky.  The folks who have done infrastructure understand the glue that is needed for provisioning and allowing users control over their own environment.  Giving users the ability to turn the knobs that control things takes a lot of work.  Today that is VMware Lab Manager under the covers with some special glue for provisioning.  Lab Manager has some challenges today because it wasn’t designed for a multi-tenant environment.  Over the next year you’ll start to see products come out that address this for service providers.  VMware is also heavily focused on delivering more APIs which allow companies like RightScale to hook into VMware to provision and manage virtual machines.  VMware categorizes all of its cloud computing initiatives under the vCloud umbrella.  This will include all of their cloud-focused products and APIs.  The roadmap has developed rapidly over the last year or two.

Helping small, medium and large businesses build out their internal clouds has been a big focus as well.  It needs to be easy to allow people to have the flexibility to move things between the internal and external cloud.  One of the questions I get asked the most is “How can I move my applications out into the cloud?”  It is a lot cheaper and easier to virtualize your existing software stack compared to rewriting things to fit on exotic platform as a service software at the moment.

Another thing we discussed is how enterprise companies don’t really like “elastic” or “usage-based” billing models.  They actually prefer allocation-based where they billed in a consistent fashion.  I’ve never given a lot of thought to this but it makes sense.  A lot of companies do business the way they do because it works well for them.

Today VMware has over 500 service providers who are either in the process to getting a cloud offering off the ground or have one today.  The team at VMware is small when compared to the rest of the organization but things have been progressing fast.

We discussed competition briefly but both agreed how things are changing rapidly.  It was obvious that VMware is agressively ramping up its vCloud offering and the internal structure to go along with it.  The benefit they have as a company is that they’re able to leverage so many of their existing products and IP.

Many thanks to Mike for sparing some time to discuss VMware’s vCloud initiative.