Thursday, July 31, 2014

The Cloudcast #155 - Large Scale Distributed Infrastructure with Mesos

Today’s show is dedicated to the memory of a good friend of the show, Jim Ruddy, who left us last week in a tragic car accident. Jim loved his family, life and exploring all the new technology that we cover on the show. He’ll be deeply missed by everybody that knew him.

Topic 1 - We met at OSCON last week and you introduced me to Apache Mesos. Give us some of your background and what you do at Twitter with regard to Mesos?

Topic 2 - Let’s talk about the types of challenges that Mesos attempts to solve at Twitter and why so many webscale companies are using it. (Twitter, AirBnB, CloudPhysics, eBay, Netflix, PayPal, Vimeo, etc.)

Topic 3 - As I started digging into the Mesos architecture, it’s a fairly complex set of inter-related activities (Mesos Controller, Hadoop, ZooKeeper, Docker). Can you walk us through the basics of what area function does or some common use-cases?
  • Zookeeper - like a CMDB - keeping track of configs
  • Hadoop Scheduler - collecting data to feed the Mesos scheduler?
  • Mesos Master - making scheduling decisions (using hadoop data) and pulling configs from Zookeeper?

Topic 4 - Mesos appears to be primarily focused on compute, but does it have any logic to also interact with networking elements (network segments, load-balancers, firewalls, etc.)?

Topic 5 - Is Mesos a system that could live on top of something like OpenStack, or is it primarily focused on containers or bare-metal compute clusters?

Topic 5 - Some of our listeners have heard about this thing that Google released, called Kubernetes, which they use to build their Google Compute Engine. On some level it sounds like another resource scheduler, but we also heard about Mesos + Kubernetes interaction. Are these groups solving similar problems, or are they finding new ways to use tools/frameworks that might have been used in other ways before (eg. Hadoop)?

Topic 6 - There is an event happening in August (21st-22nd) called MesosCon, in Chicago (along with LinuxCon). What are the interesting things that will be happening those days?