Continuuity Loom 0.9.7: Extensible cluster management

Jun 10 2014, 11:12 am

Derek Wood is focused on DevOps at Continuuity where he is building tools to manage and operate the next generation of Big Data applications. Prior to Continuuity, Derek ran large scale distributed systems at Wells Fargo and at Yahoo!, where he was the senior engineering lead for the CORE content personalization platform.

In March, we open sourced Continuuity Loom, a system for templatizing and materializing complex multi-tiered application reference architectures in public or private clouds. It is designed bottom-up to support different facets of your organization - from developers, operations and system administrators to large service providers.

After our first release, we have heard a lot of great things about how people are using Continuuity Loom and have also heard about features that have been missing. After taking in all the feedback, we are excited to announce the next version of Continuuity Loom codenamed Vela. The theme for this release is “Extensibility” as we have been working towards making Continuuity Loom integrate with your standard workflows for provisioning and managing multi-tiered applications, as well as making it easier to extend Continuuity Loom itself to your needs.

Highlights of Loom 0.9.7

  • Ability to register plugins to support surfacing of plugin defined fields for configuring providers and automators

  • Support for finer grained dependencies

  • Cluster life-cycle callback hooks for integrating with external systems like metering, metrics, etc.

  • Extend your cluster capabilities by adding new services to an existing cluster or reconfiguring existing services

  • Smart support for starting, stopping and restarting services with dependencies automatically included during service operations

  • Personalizable UI skins

  • More out-of-box Chef cookbooks including Apache Hive™ recipes and recipes for supporting Kerberos enabled clusters

Detailed overview of Loom 0.9.7 features

Cluster Reconfiguration

How many times have you had to change some configuration setting of a service on a cluster and had to remember to restart the right services in the right order for the change to be effective? The reality is, when making changes in multi-tiered application clusters, there are a lot of things to remember. Wouldn’t it be simpler if you could change the configuration that you want and let the system figure out everything else for you, ensuring the change you just made is active without any hassle? With this release of Continuuity Loom, you don’t have to worry any more about what services need to be restarted. Continuuity Loom automatically figures out service stop and start order based your service dependencies. You can find more information about how this is done here.

Add missing and manage cluster services easily

Let’s take a concrete use-case: let’s say your administrator has configured a template that allows you to create a Hadoop Cluster (HDFS, YARN, HBase, Zookeeper) with Kerberos security enabled. As a new user, you would like to try building a basic cluster with just HDFS and YARN until you are ready to add more. In that case, Continuuity Loom provides an easy way to remove services that you don’t need during creation time and then subsequently add them back to the live cluster with just a few clicks. With this new release, users will now have the ability to stop, start, and restart services without having to worry about what additional or dependent services need to be restarted.

Plugin Registration

In line with our theme of extensibility, we wanted to ensure that developers are able to write custom Automator and Provider plugins. As part of this, plugins now define the fields they require, which get surfaced in the UI and passed to the plugin during task execution. Particularly for Provider plugins, this allows you to provide different options for provisioning clusters. For example, Rackspace requires a username and API key, while Joyent requires a username, key name, and private key file. It is now possible to write your own plugins and describe the specified fields required. With the addition of this feature, you can also write support at the API level for any container (like Docker), OS, or cloud provider. You can learn more about this feature here.

Finer Grained Dependencies

Prior to this release, all service dependencies were applied at all phases of cluster creation. This created unnecessary solution space exploration and execution of dependencies when they didn’t make sense. This release includes a feature called fine grained dependency management for services. This feature allows Continuuity Loom administrators to specify required or optional service dependencies that apply during runtime or install time (applied only when the service is installed and available on the machine). This is specified during service creation so users don’t have to worry about it. It provides granular control over the deployment of services and opens up support for HA Hadoop clusters and secure hadoop clusters, which require external Kerberos Key Distribution Centers (KDCs). You can learn more about this feature here.

Cluster life-cycle callback hooks

Often times, you need the ability to integrate Continuuity Loom with other workflows that exist in your organization. This feature allows the Continuuity Loom administrator to configure a callback class to insert your own custom logic before and after cluster operations. Out of the box, Continuuity Loom provides a HTTP callback implementation for the interface ‘ClusterCallback’. You can use this feature to integrate the cluster life-cycle with monitoring systems, metering systems, and even your favorite chat application to alert when clusters are created, deleted or upon any failed operations. You can learn more about this feature here.

Personalized UI skins

When you install Continuuity Loom on-premise wouldn’t you like the ability to change the color scheme and logo to make it fit well with the other tools in your organization? With this release you will have the ability to change the color, skin and logo of your Continuuity Loom installation.

This has been an exciting release for us. Check out the Release Notes for more details about this release. Give it a spin by downloading the standalone version for your laptop and visiting the quickstart guide to spin up clusters in cloud.

Help us make Continuuity Loom better by contributing and please report any issues, bugs, or ideas.

Coming soon - Continuuity Loom for free in the Cloud

Be sure to sign up at tryloom.io to be among the first to know when the free, cloud-based version of Continuuity Loom is available.

Comments

Introducing Continuuity Loom: Modern Cluster Management

Mar 20 2014, 11:28 am

Jonathan Gray, CEO & Co-founder of Continuuity, is an entrepreneur and software engineer with a background in open source and data. Prior to Continuuity, he was at Facebook working on projects like Facebook Messages. At startup Streamy, Jonathan was an early adopter of Hadoop and HBase committer.

Our flagship platform, Continuuity Reactor, utilizes several technologies within the Apache HadoopTM ecosystem, so as we started building it, our developers needed an easy way to test Reactor on real distributed systems. They needed a fast, self-service way to provision clusters, both on our own hardware and in public clouds, because testing against real clusters is such a critical and frequent component of our development cycle. Our vision was to enable all Continuuity developers to be able to quickly create a cluster with the push of a button.

We started off with simple scripts. But scripts had multiple issues - they required a lot of setup, there was no central management keeping track of available clusters, and the scripts had to be changed whenever we moved from one cloud to another, or one software deployment to another. Next, we started looking at different SCM (software configuration management) technologies and picked Chef. Combining scripts and Chef recipes, we’re able to get to one-button push, but it became extremely complex for two reasons - there were an increasing number of customizations needed for each new type of cluster and a significant amount of setup was needed on every developer’s box.

When we launched Reactor, we wanted to enable developers to deploy our single-node Sandbox Reactor to a public cloud directly from the website. This forced us to think harder about designing a complete production system while still providing an easy and flexible solution for our developers. We wrote a combination of scripts and Chef recipes, designed a system that can templatize the Hadoop cluster, added capabilities to make it extendable to other services and providers, and made it very developer and ops friendly - that’s when Loom was born.

Today, Loom is used to provision Sandbox Reactors on the Continuuity Cloud and as an internal DevOps tool to test new and incremental features on an ongoing basis. We’ve been developing and using Loom for almost a year and we’re noticing the benefits: our developer and DevOps productivity has doubled, and our development and test cycles have become faster.

We built Loom for ourselves. But today, we’re open sourcing it for anyone to use because we believe that the broader developer community can benefit from it.

What is Continuuity Loom?

Continuuity Loom is cluster management software that provisions, manages, and scales clusters on public clouds and private clouds. Clusters created with Loom utilize templates of any hardware and software stack, from simple standalone LAMP-stack servers and traditional application servers like JBoss to full Apache Hadoop clusters comprised of thousands of nodes. Clusters can be deployed across many cloud providers (Rackspace, Joyent, etc.) and virtualization platforms (like OpenStack) while utilizing common SCM tools (Chef, scripts, etc.).

Loom from Your Laptop

Developers can use Loom to provision and decommission a cluster directly from their laptop. Loom creates a very simple user API that allows the developer to choose the type of cluster, how many nodes, and which cloud. Any developer can go from zero to Hadoop, from their laptop to any cloud, in minutes.

Worry-Free Provisioning

Loom simplifies the installation and configuration of any software stack, including Hadoop, and ensures that all installations are verified before a cluster is made available. Developers can create custom cluster types ranging from single VM application servers to large-scale, distributed, heterogeneous clusters.

Download Loom for Free

As a developer, building Big Data applications using Continuuity Reactor is just one part of the equation. The other part involves provisioning and managing your Hadoop cluster, and deploying your applications to the Hadoop cluster. With Loom, we’re empowering DevOps in a truly self-service way. Download and run Loom on a Mac laptop (currently experimental), and start or spin up Hadoop clusters and other software stacks across any private or public cloud.

Open Sourcing Loom

Loom is a relatively new project but highly extensible in the providers and services that can be supported. Open sourcing Loom allows others to add support for new software stacks, currently unsupported recipes/scripts, and new infrastructure/provisioning systems. Help us make Loom better by contributing here and please report any issues, bugs, or ideas.

Loom for the Enterprise

In many companies, developers must submit requests to individuals in other parts of the organization to obtain a Hadoop cluster or to run distributed software on a group of machines. This process can be incredibly slow and burdensome for both the developer and IT, decreasing developer productivity and increasing IT overhead. With Loom, the IT department can codify reference architectures and maintain a centrally controlled catalog of cluster templates that can be provisioned directly by developers. IT administrators then use a centralized dashboard to monitor and manage multiple clusters concurrently across multiple clouds.

Download Loom to start managing and provisioning your own clusters today, or contact sales if you need enterprise support for Loom.

0 notes

Comments
blog comments powered by Disqus