Continuuity Loom 0.9.7: Extensible cluster management

Jun 10 2014, 11:12 am

Derek Wood is focused on DevOps at Continuuity where he is building tools to manage and operate the next generation of Big Data applications. Prior to Continuuity, Derek ran large scale distributed systems at Wells Fargo and at Yahoo!, where he was the senior engineering lead for the CORE content personalization platform.

In March, we open sourced Continuuity Loom, a system for templatizing and materializing complex multi-tiered application reference architectures in public or private clouds. It is designed bottom-up to support different facets of your organization - from developers, operations and system administrators to large service providers.

After our first release, we have heard a lot of great things about how people are using Continuuity Loom and have also heard about features that have been missing. After taking in all the feedback, we are excited to announce the next version of Continuuity Loom codenamed Vela. The theme for this release is “Extensibility” as we have been working towards making Continuuity Loom integrate with your standard workflows for provisioning and managing multi-tiered applications, as well as making it easier to extend Continuuity Loom itself to your needs.

Highlights of Loom 0.9.7

  • Ability to register plugins to support surfacing of plugin defined fields for configuring providers and automators

  • Support for finer grained dependencies

  • Cluster life-cycle callback hooks for integrating with external systems like metering, metrics, etc.

  • Extend your cluster capabilities by adding new services to an existing cluster or reconfiguring existing services

  • Smart support for starting, stopping and restarting services with dependencies automatically included during service operations

  • Personalizable UI skins

  • More out-of-box Chef cookbooks including Apache Hive™ recipes and recipes for supporting Kerberos enabled clusters

Detailed overview of Loom 0.9.7 features

Cluster Reconfiguration

How many times have you had to change some configuration setting of a service on a cluster and had to remember to restart the right services in the right order for the change to be effective? The reality is, when making changes in multi-tiered application clusters, there are a lot of things to remember. Wouldn’t it be simpler if you could change the configuration that you want and let the system figure out everything else for you, ensuring the change you just made is active without any hassle? With this release of Continuuity Loom, you don’t have to worry any more about what services need to be restarted. Continuuity Loom automatically figures out service stop and start order based your service dependencies. You can find more information about how this is done here.

Add missing and manage cluster services easily

Let’s take a concrete use-case: let’s say your administrator has configured a template that allows you to create a Hadoop Cluster (HDFS, YARN, HBase, Zookeeper) with Kerberos security enabled. As a new user, you would like to try building a basic cluster with just HDFS and YARN until you are ready to add more. In that case, Continuuity Loom provides an easy way to remove services that you don’t need during creation time and then subsequently add them back to the live cluster with just a few clicks. With this new release, users will now have the ability to stop, start, and restart services without having to worry about what additional or dependent services need to be restarted.

Plugin Registration

In line with our theme of extensibility, we wanted to ensure that developers are able to write custom Automator and Provider plugins. As part of this, plugins now define the fields they require, which get surfaced in the UI and passed to the plugin during task execution. Particularly for Provider plugins, this allows you to provide different options for provisioning clusters. For example, Rackspace requires a username and API key, while Joyent requires a username, key name, and private key file. It is now possible to write your own plugins and describe the specified fields required. With the addition of this feature, you can also write support at the API level for any container (like Docker), OS, or cloud provider. You can learn more about this feature here.

Finer Grained Dependencies

Prior to this release, all service dependencies were applied at all phases of cluster creation. This created unnecessary solution space exploration and execution of dependencies when they didn’t make sense. This release includes a feature called fine grained dependency management for services. This feature allows Continuuity Loom administrators to specify required or optional service dependencies that apply during runtime or install time (applied only when the service is installed and available on the machine). This is specified during service creation so users don’t have to worry about it. It provides granular control over the deployment of services and opens up support for HA Hadoop clusters and secure hadoop clusters, which require external Kerberos Key Distribution Centers (KDCs). You can learn more about this feature here.

Cluster life-cycle callback hooks

Often times, you need the ability to integrate Continuuity Loom with other workflows that exist in your organization. This feature allows the Continuuity Loom administrator to configure a callback class to insert your own custom logic before and after cluster operations. Out of the box, Continuuity Loom provides a HTTP callback implementation for the interface ‘ClusterCallback’. You can use this feature to integrate the cluster life-cycle with monitoring systems, metering systems, and even your favorite chat application to alert when clusters are created, deleted or upon any failed operations. You can learn more about this feature here.

Personalized UI skins

When you install Continuuity Loom on-premise wouldn’t you like the ability to change the color scheme and logo to make it fit well with the other tools in your organization? With this release you will have the ability to change the color, skin and logo of your Continuuity Loom installation.

This has been an exciting release for us. Check out the Release Notes for more details about this release. Give it a spin by downloading the standalone version for your laptop and visiting the quickstart guide to spin up clusters in cloud.

Help us make Continuuity Loom better by contributing and please report any issues, bugs, or ideas.

Coming soon - Continuuity Loom for free in the Cloud

Be sure to sign up at tryloom.io to be among the first to know when the free, cloud-based version of Continuuity Loom is available.

Comments

Continuuity Reactor 2.1: New Developer Experience and Windows Support

Mar 6 2014, 8:04 am

Sreevatsan Raman is a software engineer at Continuuity where he is building and architecting a platform fueling the next generation of Big Data applications. Prior to Continuuity, Sree designed and implemented big data infrastructure at Klout, Nominum, and Yahoo!

Last November at Strata + Hadoop World 2013, we came out of public beta and launched Continuuity Reactor 2.0. Our goal was to enable developers and businesses to deploy production Big Data applications on our platform. Since then, we’ve been maniacally focused on making our developer experience better for all Java developers, with or without Hadoop experience. Today, we’re happy to announce the next version of Continuuity Reactor: 2.1.

In Continuuity Reactor 2.1 release, we have the following new features:

New Developer Experience

We completely revamped our developer documentation and out-of-the-box experience to make it easier for Java developers–who may not be familiar with Hadoop or Big Data–to get started with Continuuity Reactor. We added real-world examples for you to get a quick and simple overview of our platform, a helpful guide to the Reactor user interface, and a streamlined SDK download so you can get started without missing a beat.

Windows Support

Some developers build Big Data applications on Windows machines. So, we added support for running single-node Reactor on 64-bit Windows (Windows XP and above).

Additional Features

The new release will add support for HBase 0.96, which has features like improved scalability, lower mean-time-to-recovery, and support for larger clusters (more than hundreds of nodes). We also fixed a number of bugs and made our platform more stable.

We are working hard to solve the problems faced by both new and experienced Big Data developers. Reactor unifies the capabilities you need, in an integrated developer experience, without the worries of distributed system architectures or scalability. You will hear more from us as we progress, and we would love to hear from you. Feel free to leave a comment or send us feedback at support@continuuity.com.

Download the Continuuity Reactor 2.1 SDK and visit the developer documentation to get started.

Happy hacking!

—Sree

0 notes

Comments

What do you do at Continuuity, again? Part 2

Feb 4 2014, 8:54 am

Alex Baranau is a software engineer at Continuuity where he is responsible for building and designing software fueling the next generation of Big Data applications. Alex is a contributor to HBase and Flume, and has created several open-sourced projects. He also writes frequently about Big Data technologies.

Let’s make it awesome!

In our previous post, we introduced the basics of our Continuuity platform by using a simple example of a real-time processing application. In this post, we’ll take a step forward and introduce you to the details of building a non-trivial, real-time, data processing application.

Read More

0 notes

Comments

Programming with Apache Twill*, Part II

Jan 21 2014, 10:41 am

Terence Yim is a Software Engineer at Continuuity, responsible for designing and building realtime processing systems on Hadoop/HBase. Prior to Continuuity, Terence spent over a year at LinkedIn Inc. and seven years at Yahoo!, building high performance large scale distributed systems.

In the Programming with Weave (now Apache Twill), Part I blog post, we introduced the basics of writing a distributed application on Hadoop YARN using Twill. In this post, we are going to highlight some of the important features in Twill.

Read More

0 notes

Comments

What do you do at Continuuity, again?

Nov 20 2013, 12:10 pm

As Continuuity gets more traction, my friends ask me about what I do at Continuuity. The short answer is - we’ve created a platform which makes building Big Data applications easier. Let me try to give you more details with a short example. Let’s imagine you need to implement an app.

The app

This example app is very simple. I will not reason why one should implement it. The point of the example is to let me walk you through the developer experience of implementing a Big Data app using Continuuity Reactor.

Read More

0 notes

Comments

Twill, formerly Weave, accepted into the Apache Incubator

Nov 14 2013, 10:51 pm

For the past few years, applications have been generating hundreds of petabytes of data. Analyzing this data can create real business value, but until recently businesses had discarded the data because it was either too hard to analyze or too expensive to store using traditional relational databases.

The answer to this problem is a free, open-source technology running on commodity hardware: Apache Hadoop. Hadoop makes it cheap to store Big Data and easy to extract valuable insights using a batch-driven analysis method called MapReduce. It turns every web app into a data-driven app.

Read More

0 notes

Comments

Programming with Weave, Part I

Nov 11 2013, 10:36 am

Read first blog post of the Weave series

In this second blog post of the Weave series, we would like to show you how writing a distributed application can be as simple as writing a standalone Java application using Weave.

Writing applications to run on YARN

Before we dive into the details about Weave, let’s talk briefly about what a developer has to do in order to write a YARN application, other than standard MapReduce. A YARN application always consists of three parts:

Read More

0 notes

Comments

Simplifying YARN - Introducing Weave to the Apache Hadoop Community

Jun 11 2013, 6:21 am

This post is the first in series that introduces Weave, Continuuity’s recent contribution to the Apache Hadoop community.  Through this blog series, we will share our experiences with YARN, the motivation for building Weave, and provide an overview of the Weave architecture along with use cases that show the problems Weave can help you solve. 

Read More

0 notes

Comments
blog comments powered by Disqus