Continuuity Reactor 2.3: SQL and Security Release

Jul 23 2014, 10:22 am

Alex Baranau is a software engineer at Continuuity where he is responsible for building and designing software fueling the next generation of Big Data applications. Alex is a contributor to HBase and Flume, and has created several open-sourced projects. He also writes frequently about Big Data technologies.

The Continuuity Reactor platform is designed to make it easy for developers to build and manage data applications on Apache Hadoop™ and Apache HBase™. Every day we’re passionately focused on delivering an awesome experience for all developers, with or without Hadoop expertise. And today, we’re excited to release the next version of our platform, Continuuity Reactor 2.3.

In addition to continued stability, scalability, and performance, we have added a number of significant new features in Continuuity Reactor 2.3:

Ad-hoc SQL Queries

Procedures are an existing, programmatic way to access and query your data in Reactor, but sometimes you may want to explore a Dataset in an ad-hoc manner rather than writing procedural code. Reactor now supports ad-hoc SQL queries over Datasets via a new API that allows developers to expose the schema of a Dataset and make it query-able through a REST API. This enables the submission of SQL queries over Datasets along with retrieval of the results, submitted via REST and executed via Apache Hive or other Hadoop-based SQL engines.

Security Enhancements

We’re committed to making Hadoop applications secure. Continuuity Reactor now supports perimeter security, restricting access to resources only to authenticated users. With perimeter security, access to cluster nodes is restricted by a firewall. Cluster nodes can communicate with each other, but outside clients can only communicate with the cluster through a secured gateway.

Using Reactor security, the Reactor authentication server issues credentials (access tokens) to authenticated clients, and clients then send these credentials with their requests to Reactor. Calls that lack valid access tokens are rejected, limiting access to only authenticated clients. You can learn more about the authentication process on the Reactor Security page.

Additional Release Highlights

Other key enhancements in 2.3 include new Application, Stream, Flow, and Dataset features such as:

  • Stream support for data retention policy; reconfigurable at runtime, while in use
  • Stream support for truncate via REST
  • Simplified Flowlet @Batch support with process methods no longer requiring an Iterator
  • New Datasets API that gives more power and flexibility when developing custom Datasets
  • Dataset management outside of Applications exposes REST interfaces to create, truncate, drop and discover Datasets
  • New Application API with an improved way to define application components

Finally, we have added Reactor Services, an experimental feature that allows the addition of custom User Services that can be easily discovered from within Flows, Procedures and MapReduce jobs. We’ll have more services capabilities in our next release, but you can get an early preview of one of the features we are most excited about right now!

Try Reactor 2.3 Today

We are working hard to solve the challenging problems faced by both new and experienced data application developers and to enable a much more fun and productive development experience for Hadoop. Reactor unifies the capabilities you need when developing on Hadoop into an integrated developer experience so that you can focus on your application logic without the worries of distributed system architectures or scalability. Download the Continuuity Reactor 2.3 SDK and check out the developer documentation to get started.

We are excited about the latest release and would love to hear your thoughts. Please feel free to send us feedback at support@continuuity.com.

Comments

Behind the scenes: Hacking our way to success

Jul 7 2014, 8:47 pm

Sreevatsan Raman is a software engineer at Continuuity where he is building and architecting a platform fueling the next generation of Big Data applications. Prior to Continuuity, Sree designed and implemented big data infrastructure at Klout, Nominum, and Yahoo!

We just wrapped up our latest hackathon and it was a great reminder of the unique engineering culture we have at Continuuity. We have created a new application development platform, Continuuity Reactor, which is focused on allowing developers to quickly and easily build Big Data applications.

Building a platform that no one has created before is a big challenge. We break this huge effort into a continuous cadence of platform releases that are delivered to production frequently. Before every release we take a break from our daily efforts and hack on our platform for 48 hours where we stretch our imaginations and the platform capabilities we just built.

Every hackathon gives us an opportunity to dog-food our technology. We come together wearing our developer hats to build features and applications, incorporating our lessons learned into continually improving the developer experience, with the goal of making Hadoop more simple and accessible.

One of my favorite aspects of our hackathons is how the whole company comes together to build cool stuff and have fun. From our CEO to our engineering team to people in non-technical roles, everyone participates. Here are some thoughts and experiences about our company, culture, and hackathons from our awesome engineering interns:

Shu Das, University of Michigan

The unique aspect of Continuuity that I like is that everyone has a clear sense of his or her agenda and responsibilities, so we’re empowered to stay on top of our game. Not only do I have the resources I need and responsiveness from the rest of the team, but also the working environment at Continuuity is lively and enjoyable.

My first project was building an application on Reactor that visualizes data about the test cases we run on our code. This work gave me great insights into what our platform is, how to use it, and how our technology can be used for simplifying Hadoop. I really appreciate the fact that the feature I worked on is used daily, as a component of the development lifecycle, and not left off as a side project.

For the hackathon, I teamed up with Kenneth and Gourav (see below) to build a Reactor application that can be used to aggregate, correlate, and visualize data - for instance, metrics, logs, or any other events. It was amazing to see the application built in a very short amount of time using new core functionalities of the platform and dogfooding the new APIs, runtime, and documentation.

Gourav Khaneja, University of Illinois

The work here is interesting because the problems we’re solving are hard. One of my favorite aspects of Continuuity is the willingness of team members to help each other to work through challenges. For example, even during crunch time, every Continuuity member is willing to stop what he or she is doing to help out a fellow employee. I learn a lot from the team on a daily basis.

When I joined, I was tasked with optimizing resource allocation in YARN using Apache Twill. YARN has a large codebase and although my previous experience with a large code base was limited, I was able to come up to speed quickly with great mentorship from the team and contributed towards a major feature in Twill.

Kenneth Le, University of California, Berkeley

Interns are involved in relevant projects right away. While we receive guidance when needed, the focus of the internship program is more on empowering us to deliver and learning more via open communication about the various projects that other people are working on.

My first project was improving a developer tool that is used to deploy code to clusters. The existing tool took about 30 minutes to build and deploy the entire code base. The newer version, which I rewrote in Python, takes about 6 minutes, thus saving developers a lot of time in their development life-cycle.

Julien Guery, Ecole nationale supérieure des Télécommunications de Bretagne

This is an extremely technical company solving challenging problems. One of the first things I noticed is that the interns get to be part of the core engineering team and are involved in all aspects of the company.

In my first project I learned a lot about Apache Hive and the Reactor platform while working on a feature to bring ad-hoc quering capabilities in to our platform. I had great mentors who taught me how to test and debug and gave me insights into the architecture of the systems, and now I can dive right into new projects and teams without fear.

During the hackathon, I used our APIs to build a Python SDK. I wanted to showcase how Python developers can easily write big-data applications using our platform and my efforts during the hackathon demonstrated how this could be accomplished. The hack was well received and a updated version of this SDK will be made available in a future release.


Our team is working to solve a difficult problem – making Hadoop a platform upon which data applications can be built by all developers. Whether at our hackathons or at our weekly company-wide demos, we are constantly sharing and collaborating so everyone can understand the impact that they have and the context of how their contributions map to the overall vision and mission of the company.

If you’re interested in learning more about our culture and careers opportunities at Continuuity, check out http://continuuity.com/careers.

Comments

Hadoop Summit: Where is the value? Where are the apps?

Jun 24 2014, 8:00 am

Jonathan Gray, Founder & CEO of Continuuity, is an entrepreneur and software engineer with a background in open source and data. Prior to Continuuity, he was at Facebook working on projects like Facebook Messages. At startup Streamy, Jonathan was an early adopter of Hadoop and HBase committer.

Coming out of Hadoop Summit, one thing is clear to me – while there has been significant growth and success of the ecosystem, it is still early days and Hadoop is still exceptionally hard to consume for most organizations. As a result of this persistent issue, there weren’t many major announcements, nothing exceptionally new or different released, and the buzz remained largely centered on YARN and Spark, both of which are several years old.

While we saw reports of early adopting companies seeing real value created with Hadoop, the focus was more technical this year than I anticipated—from the keynotes to the breakout sessions to the show floor, this year’s summit seemed more about the endless variety of different technologies than use cases and actual return on investment realized. A brief overview of a few other trends we observed is below:

Hadoop is not quite enterprise ready…yet

Hadoop Summit generated significant discussion about whether Hadoop is truly ready for real, production enterprise use. Of particular concern is security and related issues of privacy and data policies needed for companies, especially those dealing with customer or financial information. Recent acquisitions of Hadoop security upstarts by the major Hadoop distributions indicate that this will continue to be an important area of focus in the near term.

Hadoop vs. The EDW: To Replace or To Augment

Another hot topic is whether Hadoop is a replacement for the traditional EDW or if it is only to augment and offload certain workloads. In years past, this has been much more of a debate; however this year it seems clear that most have accepted a symbiotic relationship for the time being. While I do expect this to change, it is evident today that there is a significant gap in the capabilities of the Hadoop stack compared to proprietary EDW technologies.

Hadoop is becoming more fragmented

This year it became apparent that the Hadoop ecosystem is splintering into multiple and often competing projects. Competing vendors are establishing parallel but increasingly separate stacks while differentiated vendors are marketing overlapped messages. There has been an explosion in the variety of ways to work with Hadoop and in the number of companies trying to make Hadoop consumable, and it’s becoming even more confusing to choose which path is best to follow. This is true not only for business leaders who are making decisions about Big Data projects in their company but even for knowledgeable developers.

Hadoop (still) needs to be simplified

This mass confusion in the market is undercutting companies’ ability to achieve value and realize what they want from their Big Data initiatives. A lot of attention is still being paid to the infrastructure rather than the applications, so although the disruptive value of Big Data should be at the forefront, it remains elusive for most.

The Big Data Application revolution is still forthcoming. It is still early days, Hadoop is still very difficult, and very few people understand how to work with it. That’s why we are building a platform that focuses on making Hadoop easier for developers, allowing anyone to build applications (today in Java) without worrying about the low-level infrastructure. Rather than grapple with myriad technology options, they are free to focus on what matters – turning their brilliant ideas for data into apps that solve real problems. This is where Hadoop can produce desired outcomes – in data applications that quickly provide measurable value.

Adding Jet Fuel to the Fire

Not to be left out of the new choices in the Hadoop menagerie, in case you missed it, we announced a project in collaboration with AT&T Labs: a distributed framework for real-time data processing and analytics applications, codenamed jetStream. Available in open source in Q3 2014, you can find more information about this effort in our recent blog post and at jetStream.io.

Comments
blog comments powered by Disqus