Join Newsletter
Trusted Business Advisors, Expert Technology Analysts

Taneja Blog

Taneja Blog / Big Data / Data Center Systems

What was BIG at Hadoop Summit 2015

At this month's Hadoop Summit 2015 I noted two big trends. One was the continuing focus on Spark as an expansion of the big data analytical ecosystem, with main sponsor Hortonworks (great show by the way!) and most vendors talking about how they support, interact, or deliver Spark in addition to Hadoop's MapReduce. The other was a very noticeable direction shifting focus from trotting out ever more gee-whiz big data use cases towards talking about how to make it all work in enterprise production environments. If you ask me, this second trend is the bigger deal for IT folks to pay attention to.

Spark for sure represents a whole "nother" set of opportunities for ambitious organizations to find and mine value out of their big data sets, leveraging the speed and power of in-memory computing at scale. If you haven't looked at Spark yet, its probably time to give it the once over and see if it can address any of those challenges that MapReduce didn't quite meet.

Yet the enterprise focus of the other discussions going on was more validating for those of us who have kept beating the drum of big data for everyone. Finally I think we've reached a stage of maturity in which the big data solution space starts to catch up with enterprise needs for security, availability, reliability, data protection, business continuity and so on. I'm not totally convinced about data lakes taking in ALL data center data, but with recent technologies coming into the main distributions to provide auditing, access control, data management, and fast encryption, enterprise needs are being quickly addressed.  And with increasing deployment and provisioning solutions that are making big data clusters more and more one-button "cloud-able", I'm certain that the rank and file IT shop will soon be able to deploy and support big data solutions as easily as they do RDBMS databases today (although that might not be such a comforting statement to some!).

I'll be writing more about some of these enterprise trends in big data soon, including a feature article coming up on the latest storage approaches and options for Big Data.  In the meantime, check out this first part of a two part article on machine learning I wrote for Modern Infrastructure that just came out.

  • Premiered: 06/29/15
  • Author: Mike Matchett
Topic(s): Big Data Hortonworks Hadoop


There are no comments to display. Scroll down to leave your own!


Leave a Comment

You must be logged in to comment. Click here to log in or register if you don't have an account.