Join Newsletter
Trusted Business Advisors, Expert Technology Analysts

Taneja Blog

Taneja Blog / Big Data / Cloud / Data Center Systems

IoT Goes Real-Time, Gets Predictive - Glassbeam Launches Spark-based Machine Learning

In-Memory processing was all the rage at Strata 2014 NY last month, and the hottest word was Spark! Spark is big data scale-out cluster solution that provides a way to speedily analyze large data sets in-memory using a "resilient distributed data" design for fault-tolerance.  It can deploy into its own optimized cluster, or ride on top of Hadoop 2.0 using YARN, (although it is a different processing platform/paradigm from MapReduce - see this post on GridGain for a Hadoop MR In-memory solution).

I've covered a few in-memory topics lately here and in some of my recent columns. But I haven't done justice to Spark itself and perhaps its biggest onrushing use case - taming the real-time data from from the Internet of Things (IoT).  Spark has a few add-on solutions for graph data, machine learning, and SQL access (all also hot topics!). In particular, the machine learning I think represents an incredibly interesting opportunity for getting value out of new streams of IoT data. Just having big IoT data won't be enough, and just loading it into something like Splunk will make it accessible but not valuable in itself.  The big payoff here is in having our big compute clusters actually compute something that gives us intelligence - from dynamic operational decision support and tactical "best" option recommendations, to the comparative evaluation of strategic models of future direction and performance.

I'm not suggesting we should all be thinking about how to get into computer-assisted stock trading, but within our own lines of business there are already likely vast data sets that have been going to waste. As a fast, low-risk subscripion based approach for example, we looked recently at Glassbeam and their SaaS solution for vendors of IoT-like equipment (their customers can produce everything from thousands of storage arrays with TB of daily call home data each to billions of small devices scattered everywhere producing hourly micrologs).  Just this week Glassbeam upped their game with a release of new Spark-based capabilities, including the machine learning layer. Now instead of looking at data the next day or week, Glassbeam can support a near real-time look at your whole IoT. And what I'm most excited about, start to help you build those near real-time predictive analytics.

I'm betting that most folks haven't really done due diligence on their data sets to really envision what value might be locked up within. We tend to focus on the pain points, and to some extent Spark and MLlib can unlock some long-awaited medicine to thorny problems. But the bigger opportunity I think will come to those who seek out new opportunities, not just try to fix old pain points.  So here is a challenge for 2015 - Make a resolution to spend a half day brainstorming with your best and brightest techies and business folks together about what might be possible with these new very accessible, affordable, and available capabilities.

  • Premiered: 11/21/14
  • Author: Mike Matchett
Topic(s): Glassbeam Big Data Spark In Memory GridGain Machine Learning


There are no comments to display. Scroll down to leave your own!


Leave a Comment

You must be logged in to comment. Click here to log in or register if you don't have an account.