How HTAP Can Tame the Internet of Things Data Deluge

By Sylvie Barak

600 ZB a year. 

Say what?

That’s the total amount of data Cisco forecasts will be pumped out of IoT devices by 2020, up from a whopping 145 ZB measured by the company in 2015 and reported in its Cisco Global Cloud Index: Forecast and Methodology, 2015–2020.

That gargantuan level of Internet of Things data will be thrust into existence by the approximately 20 billion connected devices Gartner says we can expect to have ticking away by the end of 2020.

What do we do with all that IoT data? How can we possibly make sense or use of it? The answer to that question lies, not on the surface, but below it, at the level of the data architecture.

The structure of the architecture needs to allow the data room to scale, because, let’s face it, when it comes to data, less is not more. The architecture also has to be reliable enough to allow constancy, because data ideally comes in a strong, steady stream. In modern business, the key to success is applying the best algorithm to the most data to deliver real-time insights. This only works, however, when data read/write speeds are so fast, it’s practically instinctual. And that’s a hard problem to solve for when you’re drowning in oceans of data.

Enter the sexy-sounding hybrid transactional and analytical processing (HTAP) with in-memory-computing (IMC), which has analyst firms practically drooling. Gartner, which coined the term, reckons HTAP will have a “transformational impact on digital business,” while Forrester has bluntly declared that “traditional architectures don’t cut it anymore.” Ouch.

Defining HTAP

So what exactly is HTAP? 

An HTAP architecture supports the needs of many new IoT use cases that require scalability and real-time performance. It enables instant decision making by bringing transactional data and analytics together at the time of the transaction. An HTAP architecture is best enabled by IMC techniques and technologies to provide analytical processing on the same (in-memory) data that is used to perform transaction processing.

By removing the latency associated with moving Internet of Things data from operational databases to data warehouses and data marts for analytical processing, an HTAP architecture enables real-time decision-making and situation awareness on live transaction data as opposed to after-the-fact analysis on stale data. 

Until recently, caching was considered to be the best closed solution to enable fast transactions and analytics, and companies typically preferred spending their hard-earned cash on those solutions. But these have bumped up against significant limitations of scale and reliability.

Today, however, in-memory HTAP solutions are becoming more common thanks to new developments in memory-based storage like SSD and Flash and databases that can leverage non-volatile memory. The simplified architecture of HTAP solutions also offers considerable cost savings potential compared with two side-by-side solutions—one for online transactional processing (OLTP) and one for online analytical processing (OLAP).

An Array of Applications

That’s all well and good, but what are the actual use case scenarios where HTAP comes into its own for IoT? The answer, perhaps unsurprisingly, is a large and growing number.

For a start, any device that needs high-speed data processing and the ensuing real-time insights to drive actions. That’s pretty much anything involving a sensor for operational purposes: autonomous vehicles, manufacturing machinery, inventory monitoring technology, patient monitoring in healthcare, connected home gadgets, and more. Not to mention mining the explosion of data resulting from video captured by things like smartphones, GoPros, body cams and CCTV. 

It’s a travesty that 90% of generated data doesn’t get analyzed and acted upon (that’s according to Gartner, not me), and as proliferation of sensors and Industrial Internet of Things apps grows, so does the amount of useless data.

Enter edge analytics, a particularly, um, edgy approach to dealing with the problem.

Instead of shipping all the data off to the cloud for processing, edge analytics allows real-time processing to happen at the point where the Internet of Things data is being generated. Built-in analysis, so to speak.

As the density of high-data-rate IoT sensors grows, edge analytics makes more and more sense, and that’s where HTAP shines.

HTAP applications, just like most IoT cases, run algorithms that do analysis and make decisions on quickly-moving data. These applications require high-speed processing for high volumes of data to power algorithms. Currently, only a handful of companies are even capable of delivering high-speed and large-volume data processing that is critical for HTAP with an in-memory database. 

Brian Bulkowski, the CTO of Aerospike, which has focused on enabling HTAP solutions, builds its database with hybrid memory architecture. On the Aerospike website, it explains that, with this form of architecture, “the index is purely in-memory (not persisted), and data is stored only on a persistent storage (SSD) and read directly from the disk.”

Bulkowski explains his preference for hybrid memory architecture by stating that “more and more businesses need to drive decisions in the moment and deploy intelligent transactional applications.” Such applications require a database that can process high data volumes at extreme speed and “do it without stacking up thousands of servers,”Bulkowski adds. “For us, it was clearly a problem that can only be addressed at the core architecture and hybrid memory architecture helped us solve this problem.”

Companies that provide high-speed processing for HTAP as a service also need to ensure they have an in-memory compute grid for high-performance parallel processing of queries and in-memory streaming processing so that incoming data gets analyzed in real time.

The payoff, however, could be enormous for those who get it right.

“Customer experience, operational excellence and new revenue sources are the strategic outcomes for business. Intelligent control systems, real-time asset optimization, in-store targeting, anomaly detection, real-time capacity planning are some areas with immediate business impact,” notes Bulkowski.

The bottom line? If we want the world of IoT to fulfill its promise of integrating fully into our lives, performing at a high level, having scalability and dealing with a burgeoning stream of data that has to be interpreted and turned into action, in milliseconds, HTAP is where we need to stack our chips.