Enter the Matrix of Log Data

The groundbreaking science-fiction film The Matrix opened with a cascade of green digital rain—code that represented the activity of the virtual world.

There is a similar torrent of data that is being gathered by logs. But whereas in the Matrix, this code could form buildings, cars, and other objects, in the real world, a torrent of log data is gathered by them.

The implications of log data in the automotive industry alone are staggering. Referring to the ability of cars to track GPS logs, Ford’s Global VP/Marketing and Sales, Jim Farley declared two years ago: “We know everyone who breaks the law,” He distanced himself from that statement soon after that.

The fact remains that connected cars log an increasing amount data. The vast majority of cars today have event recorders—so-called “black boxes”—that monitor crash data. Some cars even track whether you wear a seatbelt. And if you are wearing one, some cars log whether you are accelerating or braking hard enough to cause the seat belt to lock up to brace for a potential accident. At least one automaker even track whether their customers favor listening to CDs or MP3s. “The CD/MP3 example is comparatively harmless, but should a car maker really know what my driving style is?” asks Loggly’s senior director of product marketing Sven Dummer.

While issues like that may have no easy answer, logs can play an invaluable role in determining who was at fault in an accident and can prove useful for a spectrum of other applications. Consider how carmaker Tesla, for instance, was able to argue against a driver who claimed that a Tesla Model X SUV accelerated on its own, crashing into a building. The company released log data indicating that the car crashed because the driver suddenly hit the acceleration pedal—not because the car somehow decided to do that. And now that there are reports of the first death associated with a self-driving Tesla that surfaced in late June, the National Highway Traffic Safety Administration is doubtlessly considering that car’s log data as crucial evidence as they investigate the matter.

Logs: A Common Denominator for the IoT

Log data is all around us, whether they are gathered by vehicles, smartphones, smart televisions, wireless routers, industrial machines, and oil rigs. “Pretty much all software and every device that runs software writes log data,” Dummer says.

One of the primary applications of log data is troubleshooting. “That data is a very powerful for debugging software and understanding why a device failed,” Dummer says. “The log data will give you a tremendous volume of data that can help you find out what to do.”

Log data can also be used for business intelligence and is often used to inform product design. As alluded to before, car makers use log data to help inform the feature sets that go into new vehicles. Video game developers are using log data to help inform game design. “The applications are practically endless,” Dummer says.

The information is also useful for protecting against hackers. “Log data will tell you what IP address a denial of service attack is coming from or what group of IP addresses is responsible,” Dummer says. “You can configure your firewall to block them.”

Indeed, logs already play a crucial role in the IoT. “I think, overall, logs are the only common denominator of all of these systems,” Dummer explains. “Devices that didn’t used to be connected are all connected, and virtually all of these devices are gathering log data. The type of data you can record is indefinitely large. There are a lot of implications in almost all areas of society.”

Sorting Through the Matrix of Log Data

In many cases, the volume of log data is so vast that it would be impossible for a single person to follow the data in a terminal window. “A user would see a Matrix-type of a screen,” Dummer says. “The volume of log data has increased tremendously over the past few years,” he says. “One common example in the literature is an oil rig with tens of thousands of sensors that generate terabytes of data per day. That is something of a special case, but even an average company generates gigabytes of data every day.”

Although log data may be ubiquitous, it is not standardized. Loggly can digest log data from different formats and sources and then display in a single dashboard view. Think of it is as sort of Google Analytics that can accept virtually any log data. “We aggregate log data and allow you to run analytics,” Dummer says. “Loggly is kind of like the Swiss Army knife. You can send any log data from any source to Loggly, whether it be a server, desktop computers, or devices, or a combination of those.”