RiskIQ, an external threat management company, is using the MapR Converged Data Platform to provide threat detection on large datasets. RiskIQ’s external threat management platform scans, analyses and stores entire websites, mobile app stores, and social media outlets across the internet. As the amount of data and sophistication of security threats grew, RiskIQ wanted to be able to provide clients with faster, more comprehensive threat detection analysis. RiskIQ selected the MapR Platform and has been using it as a distributed storage system for several years. More recently the company is using MapR with Apache Spark, Hive, Parquet, and Oozie and has introduced new threat detections product offerings for their customers.
Chris Kiernan, RiskIQ CTO and co-founder, said: “The MapR file system architecture is something we appreciated from the start. We knew we could leverage MapR for almost anything we wanted to do. With the node management and the way clustering is done, we always knew it was built in the right way if we needed to do analysis.”
RiskIQ’s 100 web crawlers collect about 10 to 20 TBs data each day from across the internet. This continues to grow as RiskIQ adds depth and new datasets to their crawl data. To reduce the size of the data, the company developed a technique to create Parquet files from raw crawl data that are 10x smaller and sent via NFS into their warehouse for analysis. “We can query an entire day’s worth of files in minutes instead of hours,” said Kiernan. “There are all kinds of things we can do, now that we have the data in a compact format. It’s a central part of our architecture.”
RiskIQ was able to keep costs down by building the new data analysis use case on top of their MapR cluster. “We continue to use the cluster as a production file system while, at the same time, we’ve built an entire warehouse using the same infrastructure for a very small price point,” said Kiernan. “We have been able to cut Capex and Opex in half. We would have had to pay twice as much to build a vanilla Hadoop cluster. If we had built this in Cloudera, we would have needed separate clusters for production and analytics. It wouldn’t be a dual-purpose system.”
“The fact that MapR makes sure that everything is compatible has worked really well. If we want to try a new technology, we can install it and it’s ready to use,” said Adam Hunt, RiskIQ’s chief data scientist. “In my previous job, I worked with vanilla Hadoop, and the cluster did go down. That’s just not acceptable. We’ve never had that issue with MapR. It’s rock solid. We don’t see performance degradation no matter what we do to the cluster and upgrades are seamless.”
With the MapR Platform, RiskIQ has developed new products that has allowed them to push new types of data into their application that helps them understand things about websites they may not have understood before. “We have built all new parts of our products based on this new analysis, so it has been absolutely instrumental to our host reputation service. We can now answer all of the ad hoc questions we could never answer before to provide even more advanced detection for our clients. It’s improving the way we run the business,” said Kiernan.
