Building a Modern, Scalable Cyber Intelligence Platform with Confluent Kafka

As advanced cyber threats continue to grow in frequency and sophistication, many large enterprises need to operate more effectively to protect their environments, especially as they grow in new markets. At Intel, we addressed this need by transforming from a legacy systems, to a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Confluent Kafka. Our CIP helps our Security Operations Center identify and respond to threats faster, and supports hundreds of use cases coming from our entire Information Security organization. Today, our CIP ingests tens of terabytes of data each day and transforms this data into actionable insights with context-smart applications, streams processing, and machine learning. With Confluent Kafka serving as the core pub/sub message bus, we built a massive security data pipeline that achieves economies of scale by acquiring data once and consuming it many times. This pipeline helped us reduce our technical debt by eliminating legacy point-to-point custom connectors for our security controls and analytic solutions. At the same time, Kafka has given us the ability to operate on our data in-stream, shifting the needle of Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR) towards Near-Real-Time (NRT) in many cases. In our session, we’ll share how we architected and implemented CIP with pub/sub messaging system and streams processing, and some of the valuable lessons that we learned along the way. We’ll discuss the benefits of a highly integrated, yet loosely coupled set of security capabilities. We’ll also talk about some of our new threat detection techniques, such as:
• Filter, enrich, aggregate, join, and normalize data in-stream to deliver contextually rich and clean data, downstream to things like SIEM
• Apply logic to automate mundane tasks such as deduplication, auto branching, and filtering out false positives, extraneous, and/or bad data
• Maximize cluster availability and performance with tools like Confluent Control Center (C3), Replicator, and Multi-Region Clustering (MRC)
• Hunt for threats in-stream and NRT, with Kafka Streams and machine learning techniques Screen reader support enabled.