Whether investigating an active intrusion, or just scanning for potential breaches, modern cybersecurity teams have never had more data at their disposal. Yet increasing the size and number of data haystacks is making needles much harder to find. Ironically, more and better data has led to an explosion of cybersecurity tools, with agentic AI systems just the latest attempt to address the situation.
One of the first suggestions made by vendors when introducing a new tool is that customers should normalize their data to a common standard, and unify their data into a single repository. In a Tolkienesque appeal to centralization of security logs, there should be one data lake to rule them all, one standard schema to find them.
The idea has intuitive appeal: converting data from different sources into a centralized, standard format makes it easier to make sense of the data. Different systems might refer to the same concept with different names. An IP address might be variously referred to as source_ip, source_ip_addr, client_ip, user_ip or any number of other terms for essentially the same idea. Declaring that all such references shall be mapped to the single, global name of source_ip makes understanding easier, especially for new hires.
This was part of the appeal of systems like Splunk or Elastic or MongoDB originally. Rather than wrestle with carefully defined relational database schemas (reduced to fourth normal form, of course), why not dump everything into one big pile and then let new, advanced algorithms make sense of the data? Why have just a couple of indexes when you could index everything, all the time? Why limit yourself to rows of structured, columnar data when you could store whole documents?
So we did. A lot. And now there are many single sources of truth scattered around the enterprise. Each new tool introduces a new way to store data, and navigating them all is a complex undertaking for even the most experienced senior security analyst.
And now the cybersecurity skills shortage fights with tool and data sprawl for scarce resources. Newly hired staff have yet to learn which tool does what, and how to correlate between different data sources. The effort of maintaining the complex extract-transform-load (ETL) toolchains that keep the normalized datasets in sync drags senior staff away from investigations.
Vendors like Komprise already attempt to automated the process with Komprise Smart Data Workflows augmenting existing data with a global index to help speed access. Microsoft’s plans for Azure Sentinel involve providing multiple views into a centralized data store to suit multiple ways of working. Yet normalizing data and storing it in one place tends not to happen.
“What we saw in practice, over and over—I talked to 40-some-odd, you know, Global 500s before starting Crogl—nobody’s data is normalized. Not even within a single data lake, let alone across multiple lake houses and data lakes and tools,” said Monzy Merza, co-founder and CEO of Crogl.
Cybersecurity startup Crogl has just been granted a US patent for its work on making disparate data sets easier to query and explore. Rather than unifying data in a single place, or normalizing the data into a standard format, Crogl attempts to mimic the way we humans link similar concepts together as we hunt for threats and breaches.
Crogl creates a kind of ‘synonym’ index that sits in front of existing systems and datasets, mapping similar concepts together and where they might be found. Picture a librarian who knows that when you say you’re looking for books about monsters, you might be interested in ghosts, zombies, banshees, Godzilla, Mary Shelley’s Frankenstein, the deep sea blobfish, or any number of dinosaurs. If you search for an IP address, you want to find everything that looks like an IP address, whether it is explicitly called an IP address or not.
“It’s just like a metadata layer, it’s a semantic layer on top, so we can map back to specific parts of the datasets, regardless of where those data sets are resident,” said Merza.
Part of the idea is to make finding correlations between datasets easier without having to move or copy the source data first. Not just the obvious, with rigorously curated datasets adhering to a fully normalized schema, but also the unusual, the new, the novel, and the weird. Especially when responding to an incident, we want to be able to quickly find anomalies within the sea of ordinary and expected data. We don’t want to miss something just because no one has had time to map a new log entry format to the core index schema.
There’s also the human factor. Busy senior analysts don’t always remember the subtle query semantics of every tool. In the heat of an incident, they may not remember that, oh yes, on this system it’s called user_email_src not client_src_email. Nor should they have to. The tools available to both experienced senior staff and recently hired juniors should help them navigate the semantic space with similar ease. Skill at investigating shouldn’t rely purely on memorizing an ever-increasing set of technical minutae about different products.
This is not a call to abandon efforts at standardization or experience with using technical tools; far from it. Instead, let us recognize the practical realities that confront us. Where standards like the Common Information Model or the Comment Event Format are useful, we should use them. When we have a choice to use a normalized field structure, we should choose to use it. Embracing common standards when we can helps everyone, especially our future selves.
But we should not give up on making progress merely because we do not have a perfect, fully-normalized logging infrastructure. We can meet existing systems where they are and work with them as best we can. By layering assistive tools on top, we can free up time to focus on standardizing where it helps us most. We can have a range of different tools at our disposal, each with its own niche of value, and also enjoy a few large-scale defaults that generally work well most of the time.
