The standard enterprise SIEM pulls safety log information from sources throughout the IT setting, then normalizes it, analyzes it and retains it. However as a result of SIEM suppliers usually cost extra to carry extra information, organizations usually should retain much less information than they would favor and settle for the constraints of subsequent analyses.
Moreover, SIEMs retain information in their very own, typically proprietary codecs. Actually, how SIEM distributors parse and normalize information is a method they differentiate themselves from rivals. Every seeks to make use of distinctive schemas, compression methods and specialised databases to enhance each outcome high quality and pace. Consequently, enterprises have restricted enter into how their information is ingested and digested, and proprietary parsing and codecs could make it tougher to vary distributors.
Some CISOs — discovering the constraints and trade-offs of information ingestion and retention in SIEM too constricting — are selecting to decouple their safety log information feeds from their SIEMs. By doing so, they usually acquire freer entry to the information, enhance management over retention timelines, enhance analytical capabilities, rein in SIEM prices and break freed from vendor lock-in. However decoupling information from the SIEM additionally has its challenges and requires important dedication, funding and planning.
How decoupling information from the SIEM works
To decouple safety information sources from the SIEM, safety groups insert techniques that they management in the course of these information flows. In follow, this implies establishing a separate, devoted information retailer to carry the safety log information, usually an information lake residing in a relatively cheap cloud storage service. It additionally means establishing a brand new information pipeline that takes in log information, preprocesses and normalizes it after which dumps it within the information lake. The enterprise then feeds its SIEM with information from the lake.
Advantages of decoupling SIEMs from information pipelines and storage
Establishing an impartial, enterprise-controlled information layer between the sources of safety log information and the purposes that devour it — e.g., SIEMs and different instruments equivalent to person and entity habits and analytics — allows the enterprise to do the next:
- Dictate the information schema for log data.
- Utterly management filtering of data and simply range it by vacation spot.
- Utterly management the retention horizons for each sort of information from every platform.
- Precisely and simply observe all safety information sources and all safety information customers.
- Simply implement constant adherence to institutional polices on information assortment and retention.
- Simply add new safety instruments that want entry to current information feeds.
- Simply change — and even drop — SaaS and SIEM distributors with out shedding information.
Buying and selling costlier SIEM-based storage for cheaper cloud bulk storage may even most likely cut back the price of storing safety information, per se. However — and that is necessary to know — that value discount may not lead to web financial savings, as new instruments or companies and workers time prices might overbalance these financial savings.
Challenges of decoupling SIEM from the information layer
After all, together with its advantages, decoupling information from SaaS or SIEM platforms additionally comes with challenges. These embrace the next:
- Designing a robust, safe, scalable and cost-efficient information lake and information pipeline, together with choosing applicable information change protocols and information storage schemata.
- Engineering a robust, safe, scalable and cost-efficient information lake and information pipeline, together with choosing instruments and companies with which to construct it and testing it adequately earlier than placing it into manufacturing.
- Migrating to the brand new structure with out information loss or interruptions in safety scanning.
- Working and supporting the information lake and pipeline effectively, together with making certain backups and continuity of service within the face of disruptions.
- Dealing with latency created by interposing the brand new layer — requiring consideration within the design, engineering and operations phases, in addition to steady monitoring to make sure latency is inside acceptable limits.
- Dealing with compliance, as the brand new information layer should respect and implement any relevant necessities — relying on firm sort, sector and geography — for information at relaxation and in movement.
A decoupling toolbox
CISOs creating a brand new enterprise safety information lake might want to decide their methods within the following areas.
SaaS information extraction
SaaS information extraction instruments could be in-built home utilizing SaaS APIs. Alternatively, third-party approaches embrace such proprietary SaaS safety posture administration platforms as Obsidian Safety, NetSkope SSPM and AppOmni, in addition to open supply instruments equivalent to Mondoo and OpenASPM.
Knowledge pipeline
The info pipeline is the ingestion and pre-processing instrument that receives uncooked logs and spits out data for the information lake in standardized format(s). Business merchandise right here embrace Cribl, DataDog and Splunk. Open supply choices embrace Vector, Logstash and Fluentd.
Knowledge storage
Most bigger organizations have already got expertise with information lakes, in addition to most well-liked distributors, equivalent to Snowflake and Google BigQuery, or open supply choices, equivalent to Apache HDFS or MinIO.
Enterprises even have to think about information codecs. Open requirements ought to be everybody’s first alternative: Open Cybersecurity Schema Format for the log data heading out to SIEMs or elsewhere, for instance, and storage codecs equivalent to Apache Parquet or Delta Lake for the information lake correct.
By decoupling cybersecurity information ingestion and retention from their SIEM platforms, CISOs can acquire management, flexibility and depth whereas probably lowering prices. However they should make investments important sources to seize these advantages.
John Burke is CTO and a analysis analyst at Nemertes Analysis. Burke joined Nemertes in 2005 with practically twenty years of know-how expertise. He has labored in any respect ranges of IT, together with as an end-user help specialist, programmer, system administrator, database specialist, community administrator, community architect and techniques architect.
