SIEM deployment unpacked

Rafal here, first post on SecOps Unpacked, nice to meet 👋

Over my career I’ve participated in and / or led ~30 SIEM onboarding projects, probably closer to 50 if we count individual deployments (1 client, multiple SIEMs type of deal).

Eventually I worked my way towards a SIEM onboarding approach which I will be sharing in this article. It is data driven, environment agnostic and works well for both MSSPs and in-house SOCs.

Let’s go.

Why can’t you just common sense your way to a solid SIEM deployment and, by extension, a decent security monitoring strategy? After all, you’ve (hopefully) worked with a SIEM before, you’ve seen the detections, queried the data. Why shouldn’t you just sit down with your team and try to just… figure it all out?

Good question.

For MSSPs that’s obviously a bad idea because you’d be turning each engagement into a bespoke one and you don’t want that.

For in-house SOCs it might work, but I don’t think it would be a long term solution. Ideally you’d want your security monitoring approach to outlast the current team.

I doubt you can common sense your way to a solid security monitoring strategy. Mostly because, we’re all biased whether we admit it or not.

What I saw working on my 30+ SIEM onboarding projects is that without a solid, data driven approach, the project plan defaults to the perspective of the most senior stakeholder. Roughly this:

STORY TIME START

I once worked on a SIEM onboarding project for two almost identical companies. Same industry, same size, same technology stack. Both ended up with completely different SIEMs. This was purely because the CISO of the first company used to be a sysadmin, while the SOC manager at the other had a purely networking background.

One SIEM ended up being heavy on endpoint data and detections; the other prioritized network visibility. To this day, I'm not sure which one was more effective at detecting the right threats.

STORY TIME END

There are other reasons too, but I don't want to go into depth or this article turns into a book. For example:

My thesis is that you can’t common sense your way to a solid security monitoring strategy. I believe you need an approach that’s data driven, reusable across different environment and ideally one that can be easily updated to account for new attack vectors.

Here it is:

SIEM onboarding or migration is a large project. And what do we do with large projects? We chunk them down into manageable pieces.

Here’s how I do it.

I chunk the SIEM onboarding project into three phases, each taking roughly 30 days. Once the project is complete, you (or your client) ends up with a fully functional threat detection system.

Obviously, threat detection and response content needs to be updated on a reasonably frequent cadence, but that is a story for another time. Let’s dig in:

Two of the most common questions I’d get when onboarding clients to a SIEM was:

Some would argue (I’m some) that answering those two questions solves 80% of your security monitoring strategy. The approach that worked the best for me, and one I’d like to instill in you, is this:

Threat intel informs you what detections you need. Knowing what detections you need, it's easy to decide what data to collect (at least the hot-storage portion of your data).

Let me repeat:

In this order. Here’s how to operationalize it.

First, you want to understand the monitored environment. If you're an MSSP, you can send a questionnaire to your clients. Figure out the number and types of systems, the existing security stack, maybe also regulatory commitments.

Then you want to prepare a threat landscape report. Nothing complicated:

Your goal is to list the top 5–10 MITRE techniques most likely to be used against you or your client. The next step is drafting detections for those techniques and understanding what data needs to be collected to support running those detections and the subsequent investigations.

Once this is done, you’re ready to move to the first phase of SIEM onboarding: centralize & learn.

The first phase of SIEM onboarding is straightforward. You grab alerts generated by your other threat detection tools and send all of them to your SIEM. And yes, I expect you to have a bunch of tools deployed before thinking about getting a SIEM. Otherwise, you might not need one just yet.

On top of that, a SIEM provides a single mechanism for fine-tuning by using KQL, SPL, or whatever other query language, and that makes exclusions easier to document and manage. If you were to fine-tune downstream, the way you'd do it would be different for every tool, and that requires specialized knowledge.

And finally, you get to learn the new tool as you work on those alerts. And by starting with centralizing alerts you get a chance to get familiar with that new environment while the workload is still “light”.

In your first 30 days you want to do two other things: start working on your list of crown jewels and lay down basic security monitoring processes.

We start discussing crown jewels early because organizations often don't have a clear vision of what theirs are. That gives them at least 60 days (phase 1 + phase 2) to come up with a list.

As far as the processes are concerned, you want to figure out at least three things:

That's it. Alerts are flowing in, fine-tuning is taking place, you've established basic processes. You can already start monitoring.

Now you're ready to build detections:

The goal of pre-onboarding was to build a list of techniques most likely to be used against you or your client. This phase focuses on creating detections that trigger when those techniques are observed in the environment.

To remind you of what we're going for:

We have threat intel, we’ve ideally drafted detection rules. We’re ready to start onboarding data to our SIEM.

Now, when sending data to a SIEM, we’re met with different data storage tiers. There’s typically some type of hot storage (readily available, also expensive) and cold-er storage (not immediately available, but cheaper). A simple rule to follow is that data that directly supports running detections and investigating them goes to hot storage. What you do with your cold storage is your business, good sir.

Detection engineering workflows can get complex, but if you're just starting out, I suggest you make sure your detections are tested and have a clear SOP. No need for an elaborate detection engineering pipeline at this stage.

The threat intel → detections process should be repeatable. You do it once during onboarding and repeat every quarter or so. This is mostly because the threat landscape continues to evolve, and the detections you've written today may not be the ones you need a year from now.

Lastly, having a structured detection engineering process makes sense, but in reality you will have to write a lot of ad-hoc rules based on ongoing incidents, threat hunts, investigation findings, or just the new attacks that seem to be popping up every other week these days. My approach is this: if there's an urgent need to write a detection, write it. If not, fall back on your detection backlog.

Very simple if you think about it.

Once alerts are in and custom detections have been created, you already have a solid SIEM in place. You could very well choose to finish the work here. Still, I think it makes sense to take the onboarding one step further and aim to protect your crown jewels.

This part can get difficult, but bear with me.

We focus on crown jewels last. This is very important because one of the more common SIEM onboarding anti-patterns is setting up monitoring for crown jewels too early.

By the time a threat actor reaches a crown jewel, they're typically late in the attack chain. If you focus on monitoring the most critical assets first, you decrease your ability to detect and stop the attack at earlier stages.

You need to know that companies rarely have a clear concept of crown jewels, and the effort to come up with a list often ends with the security team deciding on their own. Here's roughly the workflow:

However you arrive at the list of crown jewels, make sure to come up with a concise list. This means 3-5 systems, not 50. From that point on, the work seems relatively straightforward - collect data, write detections. And yet, you should be ready to face the following issues:

If data collection isn't possible, use crown jewels to add context to your existing detections. Automate alert prioritization so that anything related to crown jewels stands out to analysts. Even if you can't pull logs directly, automation can help highlight events that touch crown jewels one way or another.

That's it. You're mostly done. The next steps are the finishing touches:

Aaaand that’s the 3-phase SIEM onboarding approach, developed through plenty of trial and error. If there is one way you take from this article I'd like it to be the value of relying on threat intelligence to build a SIEM.

Feel free to steal this approach. And if parts of it are not clear - feel even more free to reach out to us on LinkedIn.

When should you start monitoring?

Most MSSPs I've worked with have a flat fee for SIEM setup and then a recurring fee for security monitoring services. Typically, security monitoring kicks in after the SIEM is fully stood up. This in essence means that you're not paid for monitoring until the SIEM onboarding finishes.

But what if the SIEM onboarding project runs longer than expected? I typically plan 90 days for a full end-to-end onboarding, but I've seen it stretch into years. Many reasons:

One MSSP I worked with had a nice way around this. They built it into their contracts and called it a hybrid-operations model. What this means is that the SOC team started monitoring as soon as alerts hit the SIEM, which happens in the first phase of onboarding. You don't have to wait for the project to finish to start getting paid.

It accomplishes two important things: it makes handover smoother because analysts have already been working in the SIEM for a couple of months, and it protects your organization against the project running longer than expected. This approach saved us a number of times.

Worth considering.