Introducing ASEF

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

AI SOC / Agentic SOC is everywhere - The phrase currently covers everything from fully autonomous investigation pipeline to a chatbot that suggests you maybe reset a password. Same label, same analyst category, same booth. And when a security team sits down to compare three of these products, there is no shared way to measure the distance between them.

Back in February we published ARMM (AI Response Maturity Model), a maturity model for AI SOC response capabilities. It did its job, and it stopped at response on purpose. The limitations section said it plainly: detection and analysis are out of scope.

The thing is not many buy just response product. They buy an AI SOC platform, and that platform it should touch everything (here we might be going into the ISOC term). The data pipeline, the detections, the investigation, the verdict, the action, and the loop that feeds lessons back into better detections.

So we extended it. ASEF, the AI SOC Evaluation Framework, is a vendor-neutral framework for evaluating AI SOC platforms across the whole security operations lifecycle: data ingestion, detection, investigation, and response, plus a cross-cutting Platform and Trust layer. That is the one-line version. Let me walk you through the rest.

Vendor Updates Section !

Here is what actually happened over the last two years. Everyone built the middle. Triage and investigation were the easier win, so that is where the products piled up. The demos are impressive. Alerts get worked fast. And both ends stay thin. Very few products can improve your detections. Very few can execute a remediation without three humans watching. At SecOps Unpacked we track this market closely, and you can see the clustering across the whole vendor landscape. The market optimized for the part that demos well.

We are not the only ones seeing it. Anton Chuvakin and Oliver Rochford just published When Marketing Fails, a paper on the gap between AI SOC vendor claims and what practitioners actually experience. Their finding, from 30 plus vendor briefings: most claims describe a future state sold in the present tense, and when reality disagrees, the product immaturity gets reframed as a buyer readiness problem. Their fix is to separate assistive from autonomous capabilities, measure adoption depth instead of feature enablement, and treat trust as something a system earns.

ASEF has a mechanism for each one. The autonomy scale separates assistive from autonomous, per capability. Automation depth measures depth, not enablement. And the trust metrics live in the ROI panel, not on the slide.

ASEF is a funnel. Five stages, and each one does exactly one job.

Screen. Narrow the market to a shortlist using your scope and your hard requirements. Requirements are binary. They filter, they never get scored. The idea is that if you want platform that does only Detection Engineering, you don’t need to evaluate the one that does only data pipeline. 

Score. Score the survivors capability by capability, per zone, on an autonomy scale.

Platform. Score the cross-cutting Platform and Trust layer separately. It never blends into the headline.

ROI. Track operational metrics as deltas against your own baseline.

Decide. Read it all together against thresholds you set before the demo.

The stages stay separate on purpose. Most bad evaluations happen when these jobs collapse into each other. A requirement becomes a score. A platform gap gets averaged away by a strong feature. A single blended number hides the weak zone. ASEF refuses all three.

Everything hangs on the Shift Map.

Data ingestion and processing. Can it onboard sources, parse, normalize, and tell you where your data gaps are.

Detection engineering and SecOps resilience. Can it author detections, map coverage to ATT&CK, run a proactive hunt off a hypothesis and turn what it finds into a detection, tune the noise, and manage detection as code.

Investigation and Triage. Can it build context, enrich, correlate, scope, and land on a verdict you can defend. Can it run a reactive hunt off an indicator, sweeping the environment for the same activity. And when the verdict is real, can it go deeper. Memory analysis, artifacts, root cause, evidence handling. Triage depth, hunt depth, and DFIR depth are not the same thing, and this zone scores all three.

Response, Remediation, and Feedback loop. Can it act, how autonomously, and does what it learned flow back into better detections. This zone is ARMM, all five planes, plus the feedback loop.

Under all of it sits Platform and Trust. Audit trail, reasoning logs, RBAC, governance, model handling. It gets its own scale and its own card, and a platform score below 50 percent raises a risk flag that no feature strength can clear.

There are 126 capabilities in the seed set, and the whole thing is data-driven. A new capability is one data entry, not a code change.

Every lifecycle capability gets scored on the autonomy scale we carried over from ARMM.

0 means no capability. 1C means the AI collaborates and the analyst does the work. 1G means the AI lays out options and the analyst picks. 1A means the AI prepared the action and waits for a human to approve it. 2 means it runs end to end, no human in the loop.

Level 2 is still rare, and that is fine. The scale exists to show the distance between the marketing and the autonomy, and to show which products are actually moving.

Two readouts fall out of this, and you read them together. Coverage, the share of capabilities above zero. And automation depth, the distribution across the levels. High coverage with low automation is a guided workflow tool with AI branding. Moderate coverage with real autonomy where it counts is a different product for a different buyer.

Builder mode scores the same capability for your reality, across three axes. Trust, how much confidence the implementation deserves. Complexity, how hard it is for your team to build and run. Impact, the blast radius if it goes wrong. Add them up, 3 to 9, and the score maps to a tier.

The same capability lands at a different tier for a mature team than for a junior one. The capability is identical. The context is not. A vendor benchmark alone was never going to capture that, and that is exactly why Builder mode exists.

ASEF does not produce one blended score, because one blended score is exactly how a Middle-only product gets to call itself mature.

The headline is a profile across the zones you chose to evaluate. The composite label is gated. A product earns a full label only when every in-scope zone clears the tier and a feedback loop exists. A product that is Expert at investigation and empty on both ends gets labeled Middle-heavy. On purpose. That label is the whole point of this framework turned into a score.

Scope matters too. You pick which zones you are evaluating at the start. A team that handles detection elsewhere and only needs investigation and response does not get marked down for skipping the left side. The report opens by stating the scope, and out-of-scope zones read as not evaluated, never as gaps.

And scope changes the reading, not just what shows. Evaluate all four zones and you get the fullness score, one composite across the lifecycle. Evaluate one or two and you get each zone scored on its own tier, in depth, down to its subdomains. No composite, no blend, no penalty for what you did not ask about. If you are buying an investigation tool, you get an investigation score, not a lifecycle grade that docks it for lacking response. Zones never blend, because blending two of them into one number recreates the same hiding problem one level down. The Platform and Trust layer is scored either way, because it applies to any platform you buy.

The screening stage has one rule I care about more than any other. A vendor fact is either present, or it is unknown. Unknown never quietly becomes a pass or a fail.

Every vendor in the screen lands in one of three states. Passes, on known facts. Excluded, because a known fact fails a hard requirement, hidden behind a deliberate reveal and labeled with the reason. Or unknown, meaning a required fact is missing, shown by default and flagged as verify in a proof of concept.

Most comparison spreadsheets quietly treat missing information as either fine or disqualifying, depending on who built the spreadsheet. Both are wrong. Excluded fails a known fact. Unknown might pass once you verify it. Keeping those two apart is the integrity of the whole gate.

And the same honesty applies to the AI itself. Chuvakin and Rochford close their paper with the question every buyer should push a vendor on: can the system admit it does not know, and what happens when it does. ASEF scores that directly. Inconclusive verdict support is a capability in the Investigation zone, because a system that forces a binary call on thin evidence is not confident, it is reckless. Override telemetry and published failure modes are capabilities in Platform and Trust. A vendor that treats analyst overrides as user error and documents no limitations does not have a feedback loop. It has a narrative.

The last layer is PICERL, 15 metrics across the six phases of the SANS incident response lifecycle: Preparation, Identification, Containment, Eradication, Recovery, and Learning. The phases are SANS's. The index built on them is ours.

You record a baseline before the proof of concept and track deltas against it. No baseline, no evaluation. And there is deliberately no single ROI number. It is a panel of deltas, so strong movement on one phase cannot hide a regression on another.

The metrics that matter most are not the speed ones. Auto-close reversal rate. Escalation accuracy. Model drift. Whether analyst corrections actually feed back into the system. Closing alerts faster is sweeping the floor faster. The question that matters is whether the SOC is getting smarter.

No vendor scores. The directory holds facts, and any editorial reference is clearly marked context, never the verdict. Your scores are your own.

No pricing. Because that is changing too often, it is up to you to take this within your org and compare what you get for the price that you pay for the product. 

No integration coverage module yet. Integrations are breadth, not autonomy, and they deserve their own percentage-based score. It is on the roadmap. Until then, integration quality gets tested where it should be, in the proof of concept.

No environment tailoring yet. A future version will let you pick your log sources and your top use-cases, scope the capability set to your reality, and surface your telemetry gaps. The design exists. The module does not.

The framework reference and the interactive guide are live at secops-unpacked.ai/asef/guide. Screen the market, score a product, export the results. ARMM stays a standalone model at armm.secops-unpacked.ai and now also lives inside ASEF as the Response zone.

Thanks to Andrei Cotaie and Cristian Miron. Their work on ARMM is the foundation this whole thing stands on.

ASEF is data-driven by design, so contributing is a small edit. A capability, a metric, a vendor fact correction. Send it through the repository or the site contact form. Every contribution says what to measure, never how good any one vendor is at it.

No current product will score well across the whole map, and that is not the point. The point is a common language for what "AI SOC" actually means, so the next conversation you have with a vendor is grounded in capabilities and autonomy instead of promises. The framework will move as the market does.