The AI SOC market is growing fast and there are products on it that are doing serious work. Some of them have strong integration capabilities, solid reasoning engines, and response actions that actually execute in production. The market has come a long way in four years.

But there is a problem with how we evaluate these products.

When every vendor says "AI-powered response," that phrase covers everything from a fully autonomous isolation workflow to a chatbot that suggests you maybe think about resetting a password. Both get the same label in the marketing material. Both show up in the same analyst reports. And when a security team sits down to compare three products, they have no standardized way to measure the gap between "our AI handles response" and what that actually means in operational terms.

Some products are close to real autonomy in specific domains. Some are strong in analysis but thin on execution. Some have broad coverage but almost nothing runs without human approval. These are all valid positions on a maturity spectrum. The problem is that there is no shared framework to place them on that spectrum consistently.

So we built one.

We call it ARMM. And yes, the name is intentional.

A decade ago, the SOAR generation solved half the problem. We built the arms. Playbooks, integrations, automated response workflows. The execution layer was there. What was missing was the brain. Every decision tree was hand-coded. Every branching logic was written by an engineer who had to anticipate every possible scenario. The arms moved, but only along rails that humans laid down manually. When the scenario deviated from the playbook, the arm froze.

Now the AI SOC generation has solved the other half. We built the brain. LLMs reason across alerts, correlate context, analyze logs, and make judgment calls that no static playbook could replicate. But somewhere along the way, a lot of products forgot to attach the arms. The reasoning is strong. The analysis is sharp. And then it hands you a summary and says "here is what you should probably do." The brain thinks. The arm does not move.

ARMM evaluates both. The reasoning quality, the decision-making maturity, the trust you can place in the AI's judgment. And the response capability, the execution depth, the ability to actually take action without three humans supervising. It weighs the arm heavier because that is where the industry gap is widest right now. But it does not ignore the brain, because an arm without a brain is just a SOAR playbook and we already know how that story ended.

ARMM is a structured scoring system for evaluating what an AI SOC solution can actually do in the response layer. It covers 80+ response capabilities across six domains: Identity, Network, Endpoint, Cloud, SaaS, and General Options. And it provides a common language so that when someone says "we handle response," there is a way to ask: at what level, across how many actions, and with what degree of autonomy?

The CyberSec Automation Blog has published over a dozen articles and podcast episodes covering what makes a good automation program succeed, how to evaluate tools, and how to structure decision-making around security automation purchases. We have built tool comparison lists, evaluation checklists, and decision frameworks. ARMM is the next step in that work.

Most existing evaluation methods for AI SOC solutions are either vendor-produced (and therefore biased toward their own capabilities) or too generic to capture the specific nuances of AI-driven response. Analyst reports compare products at a feature-list level without measuring automation depth. Vendor demos show best-case scenarios without exposing the operational friction underneath.

Our focus is narrow and deliberate: response capabilities. Most AI SOC solutions already deliver strong reporting and analysis features. They can summarize alerts, correlate indicators, and reduce false negatives in a mature environment (we emphasize mature because these solutions need access to quality logs and, in more advanced implementations, to organizational documentation and environment-specific context). Where the industry needs structured evaluation is in the response layer: the actions an AI SOC solution can take, how autonomously it can take them, and under what conditions.

We acknowledge that some of the capabilities listed in this framework may seem aspirational at this stage. That is by design. The framework is intended to serve both as a current-state evaluation tool and as a forward-looking roadmap.

We are not scoring specific vendors. The goal is to establish a shared methodology that allows security teams to answer questions such as:

For product managers working on AI SOC products, the framework serves as a competitive analysis baseline:

Because this is a fast-moving space, we are starting at version 0.1. This is a living document. Version 1.0 will be designated when the framework reaches a level of stability and community validation that warrants it.

ARMM supports two distinct approaches to scoring, each designed for a different operational question.

Evaluator Mode is the straightforward path. You score each capability on the 0-1-2 scale described above (with the 1C, 1G, 1A sub-levels) and the framework calculates your coverage rate, automation depth, and per-plane breakdown. The tier placements come from ARMM's reference tables. You do not need to factor in your organizational context. This mode answers one question: given two or more AI SOC products, which one covers more of what I need and at what automation level? It is built for procurement teams, SOC managers running vendor evaluations, and anyone who needs a side-by-side comparison without spending weeks on it.

Builder Mode adds a second scoring layer on top. Instead of relying on fixed reference tiers, you score each action across three axes: Trust (how much confidence does your implementation warrant), Complexity (how hard is it for your specific team to build and maintain), and Impact (what is the blast radius if something goes wrong). The action score becomes T + C + I, and the tier placement shifts based on your organizational reality. The same action that scores Entry for a mature team with established automation pipelines might score Explorer for a team that is deploying its first AI SOC integration. This mode answers a different question: given my team, my environment, and my risk tolerance, where should I invest engineering effort to move up the maturity ladder? It is built for product managers, engineering leads, and internal SOC teams running their own automation programs.

Both modes evaluate the same six planes and the same 80+ response capabilities. Both produce per-plane breakdowns and a composite maturity label. The difference is whether you want a product-level comparison (Evaluator) or an environment-aware implementation roadmap (Builder). The public ARMM app at armm.secops-unpacked.ai supports both.

Each response capability in the framework is scored on a three-level scale that measures the degree of automation available:

0 (Not Available): The feature does not exist in the product. There is no mechanism, manual or automated, to perform this action through the AI SOC solution.

1 (Available with Human Involvement): The feature exists but requires some form of human interaction before execution. Because human involvement can range from full collaboration to a simple approval click, this level is subdivided into three sub-categories:

2 (Fully Automated): The action is performed without any human involvement. The vendor (or internal implementation) has demonstrated that the AI SOC solution can execute this action with sufficient confidence that no human review is required. At the time of writing, level 2 is exceptionally rare for most response categories. The framework includes it to establish the target state and to differentiate products that are moving in that direction from those that are not.

In Builder Mode, each response action is evaluated across three dimensions:

This axis measures the confidence level warranted by the AI SOC implementation. It correlates directly with implementation quality: reasoning log depth, context-aware decision-making, and guardrails against hallucination.

This axis evaluates the technical friction in building and sustaining the automation, relative to the skills and resources of the team responsible for it. This is deliberately team-dependent. An automation rated C = 3 for a junior team may be C = 2 for a team of specialized AI engineers with established CI/CD pipelines for their playbooks.

This axis captures the business risk associated with the action. It is typically the most stable axis across organizations, but shifts based on asset criticality. Isolating a standard employee laptop has a different blast radius than isolating a production database server.

The scoring system builds from individual actions up to a full program assessment through five layers. Each layer uses a defined formula.

For a single response action, the score is the sum of its three axis values:

The minimum possible score is 3 (T=1, C=1, I=1). The maximum is 9 (T=3, C=3, I=3).

The action score maps to one of four maturity tiers:

The maturity score for a specific domain (e.g., Endpoint, Identity) is the arithmetic mean of all action scores within that domain:

Where n is the number of scored actions in the domain. The resulting D value maps to a tier using the same thresholds from Layer 2.

The overall program score is the arithmetic mean of all domain scores, with equal weighting across all six planes:

Equal plane weighting is a deliberate design choice. It prevents planes with more actions (Endpoint has 22, SaaS has 10) from dominating the evaluation. Each plane contributes exactly one-sixth of the overall score.

The composite label is not derived from the program score directly. It uses sequential gating logic:

The composite label equals the highest tier where at least four out of six planes independently meet that tier's threshold, and the qualification chain is unbroken from Explorer upward. A product cannot be labeled Advanced if it has gaps at the Explorer tier.

The four-out-of-six rule is intentionally forgiving. A product focused on cloud-native environments may legitimately deprioritize network-level response. That should not disqualify it from a meaningful composite label. But it still needs breadth across most planes to earn a higher tier.

The ARMM recognizes that the maturity level of an automated action is not a static property of the feature itself. It is an emergent property of the environment where it is applied. The three axes (T, C, I) are all subject to organizational variance, which means the same product capability produces different scores in different contexts.

Example: "Isolate Device" evaluated by three different organizations using the same AI SOC product:

The product capability is identical across all three. The scores differ because the Trust axis reflects implementation maturity, the Complexity axis reflects team capability, and the Impact axis (while stable here) can shift based on asset criticality. A vendor benchmark alone is insufficient. Builder Mode exists specifically to capture this variance.

The framework organizes response capabilities into six domains. The first five (Identity, Network, Endpoint, Cloud, SaaS) cover specific technical response planes. The sixth (General Options / Usability) covers platform-level characteristics that affect the operational quality of the solution independent of any specific response action.

For the first five domains, each capability is scored using the 0-1-2 system described in Section 3.1 (Evaluator Mode) or the T+C+I system described in Section 3.3 (Builder Mode). For the General Options domain, the scoring criteria shift slightly: 0 means the feature is not available, 1 means the feature is available but limited in capability or partially implemented, and 2 means the feature is fully available, functional, and tested.

Identity-related response actions target user accounts, service principals, groups, and access permissions. These actions are among the most commonly needed in incident response and are often the first automation candidates for SOC teams.

Builder Mode Reference Scoring (Mature AI SOC Program, Skilled Engineering Team):

Network-level response actions modify traffic flow, access control, and device connectivity. These are often high-impact actions with significant blast radius, making the Trust and Impact axes particularly important in scoring.

Builder Mode Reference Scoring:

Endpoint response actions operate directly on devices and their software environment. This domain has the largest number of capabilities because endpoint response spans file operations, process management, application control, forensics, and OS-level changes.

Builder Mode Reference Scoring:

Cloud response actions target infrastructure resources, access controls, and storage in cloud environments. The blast radius of cloud actions can be particularly severe because a single misconfigured change can affect multiple dependent services.

Builder Mode Reference Scoring:

SaaS response actions focus primarily on email and productivity platforms, which are among the most common attack surfaces in enterprise environments. Actions in this domain directly affect end-user workflows and communications.

Builder Mode Reference Scoring:

This domain evaluates platform-level capabilities that are not tied to any specific response action but directly affect how useful, trustworthy, and manageable the AI SOC solution is in production. The scoring for this domain uses a modified scale: 0 means not available, 1 means available but limited, and 2 means fully available and functional.

This domain is split into two sub-categories to distinguish between operational platform features and AI-specific evaluation criteria.

Evaluating each plane individually is necessary but not sufficient. Security teams making purchasing decisions and product managers tracking competitive positioning need a consolidated view that communicates the overall picture without hiding the details.

This is the most operationally significant metric and the one that separates real autonomous solutions from products that wrapped a chatbot interface around a set of API calls.

Across all covered capabilities, calculate the distribution:

A product could have 80% of capabilities covered but only 5% fully automated. That is a fundamentally different product than one with 60% covered but 40% fully automated. The first is broad but shallow. The second is narrower but operates with real autonomy where it counts.

Full Automation Rate: The percentage of total capabilities at level 2. This is the true measure of how much an AI SOC solution can operate without human intervention.

Coverage Rate: The percentage of total capabilities at any level above 0. This measures breadth regardless of automation depth.

The relationship between these two numbers tells you everything about how the product actually operates. A high coverage rate with a low automation rate means the product is a guided workflow tool with AI branding. A moderate coverage rate with a high automation rate relative to coverage means the product is autonomous in its areas of focus but limited in scope.

A complete ARMM evaluation for a product produces the following consolidated output:

Per-Plane Breakdown:

A product can reach Expert level on a specific plane by checking all the boxes for that domain. But it would be difficult to consider an AI SOC Response solution as Expert level overall if it lacks the ability to perform foundational actions like closing alerts in a SIEM. The tier system is designed to reward both depth within a domain and breadth across domains.

The reference maturity tables provided in Section 4 use example scores from a hypothetical mature AI SOC program with a skilled engineering team. These are illustrative, not universal benchmarks. The environmental dynamics described in Section 3.5 are not optional context; they are a core part of how the framework is intended to be used.

When comparing two products, the most informative comparison is not the aggregate score. It is the per-plane breakdown combined with the Automation Depth Score. Two products at the same composite tier can have radically different operational profiles. One may cover 80% of capabilities at the Collaborator level. The other may cover 50% but with 30% at full automation. These are different products for different buyers with different operational maturity levels.

This is version 0.1. The framework has known limitations:

We are building a public web application where users can input their product's capabilities and generate ARMM scoring layers automatically, along with an exportable CSV. The application is available at: armm.secops-unpacked.ai

The AI SOC market is growing faster than the industry's ability to evaluate products on consistent terms. The ARMM framework provides a structured, repeatable methodology for measuring what an AI SOC solution can actually do in the response layer, how autonomously it can do it, and what it takes to deploy and maintain that capability in a specific operational environment.

The framework is built for two audiences: security teams evaluating products and product managers building them. For security teams, it provides a checklist and scoring system that cuts through marketing language and focuses on operational capability. For product teams, it provides a competitive analysis baseline and a prioritization framework for feature development.

SOAR gave us arms without brains. The first wave of AI SOC products gave us brains without arms. The products that will win this market are the ones that connect both. ARMM gives you a way to measure how far along that connection is, and where the gaps remain.

No current AI SOC solution will check every box. That is not the point. The point is to establish a common language and a common measurement system so that the conversation about AI SOC response capability is grounded in specifics rather than promises. Version 0.1 is the starting point. The framework will evolve as the market does.