Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.
Memory makes GenAI useful in the SOC and also makes it opinionated. That opinion can drift. Some stacks become optimistic and default to benign. Others become pessimistic and default to malicious. Both are forms of bias. You can control this with explicit objectives, memory hygiene, calibration, and drift monitoring. Citations and a practical checklist are below.
Table of Contents
The observation
In a clean world, AI investigations end in one of three outcomes: benign, suspicious, or malicious. In the field, once you add memory, the model starts anchoring to prior cases and narratives. Over time, I see two failure modes:
Optimistic drift: the AI looks for evidence that activity is benign and closes too fast.
Pessimistic drift: the AI assumes breach and marks too many items as malicious, flooding triage.
Both are biased patterns reinforced by memory. This is not “AI gone wrong.” It is a predictable effect of feedback, sampling, and incentives. Anchoring, confirmation, and availability biases show up in ML pipelines just as they do in humans.
My take: I prefer a pessimistic default in security. Assume compromise and plan for worst case, then reduce noise with controls rather than miss a critical event.
Do you enjoy CyberSec Automation Blog Content?
Where the bias actually comes from
Anchoring via memory
Retrieval of previous conclusions, tickets, and notes can anchor the model’s current reasoning. If memory stores verdicts, not only facts, you increase confirmation bias. NIST and ACM both call out bias as socio-technical, not just model-level.
Label/feedback loops
If analysts reward “fast closes,” your reinforcement signal pushes optimism. If you reward “catch anything suspicious,” you push pessimism.
Dataset shift and model drift
Your traffic, tools, and attacker mix change. That is covariate shift and concept drift. You must detect and respond to it. Practical detectors include PSI for distribution shift and streaming detectors like ADWIN.
Calibration decay
Confidence scores stop matching reality as the environment changes. Track Expected Calibration Error and similar metrics to keep “probability of malicious” honest.
Domain specifics of cyber
In cybersecurity, bias does not only waste time. It can either hide attacks (optimistic) or burn out analysts and suppress signal (pessimistic). Industry pieces echo this tension in practice.
Monitoring Drift in AI SOCs
Two practical techniques you’ll hear about in ML Ops — and that are directly useful in AI SOC — are PSI and ADWIN. Both are ways of spotting when your model has started to “see the world differently” than it did at training or deployment time.
Population Stability Index (PSI)
PSI is a simple way to measure whether the distribution of features (inputs the model uses) has shifted.
For example: imagine your model relies heavily on login geolocation or file hash rarity. If the frequency distribution of those features changes a lot compared to your baseline (say, suddenly 30% of logins are from a new region), your model is now making predictions on data it hasn’t really been trained for.
We typically set thresholds:
PSI < 0.1 → stable (no real change)
0.1–0.2 → moderate shift (keep an eye on it)
>0.2 → warning level, drift may be affecting results
>0.25 → action required (retrain or re-evaluate)
In the SOC, you’d use PSI on high-value features like source IP reputation scores, authentication method, or endpoint process ancestry — the signals that drive most of your verdicts.
ADWIN (Adaptive Windowing)
ADWIN is a streaming drift detector. It looks at incoming telemetry in real time and detects if the statistical properties of the data have changed.
Think of it as a moving window: if the recent data looks very different from the older data, ADWIN flags drift.
Example in a SOC:
You’re monitoring failed login attempts per user per hour. Normally the distribution is steady. Suddenly, in the last hour, the rate jumps in a way that doesn’t fit past behavior. ADWIN detects this as a distribution change — signaling that the model’s prior assumptions may no longer hold.
ADWIN is valuable when your SOC ingests continuous, fast-changing data (auth logs, endpoint events, netflow).
Why this matters in practice
PSI tells you when your AI is drifting because the population of data has shifted.
ADWIN tells you when your AI is drifting because the stream of data is behaving differently in real time.
Both should trigger drift alerts in your SOC platform. When thresholds are hit, you either retrain the model, adjust decision thresholds, or at least shadow-test the model to confirm it’s still making reliable calls.
Why AI SOC Drift Is Different From Traditional Model Drift
There’s something specific to cybersecurity worth calling out.
When you roll out an AI SOC platform, you’ll either:
Feed it historical data (if the platform supports it), or
Let it start fresh, learning in shadow mode from day one of deployment.
In the traditional SOC workflow, when a human analyst processes an alert, the output is often a one-liner:
“False positive because xyz.”
“Normal activity, change request attached.”
That limited context doesn’t feed the model much bias. It tells the system how the case ended but doesn’t provide a lot of rich narrative that can anchor future decisions.
But with AI handling investigations, the story changes. Instead of one-liners, you get one, two, sometimes three paragraphs of reasoning explaining why an alert was closed as benign, suspicious, or malicious.
And here’s the kicker: most AI SOC platforms are built so that analysts approve or deny those AI conclusions. That means the model isn’t just learning the verdict — it’s learning from the entire summary and narrative it generated.
The result?
The more alerts you approve with “benign” narratives, the more the model drifts optimistic.
The more you approve “malicious” narratives, the more it drifts pessimistic.
And beyond optimism/pessimism, the model starts encoding other forms of bias hidden in the text — anchoring to particular arguments, data sources, or analyst preferences.
This makes AI SOC drift qualitatively different from classical ML drift. You’re not just feeding it labels. You’re feeding it reinforced narratives — which are far more context-rich and far more prone to anchoring.
How to Control Optimism vs Pessimism on Purpose
1. State your loss function
Every SOC has to decide which mistake is more costly:
False Negative (FN) → Missing an intrusion.
False Positive (FP) → Investigating noise.
In most environments, FN > FP — a breach is worse than wasted cycles. But not all teams weigh it the same:
A resource-constrained SOC may set stricter limits on FPs to avoid burnout.
A high-risk sector (finance, healthcare) will tolerate more noise to minimize missed threats.
The key is to make this explicit. Don’t let the model’s default bias define your risk appetite. Encode the loss function into:
Thresholds: e.g., “AI must be 85% confident to close as benign, but only 60% confident to escalate as malicious.”
Routing rules: e.g., suspicious verdicts always go to Tier 1, but any high-risk suspicious (based on threat intel correlation) goes straight to IR.
Treat this as SOC policy, not a hidden “prompt hack.”
2. Separate facts from verdicts in memory
One of the biggest sources of anchoring bias is how memory retrieves past cases.
If the AI sees a verdict like “benign – normal user behavior” in memory, it may bias its current conclusion toward benign.
If memory only retrieves facts (e.g., “User X logged in from IP Y at 03:00, MFA success”), the AI can evaluate evidence without inheriting the past verdict.
Practical controls:
Store artifacts, signals, and context → log snippets, process trees, enrichment.
Down-weight final labels → allow retrieval of verdicts, but apply less importance than raw evidence.
Force “evidence-first” prompts → e.g., “Summarize the facts before giving a conclusion.”
This keeps memory as a knowledge base, not a verdict repeater.
3. Enforce memory hygiene
Think of memory like a SIEM data lake — garbage in, garbage out. Without hygiene, bias compounds.
TTL (Time to Live): Don’t let outdated conclusions anchor current cases. E.g., verdicts expire after 30 days unless reaffirmed.
Source tags & provenance: Every memory chunk should record its origin — log type, analyst name, AI agent version. This makes retrieval explainable.
Retrieval filters: Prefer multi-source evidence. For example, if two independent log sources (e.g., auth + EDR) align, rank that memory higher than a single noisy source.
This reduces toxic bias accumulation and prevents “zombie verdicts” from skewing current reasoning.
4. Use structured, two-pass reasoning
Humans avoid confirmation bias by debating, and AI needs the same. A two-pass system creates an internal “red team.”
Pass A: Build a case file of observations and hypotheses. E.g., “Unusual PowerShell execution observed, correlated with new registry keys.”
Pass B: Adversarial review. AI (or a secondary agent) argues the opposite verdict. E.g., “This could also be normal IT admin activity — prior change tickets show similar actions.”
The final verdict must resolve both arguments. This mirrors human analyst workflows (peer review, escalation) and makes conclusions more balanced.
5. Calibrate regularly
Raw confidence scores are almost always misleading. Calibration ensures “80% confidence” really means “8 out of 10 times correct.”
Metrics:
ECE (Expected Calibration Error): measures mismatch between predicted vs actual accuracy.
Brier score: penalizes both overconfidence and underconfidence.
Operationalization:
Re-tune thresholds if calibration drifts.
Publish a monthly calibration report to SOC leadership showing whether the AI’s confidence is still trustworthy.
Well-calibrated AI allows you to set rational escalation policies instead of “gut feel” thresholds.
6. Monitor drift continuously
Data never stays static. Model drift is inevitable. The only question is whether you catch it.
PSI (Population Stability Index):
Compares historical feature distributions vs live data.
E.g., login location mix changes drastically.
Thresholds: >0.2 = warning, >0.25 = action.
ADWIN (Adaptive Windowing):
Monitors real-time streams for sudden changes.
E.g., spike in failed logins per user compared to historical patterns.
Response:
Trigger shadow evaluation, partial retraining, or force suspicious verdicts when drift is detected.
Automate drift alerts into the SOC dashboard (same way we alert on log ingestion failures).
7. Test with counterfactuals and red teams
SOC AI should be tested like detections: continuously and adversarially.
Counterfactuals: Hold out known attack scenarios and near-miss benigns. Evaluate AI weekly against this set.
Rotation: Refresh test data monthly to avoid overfitting.
Red teaming: Actively simulate edge cases — “what if MFA fails but user behavior is normal?” — to stress-test reasoning.
NIST AI RMF highlights this: continuous evaluation tied to business harm, not just technical accuracy.
What to ask your vendor
How do you prevent anchoring from prior verdicts when using memory or case history?
Do you provide calibration reports and can we export ECE or similar?
What drift detectors are built-in (PSI, ADWIN, custom) and how are alerts surfaced?
Can we tune the loss function or cost ratios for FN vs FP by use case?
Do you support two-agent review or adversarial reasoning before final verdict?
How do you version prompts, memories, and model configs so we can audit changes?
Show us your approach aligned with NIST AI RMF controls for measurement and monitoring.
What we know from the literature and industry
Bias can enter at data, model, and deployment stages. You must treat this as a socio-technical problem, not just tuning a model.
Cybersecurity is already seeing bias outcomes that either hide threats or inflate noise. Leaders are concerned and are asking for measurable controls.
Drift is normal in live systems. Use PSI for distribution monitoring and ADWIN for streaming change detection. Build playbooks that trigger re-evaluation when these fire.
Confidence must be calibrated if you expect analysts to trust scores. Track ECE and retrain or re-threshold when it degrades.
References
ACM: Biases in AI Systems
Interface Media: Exploring the Impact of AI Bias on Cybersecurity
Chief Executive: Why AI Bias Is a Growing Cybersecurity Concern
NIST AI Risk Management Framework (AI RMF 1.0)
Concept Drift Detection in Data Streams (ADWIN, Gama et al.)
Measuring Calibration in Deep Learning (Guo et al., 2017)
Population Stability Index (PSI) for Monitoring Model Drift (Practical Guide)
NIST: Bias in Artificial Intelligence
Closing view
I still prefer a pessimistic default in security. Assume breach. Pay the cost of extra triage while you mature memory, calibration, and drift controls. You can dial back noise with evidence requirements and better calibration. You cannot easily recover from a missed intrusion.
🏷️ Blog Sponsorship
Want to sponsor a future edition of the Cybersecurity Automation Blog? Reach out to start the conversation. 🤝
🗓️ Request a Services Call
If you want to get on a call and have a discussion about security automation, you can book some time here
Join as a top supporter of our blog to get special access to the latest content and help keep our community going.
As an added benefit, each Ultimate Supporter will receive a link to the editable versions of the visuals used in our blog posts. This exclusive access allows you to customize and utilize these resources for your own projects and presentations.