The danger of too many alerts

The theoretical threat of prompt injection (hidden instructions in email content manipulating an AI assistant) sounds alarming. But well-designed systems mitigate this through system-level safeguards that override any instructions embedded in email body content. A proper implementation ensures that email text or content within an email cannot override the system’s core directives. This is table stakes for any security-critical AI system.

The real threat may be far more practical and harder to defend against: context poisoning. Attackers could deliberately introduce activity patterns designed to trigger false security alerts. Not one false alert, but hundreds over days and weeks. Suspicious login attempts from unusual locations. File transfers that match ransomware signatures. Network connections flagged as anomalous. None of it necessarily represents actual malicious activity.

All of it could be noise carefully crafted to appear threatening. Are they ready for every emergency? Here’s how such an attack could unfold. An attacker systematically triggers alerts across multiple detection systems. Security teams investigate. They find nothing. They investigate the next day when another wave of alerts appears. Again, nothing. And the next day. The pattern continues until security teams stop responding to alerts from those systems.

They’ve lost confidence. The boy-who-cried-wolf scenario has been engineered deliberately. And this is when genuine attacks could succeed. When real credential theft occurs, when real data exfiltration occurs, when ransomware actually executes, administrators may see real alerts come in with a flood of false positives. Security teams, conditioned by weeks of investigating phantom threats, may deprioritise or overlook these alerts.

The attack could then proceed without timely detection. Unlike prompt injection, context poisoning does not necessarily depend on a weakness in the AI system's architecture. It can exploit the environment in which the system operates, even when the system is functioning as designed. The system is correctly identifying suspicious patterns. But the patterns are intentional noise injected by attackers.

The AI system becomes a liability because humans no longer trust its output. CISOs are caught in a difficult position. They need AI to process security event volumes no human team can manually analyse. But that dependency creates an attack surface. Organisations are now faced with a choice: to be skeptical of AI-generated alerts (which may diminish the benefits of using AI) or to trust AI-generated alerts without proper validation (leaving the organisation vulnerable to context poisoning).

Everyone else wants the AI engine The answer is to understand that AI detection systems are critical infrastructure that need to be protected. Organisations need to monitor their monitoring systems for alert patterns that are different from the baseline. They need cross-validation mechanisms to verify AI-induced risks against independent sources.Most importantly, CISOs need to maintain human judgement in the security decision loop.

AI accelerates threat detection. It doesn’t replace judgement about whether detected threats are real or carefully crafted noise. The future of security depends on that distinction. The writer is CEO & MD, eScan, a cybersecurity company