AI / LLM Red Teaming

LLM-shaped attack surface is new and fast-moving. Prompt injection, jailbreaks, training-data leakage, tool-abuse, and supply-chain risks of model artifacts are all in scope. The OWASP LLM Top 10 (2025 refresh) is the reference taxonomy. For the AI-engineering side see AI / LLM, AI Agent Frameworks, and AI Evals. For SAST / supply-chain see Security Scanning.

LLM red-team frameworks

• ★ PyRIT (Python Risk Identification Toolkit) (Microsoft) — multi-turn, orchestrator-driven LLM red-team automation; many converters / scoring engines. Free OSS — the Microsoft-flagship 2024–26 entry.
• ★ garak (NVIDIA / formerly leondz) — garak CLI; LLM-vulnerability scanner; probes for prompt injection, data leakage, refusal bypass, toxicity. Free OSS — the closest thing to "nuclei for LLMs."
• Promptmap — pen-test-style probe set for prompt injections.
• CyberSecEval (Meta) — coding-LLM benchmark with security-relevant tasks.
• HouYi — research framework for prompt-injection on production apps.
• TextAttack — adversarial NLP framework; broader than security but applicable.
• Adversarial Robustness Toolbox (ART) (IBM / LF AI) — broader ML adversarial / poisoning.
• Counterfit (Microsoft) — older ML adversarial CLI.

Guardrails / runtime defences (paid + free)

• Lakera Guard — paid + free tier; prompt-injection / leakage detection at the gateway.
• Prompt Armor / Robust Intelligence / Cisco AI Defense — paid commercial.
• NVIDIA NeMo Guardrails — open-source rule framework for LLM apps.
• Guardrails AI — open-source structured output + safety; Python.
• Rebuff — open-source prompt-injection detection.
• LLM Guard (ProtectAI) — open-source input/output filter.
• PromptInjectionDefences (NCC) library — research code.
• SecurityWeaver / vigil-llm — community projects.
• Microsoft Prompt Shields / Azure AI Content Safety — paid; Azure-hosted.

Datasets / probes / wordlists

• PromptBench / promptInject / DAN prompts archive — community jailbreak corpora.
• AdvBench — adversarial behaviour benchmark.
• HarmBench / Attack-Suite — model-evaluation benchmarks.
• JailbreakBench / WildGuard — academic.
• Awesome-LLM-Security / awesome-prompt-injection — curated.

Burp / proxy extensions

• Burp AI assistance (Pro 2024.6+).
• Custom Burp extensions for LLM endpoints (community); replay / mutate prompt parameters.
• See Web App Testing for proxy mechanics.

Supply-chain / model-artifact security

• ★ ModelScan (ProtectAI) — scan pickled models for malicious code / unsafe deserialization. Free.
• Safetensors — safer model serialisation format; default for HuggingFace 2024+.
• HuggingFace model audits — picklescan etc.
• See Security Scanning for SAST/SCA on the surrounding code.

RAG / agent attack surface

• Indirect prompt injection in retrieved docs.
• Tool / function-call abuse — see AI Agent Frameworks.
• MCP server abuse — see MCP.
• Output-handling injection — LLM output rendered as HTML / executed as code.
• Memory / context poisoning in long-running agents.
• Confused-deputy attacks on tool-using LLMs.

Training-time risks (link)

• Data poisoning of fine-tunes.
• Backdoors (BadNets / TrojanLLM).
• Membership inference / training-data extraction.
• ART, CleverHans — older ML-adversarial libs.

OWASP / standards

• ★ OWASP LLM Top 10 (2025 refresh) — the reference taxonomy. Free.
• OWASP AI Exchange — community wiki.
• NIST AI RMF — risk-management framework.
• MITRE ATLAS — adversarial-ML-attack matrix; "ATT&CK for AI."
• EU AI Act compliance considerations.

Evaluation pipelines (link)

• See AI Evals — evaluations are how you measure jailbreak / refusal-bypass rate over time.
• Promptfoo — eval framework that supports red-team scenarios.
• DeepEval — Python eval framework.

Practical patterns

• ★ Treat user input as untrusted. Even after the LLM. Output is also untrusted; sanitise before rendering / executing.
• Tool allowlists. Don't give an agent more tools than it needs.
• Output-class enforcement. Force JSON-schema / structured output and validate.
• Egress controls. A "code interpreter" should not be able to fetch arbitrary URLs.
• Run garak / PyRIT in CI. Drift catches.
• Red-team multi-turn, not just single-prompt. Most prod jailbreaks are multi-turn.

Pick this if…

• Default LLM red-team scanner, free: garak.
• Multi-turn orchestrated red-team: PyRIT.
• Runtime guardrail (free): NeMo Guardrails or Guardrails AI.
• Runtime guardrail (paid SaaS): Lakera Guard or Robust Intelligence.
• Model artifact scanning: ModelScan + safetensors.
• Reference taxonomy: OWASP LLM Top 10 + MITRE ATLAS.