Improper Output Handling: LLM XSS, SSRF and RCE

Q: What is improper (insecure) output handling in LLM applications?

Improper output handling is the vulnerability that arises when an application takes the text generated by a large language model and passes it to a downstream component that interprets strings — a browser, a shell, a SQL engine, an HTTP client, a templating engine, a filesystem path — without first validating, encoding, or sanitising it for that specific context. OWASP tracks it as LLM05:2025 and deliberately renamed it from the 2023 term "Insecure Output Handling" to "Improper Output Handling" to emphasise that the defect lives in how the downstream system handles the model's string, not in the model itself. The root misconception it corrects is the belief that model output is trustworthy because it came from your own model. It is not: the output is a function of the prompt, and the prompt can be influenced by an attacker either directly or indirectly through a poisoned document, web page, or tool result the model consumed. So model output must be treated as attacker-influenced data flowing toward an interpreter. When it reaches that interpreter unencoded, the model becomes a general-purpose delivery mechanism for the classical web vulnerabilities — cross-site scripting, server-side request forgery, SQL injection, command injection, and in the worst cases remote code execution — none of which are new bugs, but all of which now have a fresh and very productive source.

Q: How is improper output handling different from prompt injection?

They are two halves of the same untrusted-data problem sitting on opposite sides of the model, and conflating them is why many teams defend only one. Prompt injection is the input-side problem: untrusted text — from a user, a retrieved document, a web page, or a tool result — steers the model into doing something it should not. Improper output handling is the output-side problem: the model's generated text steers your downstream systems into doing something they should not, because you passed that text to an interpreter without neutralising it. The crucial and counterintuitive point is that the two are independent failures. You can have flawless prompt-injection defences and still be fully vulnerable to improper output handling, because even a perfectly aligned model that was never manipulated will, when asked to format a helpful answer, happily produce a javascript: link, a Markdown image that auto-loads and exfiltrates data through the request, an HTML fragment with an onerror handler, or a plausible shell command — and if your renderer or agent executes any of those, you are compromised regardless of how clean the input was. Input guardrails judge the meaning and intent of text; output handling neutralises the syntax of text at the point it meets an interpreter. A serious LLM application needs both boundaries, implemented separately, because each catches attacks the other structurally cannot.

Q: What downstream systems are most at risk from unvalidated LLM output?

Any component that interprets strings rather than merely storing or displaying them as inert text is a sink, and each maps to a well-known vulnerability class. The browser DOM is the most common: rendering model output through innerHTML or an unsanitised Markdown renderer lets an emitted script tag, event handler, or javascript: URI execute as cross-site scripting, and auto-loading image tags leak data through the outbound request itself. A shell, an eval call, or a code-execution tool that runs model-suggested commands is the most dangerous, because it converts output handling directly into command injection or remote code execution in the application's own context. String-concatenated SQL turns model output into SQL injection. An HTTP client that fetches URLs the model produced — very common in agents — becomes a server-side request forgery vector that can reach internal services and the cloud metadata endpoint at 169.254.169.254. Templating engines that render model output can suffer server-side template injection. Filesystem paths built from output enable path traversal such as returning a value like ../../etc/passwd. The unifying property is interpreter power: the more the sink treats its input as instructions rather than data, the more severe the outcome. Agents make every one of these worse because they can chain a fetched web page into the model's next action with no human in the loop, collapsing the time between a poisoned input and an executed exploit.

Q: How do I prevent XSS from LLM-generated content?

You prevent it the same way you prevent any XSS: with context-aware output encoding applied at the point the string is rendered, plus defence-in-depth from a Content Security Policy — and critically, you do not rely on trying to make the model emit safe strings. Source-side filtering, such as instructing the model not to produce scripts or regex-stripping the substring script, fails for exactly the reason input filtering fails: attackers and even benign models produce encodings, event handlers, SVG payloads, and phrasings you did not anticipate. Instead, encode for the exact context in which the output lands — HTML entity encoding inside HTML bodies, attribute encoding inside attributes, JavaScript encoding inside script contexts — following the OWASP Cross-Site Scripting Prevention Cheat Sheet. If your product renders Markdown, which most chat interfaces do, run it through a sanitiser that strips active content, disallows javascript: and data: URIs, and gates auto-loading images so they cannot exfiltrate data through their request. Layer a strict Content-Security-Policy on top so that even if an injection slips past encoding, the injected script has no permitted origin to execute from. The mental model is that the model output is untrusted input arriving at your render layer, identical to a value submitted by an anonymous user, and it deserves precisely the same allowlist-and-encode discipline rather than any trust derived from its provenance.

Q: Can an LLM cause remote code execution or SSRF?

Not by itself, but the surrounding application can very easily turn model output into both, and this is where improper output handling becomes critical rather than merely embarrassing. Remote code execution happens when an application executes model-generated content in a privileged context — for example an agent that runs a shell command the model suggested "to be helpful," an app that passes model output to eval or to a code interpreter with real credentials, or a plugin that treats a model-produced string as an instruction to invoke. Because the model's output is attacker-influenceable through the prompt, an attacker who can shape the input can shape the command, and the sink executes it. Server-side request forgery happens when the model returns a URL and the application, typically an autonomous agent, fetches it without validation; a crafted or manipulated response can point that fetch at internal-only services or at the cloud instance-metadata endpoint 169.254.169.254 to steal credentials. The defences are architectural and classical: never pass model output to a shell, eval, or interpreter that has privileges — sandbox it with no credentials and no network, or better, replace free-form execution with a typed, allowlisted set of actions the model can only select from; and validate every model-produced URL against an egress allowlist that blocks link-local and metadata ranges, per the OWASP SSRF Prevention Cheat Sheet. For these two sinks specifically the most honest recommendation is frequently not to do it at all, because there is rarely a sound reason to hand raw model output to a privileged interpreter.

Q: Why is filtering the model's output at the source the wrong approach?

Because source-side filtering tries to enumerate and block every dangerous thing the model might say, which is the same losing game as input blocklisting, whereas the durable fix makes the sink safe against any string at all. If your defence is a set of rules that strip or reject specific patterns in the output — remove script tags, reject the word DROP, filter out javascript: — you are betting that your blocklist anticipates every encoding, alternate syntax, homoglyph, and novel phrasing the model can produce, and that bet loses continuously because the space of dangerous outputs is effectively unbounded and evolves. It also produces a false sense of safety that discourages the real fix. The correct approach inverts the responsibility: instead of trusting that the output is clean, you assume it is hostile and neutralise it at the exact point it meets an interpreter, using context-aware encoding for renderers, parameterised queries for databases, sandboxing or typed allowlisted actions for execution, and egress allowlists for fetches. Encoding at the sink is robust precisely because it does not need to understand or predict the content — an HTML entity encoder makes any string safe to place in an HTML body whether or not it contains an attack, and a parameterised query makes any string safe as a SQL value whether or not it looks like SQL. That is why the guidance is to encode at the sink and never filter at the source: one approach scales to unknown attacks, the other does not.

Q: Where should the trust boundary sit in an LLM application?

The trust boundary should sit immediately after the model, treating the model's output as the network edge — untrusted input to every component that consumes it. This is the single most useful architectural reframing for the problem. Most teams implicitly draw the boundary before the model, validating user input and then trusting everything the model produces downstream, which is exactly backwards for output handling: the danger is what flows out of the model into your interpreters, not only what flows in. Concretely, every model response should pass through one output-handling layer before it reaches any sink, and that layer applies the appropriate control per destination — encode for the browser, parameterise for SQL, allowlist and sandbox for execution, egress-filter for HTTP fetches — and rejects or quarantines anything that fails a schema check. Placing the boundary in code as a single, owned, tested layer matters because the reason these bugs proliferate is organisational: the ML team owns the model, the application team owns the sinks, and the output string crosses between them with nobody appointed to validate it. Making the boundary an explicit place in the architecture — rather than a hope distributed across many call sites — is what turns output handling from an aspiration into an enforced control, and it lets you log rejects, monitor for attack patterns, and reason about the blast radius the way you already do for your public network edge.

Q: What is a maturity path for fixing improper output handling?

A practical maturity path moves from implicit trust to a governed output boundary in five stages. Level 0 is implicit trust: model output is rendered and executed directly, there is no output boundary, and classical web vulnerabilities are reachable straight through the prompt. Level 1 is a sink inventory: you enumerate every place model output reaches an interpreter — browser DOM, SQL, shell, HTTP client, templating engine, filesystem — and rate each by how much interpreter power it grants, because you cannot protect sinks you have not listed. Level 2 is encode and parameterise: context-aware output encoding at every render sink, parameterised queries for all model-influenced SQL, Markdown passed through a sanitiser, and a strict Content-Security-Policy deployed, which together close the common XSS and SQL injection paths. Level 3 is schema-constrained and sandboxed: model output is validated against typed schemas with allowlisted fields before use, any code or command path runs in a no-privilege sandbox or is replaced entirely by an allowlisted typed action set, and agent fetches are egress-controlled against link-local and metadata ranges. Level 4 is a governed output boundary: a single output-handling layer that all responses traverse, with per-sink encoders, schema validation, egress policy, logging of rejected outputs, and least-privilege sinks, owned and monitored like the network edge it functionally is. Most applications ship at Level 0 while describing the model as safe; the path forward is to recognise that the renderer, the shell, and the HTTP client are the boundary, and to invest there rather than in trying to sanitise the model.

Satyam Kumar

Retour au blog

cyber-security-patterns

Improper Output Handling: When the LLM's Answer Becomes XSS, SSRF or RCE

By Satyam KumarAugust 22, 202610 min read

improper output handling insecure output handling owasp llm top 10 llm security xss ssrf remote code execution sql injection prompt injection ai application security output encoding content security policy llm guardrails agent security appsec 2026

Improper Output Handling: When the LLM's Answer Becomes XSS, SSRF or RCE

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

Improper Output Handling: When the LLM's Answer Becomes XSS, SSRF or RCE

Frequently Asked Questions

Partager l'article

Comments

Leave a comment