Open-Source LLM Guardrail Frameworks in 2026: Presidio, LLM Guard, NeMo, Guardrails AI, and Llama Guard Compared

Quick Navigation

Difficulty: Intermediate
Estimated Time: 15-20 minutes
Prerequisites: Basic understanding of Large Language Models (LLMs), Familiarity with API gateways and proxies, Working knowledge of YAML configuration, Experience running Docker containers

Putting an LLM behind a gateway solves routing, budgeting, and key management. It does not stop a user pasting a customer's national insurance number into a prompt, and it does not stop a crafted instruction hijacking your system prompt. That is the guardrail layer's job, and it is a separate decision from picking a model.

LiteLLM Proxy has become a common place to enforce that layer, because guardrails attach declaratively per model and per request rather than being scattered through application code. This article compares the open-source options that plug into it: Presidio, LLM Guard, Guardrails AI, IBM FMS Guardrails, NVIDIA NeMo Guardrails, Llama Guard, and the proxy's own built-in content filter.

How Guardrails Attach to LiteLLM

There are three integration routes, and picking the wrong one is the most common source of wasted effort.

The first is a prebuilt provider. Presidio, LLM Guard, Guardrails AI, and IBM FMS all ship as named providers, configured with a few lines of YAML.

The second is the Generic Guardrail API, which calls an external HTTP endpoint you control. It requires no pull request against the LiteLLM repository, which makes it the correct home for anything not natively supported.

guardrails:
  - guardrail_name: "my-guardrail"
    litellm_params:
      guardrail: generic_guardrail_api
      mode: pre_call
      api_base: https://your-guardrail-api.com
      api_key: os.environ/YOUR_GUARDRAIL_API_KEY
      unreachable_fallback: fail_closed

The third is a custom Python class, referenced as custom_guardrail.myCustomGuardrail. LiteLLM extracts the content, hands it over, and writes back whatever is returned. Raising an exception blocks the call.

The mode field matters more than it looks. pre_call runs before the model call and is the only mode where rewriting or masking content reliably lands. during_call runs in parallel with the model call, so it blocks correctly but edits may arrive too late. post_call inspects the response.

Table: Integration Routes

Failure Semantics Are a Design Decision

Before comparing detectors, decide what happens when the guardrail itself is unavailable. unreachable_fallback defaults to fail_closed, meaning an unreachable guardrail blocks the request. Setting fail_open lets traffic through instead.

A separate setting, fail_on_error: false, bypasses the guardrail on any error at all. It is a complete removal of the control, appropriate only where availability genuinely outweighs the security guarantee. Fail-open bypasses are logged at critical level with call and trace identifiers, which is the only reason the setting is defensible in production at all.

Table: Failure Modes

Presidio: Narrow and Excellent

Presidio is Microsoft's open-source SDK for detecting and anonymizing personally identifiable information in text and images. It runs as two containers, an analyzer and an anonymizer, and it is a native LiteLLM provider.

guardrails:
  - guardrail_name: "presidio-pii"
    litellm_params:
      guardrail: presidio
      mode: pre_call
      presidio_analyzer_api_base: http://presidio-analyzer:5002
      presidio_anonymizer_api_base: http://presidio-anonymizer:5001

It scores well because it does one job properly and cheaply. Named-entity recognition adds tens of milliseconds, not hundreds, and there is no GPU in the path. It genuinely does not detect prompt injection or toxicity, so treat it as one layer rather than the whole answer.

Table: Presidio Fit

LLM Guard and Guardrails AI: Broader Detector Sets

LLM Guard provides input and output scanners covering prompt injection, PII, and toxicity. Guardrails AI takes a different shape: a hub of composable validators wrapped around the call, extensible with plain Python. Both are native LiteLLM providers.

The trade is latency. These are transformer-backed scanners, not regular expressions, so run them at during_call where blocking is required but rewriting is not.

Table: Broad-Coverage Providers

NeMo Guardrails: The Flexible One, at a Price

NVIDIA's NeMo Guardrails is the most capable option and the most awkward to adopt. It supports five rail types covering input, dialog, retrieval, and output, and it is the only guardrails toolkit that also models the dialog between the user and the model. Rails are written in Colang, so policy becomes a programmable flow rather than a fixed list of detectors.

It is also designed to absorb its competitors, shipping integrations with ActiveFence, AlignScore, and LangChain chains, and explicitly aiming to unify moderation endpoints, critique chains, output parsing, and individual guardrails such as LLM Guard into a single layer.

The cost is real. LiteLLM has no native NeMo support, so it must run as a separate upstream service or be wrapped in custom code. Rails that call a model add model-call latency to every request. Colang is another language for the team to learn.

The pragmatic compromise is to run NeMo as a service behind a generic_guardrail_api endpoint. That preserves Colang's expressiveness while keeping LiteLLM's declarative configuration and fail-closed semantics, with no fork of either project.

Table: NeMo Trade-offs

Scoring the Field

Scores below are engineering judgement against the criteria named, not measured benchmark results. Reweight them against your own threat model. Each dimension is scored from 1 to 5.

Table: Scored Comparison

Read the totals with care, because a flat sum misleads here in three specific ways.

Presidio leads by being narrow and excellent, and its coverage score of 3 is an accurate description of scope rather than a defect. NeMo and Llama Guard score low almost entirely on operational burden and integration effort, not on capability; if dialog-level policy is a hard requirement, NeMo's total is irrelevant because nothing else in the table does it. The built-in content filter reaching 23 is a real result rather than a consolation prize, since zero infrastructure and regular-expression speed carry enormous weight in production, even though it catches nothing semantic.

A Recommended Starting Stack

Two containers, no GPU, and defence in depth across both PII and injection.

guardrails:
  - guardrail_name: "presidio-pii"
    litellm_params:
      guardrail: presidio
      mode: pre_call
      presidio_analyzer_api_base: http://presidio-analyzer:5002
      presidio_anonymizer_api_base: http://presidio-anonymizer:5001

  - guardrail_name: "injection-scan"
    litellm_params:
      guardrail: generic_guardrail_api
      mode: during_call
      api_base: http://my-scanner:8000
      unreachable_fallback: fail_closed

Two operational details are worth building in from the start. Send the whole conversation to the guardrail in a single call rather than one call per message, so the guardrail sees full context and you pay one round trip. And expose the guardrail set to clients through GET /guardrails/list, which returns each guardrail's name, type, and mode; callers then select per request by passing a guardrails array in the /chat/completions payload.

For agentic workloads, the Tool Permission Guardrail is a separate and often overlooked control. It applies allow and deny rules by regular expression against tool names, such as ^mcp__github_.*$, and validates nested argument paths. It is provider-agnostic across OpenAI tool_calls, Anthropic tool_use, and MCP tools.

Table: Stack Composition

Conclusion

The selection question is not which framework is best, it is which layer is missing.

• Regulatory PII exposure or data residency: Presidio at pre_call
• Prompt injection and jailbreak attempts: LLM Guard or Llama Guard at during_call
• Topic and conversation policy: NeMo Guardrails, the only option modelling dialog state
• A cheap first pass with no new infrastructure: the built-in content filter
• Agent tool constraint: the Tool Permission Guardrail

These compose rather than compete. A single model can carry several guardrails, and a single guardrail can declare mode: [pre_call, post_call]. The strongest configurations in production are usually a fast deterministic filter, a specialised PII layer, and one semantic scanner running in parallel with the model call.

The open question for 2026 is whether the semantic layer stays external. Every option that reasons about meaning currently pays a full model call for the privilege, which is why NeMo and Llama Guard score so poorly on latency despite scoring well on capability. If that cost collapses, the calculus in the scoring table changes completely.

Tags: #LLMSecurity #Guardrails #LiteLLM #Presidio #NeMoGuardrails #PromptInjection #PII #AIGateway