LiteLLM External Guardrails: Open Source Options Compared

Quick Navigation

Difficulty: Advanced
Estimated Time: 20-30 minutes
Prerequisites: LiteLLM Proxy experience, LLM security basics, YAML configuration, Docker familiarity

Choosing a guardrail is usually framed as picking the most capable detector. In front of a LiteLLM proxy the integration route matters at least as much, because a guardrail that needs a fork or adds a full inference pass to every request will not survive contact with production.

For a broader comparison of guardrail frameworks independent of LiteLLM, see the LLM Guardrail Frameworks benchmark. This article is specifically about what integrates well with LiteLLM Proxy.

Integration Routes

1. Generic Guardrail API

The main external path. It calls your own HTTP endpoint, with no PR to the LiteLLM repo required, and supports pre, during and post hooks.

guardrails:
  - guardrail_name: "my-guardrail"
    litellm_params:
      guardrail: generic_guardrail_api
      mode: pre_call        # or post_call, during_call
      api_base: https://your-guardrail-api.com
      api_key: os.environ/YOUR_GUARDRAIL_API_KEY
      unreachable_fallback: fail_closed

• unreachable_fallback defaults to fail_closed. Setting fail_open proceeds when the endpoint is unreachable.
• fail_on_error: false is a full bypass on any failure. Only use it when availability genuinely outweighs the security control. Fail-open bypasses are logged at critical level with call and trace IDs.
• Static headers are sent on every request; extra_headers forwards named client headers.
• Send the whole conversation in one call rather than one call per message: full context, one round trip.

2. Custom Code Guardrail

A Python class referenced as guardrail: custom_guardrail.myCustomGuardrail. LiteLLM extracts the content, hands it to you as inputs, and writes back whatever you return. Raise an exception to block.

Use pre_call if you mask or rewrite content. during_call runs in parallel with the LLM call for lower latency, so it blocks reliably but your edits may not land.

3. Prebuilt Providers

Bedrock, Presidio, Lakera, Azure Content Safety, Model Armor, Pangea, Javelin, Pillar and others use the same config shape with guardrail: provider_name.

Comparison

Scoring

These are judgment calls rather than benchmark results, scored 1 to 5. Reweight them against your own threat model.

Reading the Scores

The totals mislead if taken flat.

• Presidio wins by being narrow and excellent. Its coverage score of 3 is honest: it genuinely does not do injection detection or toxicity.
• NeMo and Llama Guard score low on operations and integration, not on capability. If dialog-level policy is your requirement, NeMo's 16 is irrelevant, because nothing else does it.
• litellm_content_filter at 23 is a real result, not an artifact. Zero infrastructure and regex speed matter in production. It simply will not catch anything semantic.

Most Flexible: NeMo Guardrails

Five rail types across input, dialog, retrieval and output, and the only guardrails toolkit that also models the dialog between user and LLM. Rails are written in Colang, so policy becomes a programmable flow rather than a fixed detector list.

It is designed to absorb the others, with integrations for ActiveFence, AlignScore and LangChain chains, explicitly unifying moderation endpoints, critique chains, output parsing and individual guardrails such as LLM Guard into one layer. The catalog mixes NVIDIA safety models, community models, LLM self-check prompts and third-party APIs.

The cost is real: no native LiteLLM support, LLM-in-the-loop latency, and the Colang learning curve.

Runner-up: Guardrails AI, with its validator hub, custom validators in plain Python, and a native LiteLLM provider.

Pragmatic middle path: run NeMo as a service behind a generic_guardrail_api endpoint. That gives you Colang's expressiveness with LiteLLM's config-driven wiring and fail-closed semantics, without a fork.

Recommended Stack

Two containers, no GPU.

# presidio-analyzer :5002, presidio-anonymizer :5001

guardrails:
  - guardrail_name: "presidio-pii"
    litellm_params:
      guardrail: presidio
      mode: pre_call
      presidio_analyzer_api_base: http://presidio-analyzer:5002
      presidio_anonymizer_api_base: http://presidio-anonymizer:5001

  - guardrail_name: "llm-guard-scan"
    litellm_params:
      guardrail: generic_guardrail_api
      mode: during_call
      api_base: http://my-scanner:8000
      unreachable_fallback: fail_closed

Three rules that make this stack work:

• Masking must run at pre_call. At during_call the edit may not land.
• Injection scanning at during_call runs parallel to the LLM call, so it costs no added wall-clock latency.
• Defense in depth stacks: use mode: [pre_call, post_call] and multiple guardrails per model.

Related Capabilities

GET /guardrails/list returns each guardrail's name, type and mode. Guardrails can be applied per request by passing a guardrails array in the /chat/completions payload.

The Tool Permission Guardrail applies allow and deny regex on tool names, for example ^mcp__github_.*$, with optional tool-type regex and regex validation on nested argument paths. It is provider-agnostic across OpenAI tool_calls, Anthropic tool_use and MCP tools.

Tags: #LiteLLM #Guardrails #LLMSecurity #Presidio #AIGovernance