📄️ Quick Start
Quick start CLI, Config, Docker
📄️ 🐳 Docker, Deploying LiteLLM Proxy
You can find the Dockerfile to build litellm proxy here
📄️ ⚡ Best Practices for Production
1. Use this config.yaml
🔗 📖 All Endpoints (Swagger)
📄️ ✨ Enterprise Features - SSO, Audit Logs, Guardrails
To get a license, get in touch with us here
📄️ 💡 Migrating from OpenAI (Langchain, OpenAI SDK, LlamaIndex, Instructor, Curl)
LiteLLM Proxy is OpenAI-Compatible, and supports:
📄️ 🎉 Demo App
Here is a demo of the proxy. To log in pass in:
📄️ Proxy Config.yaml
Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.
📄️ 🔥 Load Balancing, Fallbacks, Retries, Timeouts
- Quick Start load balancing
📄️ 💸 Spend Tracking
Track spend for keys, users, and teams across 100+ LLMs.
📄️ Custom LLM Pricing - Sagemaker, Azure, etc
Use this to register custom pricing for models.
📄️ 🤗 UI - Self-Serve
Allow users to create their own keys on Proxy UI.
📄️ 🔑 Virtual Keys
Track Spend, and control model access via virtual keys for the proxy
🗃️ 🪢 Logging
3 items
🗃️ Secret Manager - storing LLM API Keys
2 items
📄️ 👥📊 [BETA] Team Based Logging
Allow each team to use their own Langfuse Project / custom callbacks
📄️ 🛡️ Guardrails
Setup Prompt Injection Detection, Secret Detection on LiteLLM Proxy
📄️ 💸 Tag Based Routing
Route requests based on tags.
📄️ 💰 Budgets, Rate Limits
Requirements:
📄️ 💰 Setting Team Budgets
Track spend, set budgets for your Internal Team
📄️ 🙋♂️ Customers
Track spend, set budgets for your customers.
📄️ 💵 Billing
Bill internal teams, external customers for their usage
📄️ [BETA] JWT-based Auth
Use JWT's to auth admins / projects into the proxy.
📄️ 🚨 Alerting / Webhooks
Get alerts for:
📄️ [BETA] UI - Admin
Create keys, track spend, add models without worrying about the config / CRUD endpoints.
📄️ 📈 [BETA] Prometheus metrics
🚨 Prometheus metrics will be out of Beta on September 15, 2024 - as part of this release it will be on LiteLLM Enterprise starting at $250/mo
📄️ ➡️ Create Pass Through Endpoints
Add pass through routes to LiteLLM Proxy
📄️ ✨ 📧 Email Notifications
Send an Email to your users when:
📄️ ✨ Attribute Management changes to Users
Call management endpoints on behalf of a user. (Useful when connecting proxy to your development platform).
📄️ 👥 Team-based Routing
Routing
📄️ Region-based Routing
Route specific customers to eu-only models.
🗃️ Extra Load Balancing
1 items
📄️ Model Management
Add new models + Get model info without restarting proxy.
📄️ Health Checks
Use this to health check all LLMs defined in your config.yaml
📄️ Debugging
2 levels of debugging supported.
📄️ PII Masking
LiteLLM supports Microsoft Presidio for PII masking.
📄️ 🕵️ Prompt Injection Detection
LiteLLM Supports the following methods for detecting prompt injection attacks
📄️ Caching
Cache LLM Responses
📄️ Modify / Reject Incoming Requests
- Modify data before making llm api calls on proxy
📄️ Post-Call Rules
Use this to fail a request based on the output of an llm api call.
📄️ CLI Arguments
Cli arguments, --host, --port, --num_workers