TL;DR: circuit-litellm runs LiteLLM Proxy on your own host, in front of every model you use, hosted or local. Everyone gets their own key to the gateway, with a model list and a budget. The real provider keys sit in one root-only file, and every request leaves a row saying who asked, which model answered and what it cost.
Count the places your OpenAI key lives. The production .env. Two or three laptops. The CI secrets. And that Slack thread from March where someone needed it “just for a minute”.
Somebody leaves, so you rotate it, and something breaks. The invoice doubles, and nobody can tell you which of those places did it.
The fix is old and boring, which is exactly why it works. Put one gateway in front of the models, give everyone their own key to the gateway, and keep the real keys in one place. LiteLLM Proxy is that gateway for language models. circuit-litellm runs it on your own host, and one Conn3x apply sets the whole thing up. The full walkthrough is on the Conn3x blog. This is the short version, and the bit about why.
One Apply, No Secrets in the File
Look for the API keys. There aren't any.
LiteLLM runs from its official image, pinned by tag and digest, under Podman as a systemd service. Its keys, teams, budgets and spend live in PostgreSQL, which circuit-postgres puts on the same box. The pair fits in 2 GB of memory, on Debian, Ubuntu, Fedora or Rocky Linux.
The interesting part of the config is the end:
module "litellm" {
source = "github.com/conn3xorg/circuit-litellm?ref=v0.2.1"
# hosts, SSH key and database as in the README, then:
models = [
{
model_name = "gpt-4o"
litellm_params = { model = "openai/gpt-4o", api_key = "os.environ/OPENAI_API_KEY" }
},
{
model_name = "claude-sonnet"
litellm_params = { model = "anthropic/claude-sonnet-4-5", api_key = "os.environ/ANTHROPIC_API_KEY" }
},
]
api_keys = {
OPENAI_API_KEY = var.openai_api_key
ANTHROPIC_API_KEY = var.anthropic_api_key
}
default_key_budget = 20
max_key_budget = 200
budget_duration = "30d"
}
Those last three lines are your budget policy. A key made without a budget gets $20, nobody can ask for more than $200, and spend resets every 30 days.
And notice what’s missing: a secret. The provider keys come in as ephemeral variables and land on the host in a root-only file. They never touch your plan, your state or your CI logs. Paste one into models as plain text and the plan refuses, because that value would end up in state. LiteLLM’s own master key and salt key are made on the host by the first apply, and Conn3x never sees them at all.
A Key for Alice
And nothing to rotate when she leaves
Nobody gets the master key. You use it on the host to hand out keys that can do less. A team first:
mk=$(sudo sed -n 's/^LITELLM_MASTER_KEY=//p' /etc/litellm/keys.env)
curl -s http://127.0.0.1:4000/team/new \
-H "Authorization: Bearer $mk" -H 'Content-Type: application/json' \
-d '{"team_alias": "research", "models": ["claude-sonnet"], "max_budget": 50}'
Then a key for someone on it, using the team_id that came back:
curl -s http://127.0.0.1:4000/key/generate \
-H "Authorization: Bearer $mk" -H 'Content-Type: application/json' \
-d '{"team_id": "TEAM_ID", "key_alias": "alice-laptop"}'
Alice drops that sk-... into her tools and points them at the gateway as her OpenAI base URL. She can use claude-sonnet. Ask for gpt-4o and she gets a 403. When she leaves, you delete alice-laptop. Nobody rotates anything, and nothing breaks.
Here’s what happens to each of her requests on the way through:
alice-laptop: one walks every check and gets an answer, the other stops at the model list with a 403. Both leave a row in the spend log, and neither stores the prompt.Read the animation as text
- Key
alice-laptopon teamresearch, budget $50, may use onlyclaude-sonnet. - The LiteLLM gateway on
127.0.0.1:4000checks every request in order: is the key real, is the model on the key’s list, is there budget left. Then Presidio masks names and emails (whenguardrailsis set), the request goes on with the provider key kept on the host, and a spend-log row is written to PostgreSQL. - Models behind it:
gpt-4o(OpenAI),claude-sonnet(Anthropic) andlocal(Ollama on the box). - Request 1,
claude-sonnet: all six checks pass. “My name is David Johnson” reaches the provider as “My name is<PERSON>”. Spend-log row: alice-laptop, research, claude-sonnet, tokens and cost recorded, ok, prompt not stored. - Request 2,
gpt-4o: the key is real, but the model isn’t on the key’s list, so the gateway answers403and forwards nothing. Spend-log row: alice-laptop, research, gpt-4o, failed with403, prompt not stored.
The One Setting I’d Think About
Copy the rest. Decide this one.
Every request leaves a row in LiteLLM’s spend logs: which key and team, which model answered, when, how many tokens, what it cost, and whether it worked. Alice’s 403 is in there too, as a failure. That’s the record you’d hand to an auditor, or to finance for a chargeback.
What isn’t in there, by default, is the conversation. Set log_prompts = true and LiteLLM keeps every request and answer in full. Handy when you’re debugging, or when a policy says you must keep them. It’s also a database of everything anyone ever pasted into a prompt, passwords and customer data included, readable by anyone with admin access to LiteLLM.
So pick one on purpose, and tell your people which you picked.
Local Models Get the Same Rules
Ollama, behind a key it never had
Models don’t have to be hosted. Put circuit-ollama on the same box and a local model sits behind the same keys, budgets and logs as GPT:
models = [
{
model_name = "local"
litellm_params = { model = "ollama_chat/qwen2.5:0.5b", api_base = "http://127.0.0.1:11434" }
},
]
That matters more than it looks. Ollama’s API has no authentication at all; a plain curl was enough to list the models in the private AI box post. Behind the gateway, it never has to face the network. On an Ubuntu 24.04 droplet with 4 GB, a team key limited to a local model got its answer through LiteLLM.
Two more inputs arrived in v0.2.0. guardrails sends every request through Presidio first, so “My name is David Johnson” reaches the provider as “My name is <PERSON>”. If Presidio is down, the request fails with a 500 rather than going out unmasked. And langfuse_url sends a trace of every call to a Langfuse you run yourself.
What It Won’t Do
So you don't find out on your own
CI applies it to Rocky Linux 10, Fedora 45, Ubuntu 24.04 and Debian 13, and checks the things this post leans on: a wrong master key can’t make keys, a budget over the cap is refused, the 403 happens, the spend row has a price and no prompt, and a key still works after a restart. By hand, on a 2 GB Rocky droplet with SELinux enforcing, port 4000 didn’t answer from outside and LiteLLM came back on its own after two reboots.
And the gaps:
- No TLS. LiteLLM speaks plain HTTP on
127.0.0.1:4000. Put circuit-caddy in front before anything off the host uses it. - No teams or keys made for you. That’s LiteLLM’s API, or its admin UI at
/ui. - No Redis. One LiteLLM doesn’t need it. Several behind a load balancer do.
- No backups. Back up PostgreSQL and
/etc/litellm/keys.env, separately. Lose the salt key and LiteLLM can’t read the provider credentials it stored. - Prices go stale. Requests are priced from the list in the image, so a model newer than your
litellm_versioncounts as free until you upgrade or price it yourself. - Upgrades are untested. Moving from one
litellm_versionto the next hasn’t been tried yet.
Start With the Gateway
Then tell me what it caught
If you only run one piece of this series, make it this one. It answers the three questions people ask after the first big invoice: who can use which model, how much they can spend, and who asked what.
The README has every input, and examples/local runs it on a throwaway container. Next on the Conn3x blog is a chat page, Open WebUI in front of this gateway, for the people who’d rather not use curl. If you put this in front of your team, start a discussion and tell me what the spend log turned up.
Share