You don't need a proxy to do this. I would argue you should not use a proxy to do this because the end result is a single point of failure direct in the line of your productivity.
Both Codex[0] and Claude Code[1] output OTEL traces that you can ingest and process.
Other harnesses like OpenCode, Pi, etc. you can hook into their plugin architecture and implement your own OTEL emission from a plugin (or use an off the shelf plugin[2]).
Host an OpenTelemetry collector[3][4] or rely on your existing collector infra (e.g. Datadog (no affiliation), AppInsights, etc.)
The OTEL traces will have exactly what you need. Tool calls, full prompts, token usage/cost, etc. It is less intrusive and does not become a single-point-of failure since the OTEL emission and collection is best-effort whereas a failing/overloaded proxy is going to strand your team until resolved. It's OK if you lose some OTEL traces; no productivity lost.
The configuration to enable this for Codex and CC is very straightforward (see [0][1] + [5])
If you are using a Datadog, Langfuse, Langchain, or other OTEL target, you do nothing. Just query it. Build alerts, dashboards, and datasets; whatever you normally do with your OTEL collector. A forward OTEL collector deployment is still useful for filtering but the Datadog Agent, for example, already has buffering and filtering built into it.
If you want to deploy your own collector and write your own handler (e.g. put it into a database), you can generate the stubs for the OTEL payloads and convert to domain records.
2. Generate the type/class stubs and endpoints for your platform.
3. I strongly recommend pre-filtering with the open source `opentelemetry-collector` before your custom endpoint handles the inbound telemetry payload. The collector is optimized for performing the filtering and I put it upstream of my custom ingest endpoint on a very small VM just to pre-filter and perform auth checks[0][1] on the inbound channel.
4. Do what you need to with the inbound payloads. Examples below:
I run my backend on Postgres with partitioned tables. You can forward to Elastic or other OLAP-type database targets for querying. You can see examples of what I do with the session telemetry here: https://zeeq.ai/docs/key-features/session-telemetry
Disclosure: I work on Origin [1]. We built a platform for enterprises that consumes OTEL, and other traces without a proxy or an LLM gateway. We even support collecting traces from local inference workloads.
For supported tools, we collect prompts, responses, and tool calls, then tie them to the user, organizational context and what actually happened on the machine. We can then use this data to detect when agents (or the people piloting them) do bad things [2].
Wouldn't a fair criticism of OTEL be that you only see what these harnesses are willing to show you?
There have been numerous scandals of AI companies offering CC/cursor alternatives, then indexing data they never should have; specifically Z.ai [0], ByteDance Trae [1] and Alibaba Qwen [2]. Maybe the argument should be that all harnesses should be sandboxed and monitored for all networking.
If it were me, all AI harnesses would be running on a VM with network logging enabled. Beyond not being able to trust what data they send via the harness, you should audit what data they access.
Sometimes, it gets to very hacky solutions, such as what we did at Laminar with Claude Agent SDK. Basically we spin up a tiny in-process proxy that intercepts requests and responses and emits the telemetry as OTEL traces. We repoint the Python / JS SDK to convince the harness that our proxy is the anthropic URL, and the proxy then forward the request downstream. I wrote about it some time ago [1]. This is definitely not elegant, but it generally works.
Otel is only a quarter of what the llm proxies solve for you. What happens when an agent goes down some random rabbit hole and burns through $100 before you even notice? It's nice to have post-disaster visibility with Otel, but even self hosting an LLM proxy prevents the disaster.
Shameless plug, I've been building Relay for this exact reason. You get real-time visibility into what your agents are doing, how much they're spending, and can set hard limits so things don't spiral out of control. Plus a lot of other features they bring to the table if you want.
Sure, but OTEL has all of the right qualities for the majority of use cases here.
Standards based, many off-the-shelf providers, out-of-process + buffered by default, many standard processors for traces + spans, many tools for storing and querying.
OTEL is pretty widely supported and mostly "just works" at this stage of maturity. A failing/mis-configured OTEL endpoint/collector isn't going to stop you from getting work done in the same way that a failing proxy is. You'll drop some traces, but that's probably OK in most cases.
There is a place for proxies, but those places should share the same failure boundary mode. Meaning if the proxy fails, it should necessarily stop forward agent activity. Example: proxy injection of secrets instead of local acquisition.
I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription.
I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.
The current crop of flash models are more comparable to sonnet than opus but they are very affordable.
The reality is that the subscriptions are heavily subsidized. Pretty much every sub that's not run by one of the money-burning frontier labs has had to drop it or dramatically increase prices and lower limits in the last few months. Cheap tokens are an acquisition tool, not a real product. It sounds dramatic but it really is the drug dealer sales funnel.
At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.
>At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.
So just wait until that happens? What's the point of scrimping and saving now? Not to mention you're paying for a premium in the meantime.
It's not obvious the subsidization will end. Loss leaders are a thing. Most enterprise at this point are paying API pricing
To better predict what they'll do with it we'd need to know the proportion of token usage via API vs subscription. Keeping an affordable, slightly unprofitable subscription to get people in isn't a bad idea. They also get marketing benefits from the word of mouth
Also, some subset of developers on subscriptions are going to develop products that use the API, and some of those products are going to be successful.
In the optimistic case you describe, what smart customers want to watch out for is being lured into vibing your projects into such byzantine slop-complexity that subscription plans and trailing models are no longer sufficient to make progress on them.
Vibe-coders talk about surrendering understanding to the models, trusting that there will always be a model able to overcome and work through any technical debt they happen to accumulate, but the danger that still lurks there (at least for the naive) is that you'll need increasingly large contexts, increasingly sophisticated models, and increasingly large token spends just to keep that ball rolling.
Between profitable enterprise accounts being ineligible for subscription, eager but busy people paying for underutilized subscriptions, and naive vibe-coders falling into their pit trap, you can indeed imagine subsidized subscription accounts being available for a long time and remaining an effective value hack for people careful enough not to fall into the pit trap. But only insofar as they are that careful.
Also they can always find another way to make money. Make subscription users product and sell some analytics like "show most trending project being built this week".
You cannot predict the future, but you can identify risks and attribute a likelihood. A risk is that the subsidized inference will either end or be reduced significantly in the next 1-2y. If you think the likelihood is high enough it makes sense to ensure your systems are prepared for it.
> It's not obvious the subsidization will end. Loss leaders are a thing.
Loss leaders are a thing so long as they actually do lead somewhere. If your primary value proposition is conveyed by the loss leader itself, where do you go from there?
(meta commentary)
really gnarly word choice: "drop" in "they're going to drop the real prices in your lab"?
I took your meaning as "they're gonna drop [a bad surprise on you by raising] the real prices"... so at least it's not quite as tortured as the ambiguous "support for version N dropped today" (where you often can't even infer the direction from context).
this isn't just a tangent, I'm having trouble understanding the point you're trying to make, which hinges entirely on deciding if you meant "drop" literally
They meant "they're going to drop a bad surprise by charging you API prices", so yes, a better wording would be "they're going to give you a bad surprise in the form of forced API pricing".
The term "drop" becoming a synonym for "release", "put out" or "put into effect" for things outside of music/albums is one of the most confusing linguistic innovations of the last decade.
> At some point they're going to drop the real prices in your lab
The real price for tokens at a specific level of intelligence is dropping rapidly. Between models improving and new hardware getting faster, I think prices will be going down, not up.
> Better to get a handle on it now.
I experiment with a lot of providers and models. I strongly disagree that anyone needs to start migrating now in preparation for prices maybe going up in the future.
It doesn’t make sense. Deal with a price increase when it happens, if it does.
Switching models and providers isn’t some huge lift. I can jump to a new model or even hardness and get a feel for it in a day or two. It’s not some huge undertaking that requires preparation and planning.
If prices go up, try something at that point in time. Don’t worry about it until then.
When Claude Code subscriptions really started to catch on, there was a popular idea on HN that Anthropic's end game was that basically every dev adjacent role would need to pay for a $200/mo coding plan.
Then, both Anthropic and OpenAI in lockstep developed a enterprise pricing structure that openly coerced large businesses to pay API prices where it was feasible to spend $200/day rather than the comparatively cheap $200/mo.
Even if token pricing relative to intelligence does trend down over time, that doesn't mean the pricing structure will actually translate to lower costs, particularly in the cash cow B2B market.
Companies still believe they have IP to protect. Even when all they do are basic CRUD systems or on the verge of collapsing if they don’t renew some contracts.
From experience, if you work somewhere with very disconnected higher ups a sales person shows what your token usage is vs spend on the monthly subs for whatever set of users you have and then show an API price with a sample bill for even “more users” at that same number of tokens that is 30% lower. The crux of course being that your existing user base is not going to use fewer toke a just because you signed more people up and have a discount for the status quo.
Then when you consolidate all users to a single provider you find out you have a 2x to 3x larger bill because you… logically have more users at the same average token spend than the quote.
At my previous job, all while managing a full team, I did a lot of porting of legacy apps to modern standards to maintain various obligations. It was AI gen work in between meetings and paperwork. It was seen as such a success that I was asked to move to an IC role to focus on it within the “Elite new products team” that was supposed to be all AI and the latest tech. I showed up to find that they were pushing out low quality manually written code in 12 hour days six days per week to appear like LLMs were in use to save face because they lacked a token budget or the skills to properly manage one under the APIs. They had dismissed someone who had spent multiple thousands per month on Agent swarms. It was the perfect situation to make their 3x budget issue a 10-30x budget spend issue.
For many many reasons I left that job and got a new one but that was the cherry on top.
The API/subscription split as always there. I don’t know why you’re talking about it as a bait-and-switch development.
That doesn’t change anything about what I said, though. The relative price per task or price per unit intelligence is also going down in the APIs too.
> Even if token pricing relative to intelligence does trend down over time, that doesn't mean the pricing structure will actually translate to lower costs, particularly in the cash cow B2B market.
What are you talking about? The APIs serve the same models as the subscriptions. Unless you think the price of APIs has been going up (it has not) this argument doesn’t make sense
> At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less.
This only applies under the assumption that everyone tokenmaxxes their subscription.
I have visibility across a few different Claude team subscriptions. I'd say roughly 10% of people max their subscriptions.
I think I've only seen one organization where the under-used subs don't more than pay for the tokenmaxxed ones.
And that is ignoring the 90% margin that API inference runs at.
At some point companies will need to have the “scary” conversation about the value proposition of adding another human to the team, or adding a comparable amount of token spend to the team.
$1k/person sounds like a lot, but a ten person team at $1k token burn / head / month is spending $120k. How much does a new hire cost, tco? Does velocity from one more person compare with velocity from one more person’s worth of tokens?
Sure, learning to be efficient matters. So does taking the time to consider cost vs alternative. You might be surprised by what you find, in either direction.
Counter point - models get better, ways to run them gets optimized and hardware will eventually come down in price.
The share of tasks where "cheaper" models are enough also increases for same reason. For majority of use cases it will quickly become race to the bottom. And that means part of the pie for the "premium" models of frontier AI labs will quickly become smaller and smaller.
The drug dealer gives you a free hit because it is profitable for him in the long term for you to be addicted to his product, not because he is your friend.
What? (1) Yes.. it's never that [stupid economics, they don't want their purchasers to die.. might as well sell fentanyl], (2) it's always that [hardly any way to propose alternatives.. within the scope of not killing the customer]
I'm not sure exactly what you're implying but if you're curious, there's a lot of writing about the persistence of this particular urban legend. A lot of it starts with 1920s newspapers painting pictures of dastardly drug peddlers luring in children with free drugs to get them hooked.
Turns out that drug dealers by and large prefer the straightforward approach of selling drugs to the huge market of people who want to take them.
...yes? It doesn't literally kill you if use more tokens than usual?
And we're talking about subscription plans from OpenAI/Anthropic here. Switching to a different model and/or harness if the price changes isn't a "the world has run out of opioids and I'm going to need a detox center" level crisis.
The original question was "are you sure an OpenAI subscription isn't heroin?" Different impacts of overuse (death vs slop code) feels like a valid point of difference.
This is a weirdly melodramatic analogy.
If you want to flatten everything to "all things habit forming are definitionally equivalent", you'll get some pretty funky results.
You're equivocating, they were talking about addiction, you're talking about overuse/overdose which is where the analogy breaks down for pretty obvious reasons.
If you're genuinely curious, I'd recommend searching around about the history of this persistent urban legend. It's really interesting.
But the short version is that there are lots of people who want to buy any given drug and drug dealers sell them. I don't know exactly what you mean by "WAIT for people to demand the more profitable drugs", but there typically isn't a whole lot of waiting. There's a lot of demand out there. Giving away a bunch of free fentanyl to random people in the hopes that one of them instantly gets addicted would be a pretty wasteful and high risk sales tactic, especially given the number of people that want to buy it already.
Personally at home I use the free github copilot and then openrouter when that runs out. Last month I used $0.90 of openrouter. Zero so far this month. When my bill starts hitting double digits I'll consider options.
Different matter at work but even there we are paying for API costs and have a monthly allowance (starts at $200/month but increases fairly easy to get)
You might not have a choice. For example, if you want to get a guarantee from Anthropic that they don't train bases on your prompts, you will need an enterprise deal, which is pay-by-token.
I'm with you there. I'm on expensive plans with everyone because I run out of tokens and rotate/use the models that are best in their niche...
CLI Model Pct Remaining Reset Date
claude all models 15% 3d
claude Fable 50% 3d
grok all models 0% 4d
agy Gemini 36% 5d
agy Claude and GPT models 0% 5d
codex codex 0% 6d
And I also just used two of my three get-another-week-free tokens for codex.
If you've got an agentic swarm, or are working on multiple big projects at once, or have agents parsing data... you burn through tokens fast. The last four days in API would have cost me well over $3k.
I'm always curious, what kind of software exactly are people building that requires multiple parallel agents and thousands of dollars per month in AI tokens per developer!
I think in many cases the token bloat comes from using a fleet of autonomous agents that "collaborate" over a problem. Using one coherent agent loop with a human involved might not feel as futuristic but it's more effective and efficient (right now).
We have two wildly different cohorts of people. Those who see these tools as a technological curiosity and those who see them as a means to a specific end. I know that if I finish a task with a certain set of constraints that I will get paid a certain amount of money. I'd much rather be HITL on one very good agent loop if there is real money on the table. If you are mostly having fun, being curious, or exploring (which is fine!) then the cost of subscriptions might feel safer. Running an IRL version of factorio can get very expensive.
I also think the API tokens perform better but I have zero evidence other than observing the quality of my work product relative to that of my peers. I have almost a religious fervor about using prepaid tokens now. I am still using GPT5.5 in my copilot with custom model configuration for most heavy lifting in visual studio. It is more than sufficient. My average monthly token cost is <$100. Paying for a subscription would be more expensive because I have a predictable and controllable amount of demand.
This is analogous to variable vs fixed rate electricity. Paying the real time wholesale price is a massive economic advantage if you can figure out how to control the demand side of the house.
> This is analogous to variable vs fixed rate electricity. Paying the real time wholesale price is a massive economic advantage if you can figure out how to control the demand side of the house.
rrr_oh_man is claiming that wholesale fixed rate electricity is generally cheaper than varible rate domestic electricity. Or that the wholesale-bit is higher order than the variable rate-bit. Not that it's a big issue, bob1029's point doesn't actually care about one situation being wholesale. It's just a somewhat narrow, (in this case) inconsequential, but also valid, nitpick.
I often read opinions where people say their particular method of more hands-on working with AI is the most efficient and reasonable way.
From 500 hours working on my mobile app project outside of work, I can confidently say that using anywhere between 2 to 10 conversations to parallelize work is much more effective in my project.
The models being served in the API are the same as via the subscription, there are no shenanigans here, particularly not concerning quality.
And GPT 5.5 was great at the time, I made heavy use of it for my project, but it's long outdated and already being retired.
You also mentioned using it in copilot, which is a third-rate harness that is frustratingly buggy and inefficient.
There's no comparison to Opus 5.5 in Claude Code.
Use whatever works for you, but frankly, I think you should try something new.
Just not finding this to hold true. The difference in work quality for peers isn’t a matter of how many threads they run, but how much attention they give each. Which is a skill.
> I know that if I finish a task with a certain set of constraints that I will get paid a certain amount of money.
So not salary… ok. That explains some of it. Getting paid by the task might flavor one’s approach. But also one could reason, more tasks, more pay… somewhat confusing.
> I also think the API tokens perform better but I have zero evidence other than observing the quality of my work product relative to that of my peers. I have almost a religious fervor about using prepaid tokens now
And this appears to be the other half. It looks no less futuristic than the more wireheaded approach, just different in form.
I think businesses are footing the majority of that bill right now due to regulatory requirements, but for individuals that price increase is def coming. Maybe I'm too pessimistic, but enjoy it while it lasts :')
For a lot of large scale day to day work, deepseek / zai models (the latter comes with a subscription) can be very cost effective and very competitive actually.
Another thing of note is that when coding at 'scale' to my surprise over time with memory aggregation across users/sessions these models can be better than the evals would have you believe because they seem to be differently/less (?) RL'ed (aka actually listen to instructions), which results in them being more fastidious about writing memories and benefiting from them, especially over time. These are of course mostly subjective opinions but they were surprising, and not what we were expecting.
Relevant: the proxy we wrote and use:
https://yapp.latchlabs.dev is partially open sourced: still working on releasing more of our memories stuff.
People using API keys heavily, are either making enough money from whatever they're doing to afford Anthropic's high prices, or are using a mix of locally hosted and cloud-based models. E.g. using a cloud model as a planner/orchestrator, with a local model as a subagent that follows the orchestrator's designs/instructions.
I had a subscription for a few months. I found that at the end of every week I would succumb to the sunk cost fallacy and try to squeeze all the usage out of my subscription. I'd rather pay a little extra in LLM cost to not have that issue.
> I still don't understand how people justify the cost of using API keys vs. subscription
Like a big white truck or a 600mm lens, expensive gear should be purchased by an employer. I think most folks talking about API access are getting it through work/self-employment.
Anthropic needs to retune their personal plans. Opus is the Amelia Bedelia of models. Maddening! I shelled out $100 for Fable to drive Opus and finish a personal project, but that's probably my last time loading up on usage credits. I put a second $100 towards API credits so I could dink around with API access and understand conversations like these.
I am on the 25 dollar plan and I never even come close to hitting the weekly limit. Still manage to ship quite a bit. Not sure how folks hit their Claude limits. I have handle on the codebase though.
There are tools like Paseo that call the existing CLIs meaning you can keep using the subsidized subscriptions while still having access to the json stream.
We (some friends and I) built it because we wanted to understand how these things worked and also had max accounts. It has since grown into a ton more, but is still in essence a very full featured local proxy.
For subscriptions, it is easier to collect the JSONL files from ~/.claude/projects/ and similar directories. They are better organized (e.g., subagents) than what the proxy sees.
However, the best value comes from collecting agent sessions across the whole organization. You'll need to scrub credentials and handle security.
What’s the best way to pull (ie archive) and search these across a few harnesses? I remember seeing a project or two trying to standardize them, but can’t find anything that seems to be a good sensible solution
Ask claude to do it end to end, then write a solution that fits your needs from it's experience with you driving it.
I did exactly this - probably the single most valuable piece of software I've "written". It's the backbone of every piece of code coming out of any agent under my control (via MCP toolset).
Mlflow does it pretty nice including session gathering and optional data enrichment (eg what files were touched, what model was used, git remote).
We use these session IDs for commenting on issues and stories touched by AI for context tracing.
I just wrapped around the Mlflow setup to make this simple CLI helper to set it up. Ofcourse the name suggests only Claude but it’s Codex, CC, GHCP and Agy. https://pypi.org/project/claudetracing/
GDPR doesn't cover unrecorded human memory, so you're fine to overhear whatever in a day-to-day setting. If you then do anything that looks like digital processing or cataloguing that information though, then you're making it a GDPR issue.
Unless you are a robot, you'll be fine. GDPR is about digitally recorded information. If you are a robot, it's probably best to delete that information.
If you have dyslexia and I know it, that brings it down to 20% of people. Combined with some other data, this is a pretty good factor to narrow it down to "who you are", right?
Isn't any personal data a PII? You just need a couple of personal o enable t identify individual. And if this is in an org then amount of individuals is quite small, so sometimes it needs just one personal detail.
Is a medical condition not PII? Together with the fact that it's linked to a specific person, their tasks, responsibilities, mannerisms. Even an IP address is PII, not sure how a full chat log isn't, even if it's not text-book PII.
That's not what GDPR is. I'm assuming you don't have the time or energy to research it, so the quick summary for you is that it does actually protect your privacy and gives you easy and fast avenues to acquire the data a company has about you (in a portable format), request deletion of your personal data, and requires that you explicitly consent to any processing of your data (in detail).
If a company doesn't follow these laws, the fines are proportional to the revenue(?) of the company, so the fines can get quite existential if a company fucks up.
In practice, this is not "shoved down" my throat at all. In practice, this means that I can request companies delete all my data, and they respect this, in practice. I can also request a .zip file with all data they have of me, and I will get it within a couple of days max.
It also means that e.g. your family doctor has to have you sign a document if they want to enter your data into a system, or if that system might be accessed by a 3rd party (such as IT support).
This isn't theoretical; this works. I've requested data deletion from various companies and requested data exports from various companies (and it worked without any issues every time).
It also means that, at any company I work at that handles PII, everyone is VERY careful touching anything with PII. The fact that you have to implement data retention periods, limit the amount of data you collect and retain, etc. forces companies to respect your privacy even if they don't want to or it's inconvenient.
Technically you can still run Claude-Code through proxies though, even with the subscription.
Right now I run my Claude-Code sessions through a very simple mitmproxy, logging everything that goes in & out. It's very low level, sure, but it works.
Tailscale makes Aperture available for free, and while it's not something you own, it is in your tailnet, so at least you have a certain amount of control over how things access it.
It gives a nice view of what agents are doing, with lots of configurability and extensions, and the team behind it has been updating pretty steadily.
I happen to be working on an extension to Aperture, so I have my own reason to promote it. But I do think it's a nice easy solution, and I promoted it to people before I was extending it.
I'm building a "bring your own data" layer for agentic work of all sorts, using the webhook method for now.
In the context of Aperture, I watch for agents looking for certain types/sources of data, and instead provide them with an in-tailnet API for accessing that data.
In a personal context, think email, calendar events, music playlists, fitness app records, transactions. I've also found it useful in a professional context, but I think I need to explore that a fair bit more.
It keeps credentials out of agents hands, and provides some relatively standard ways for controlling what agents can see and change.
I need a fair bit of hardening work before I try to get random folk to use it, but I might post something more here in the next week. :)
Posting in case it is useful to others - I've built a OSS Mac App that handles this across various providers and lets you examine usage in exacting detail.
LLMs work over a JSON API. You can read the JSON to see what's going on.
Using other people's harnesses I always got frustrated by not being able to see what the model sees. What are we actually sending it?
Context is pretty important when it comes to LLMs. Arguably, context is everything!
Usually the most straightforward way to see it, was to intercept the HTTP requests and dump them to a file.
This is usually trivial, just change the OPENAI_API_URL (or equivalent) to http://localhost:8080/ or whatever, which can be running a 20 line Python script, that just forwards the HTTP requests, and dumps them to a file.
It's pretty straightforward, although it really shouldn't be necessary in the first place.
I’m not sure what people are doing with these subscriptions to feel the need to monitor what’s going on this closely. A $100 OpenAI subscription feels effectively infinite for my usage at home
I've never touched my ISP-provided router. But maybe it's just how it works in my country (Malaysia), we pretty much only have 1 ISP with monopoly, so maybe that helps.
I let them install the free ISP router a year back when they installed fiber in the neighborhood. I assumed I would swap it out the next day, but I haven't had any issues like back in the day. I love shopping for a nice router as much as the next tech nerd, but decent drop-in home networking seems to be pretty commodity now days.
In similar spirit, the biggest motivation for me to set up DeepSeek Flash over Opus 5.5 would be seeing the reasoning traces, which is a mere curiosity.
> We need answer explain LLM reasoning traces. Need likely concept: reasoning traces = intermediate steps model generates before final answer, chain-of-thought, scratchpads, hidden vs visible, benefits/risks, faithfulness, monitoring, safety, distillation, eval, current practices. Need explain clearly. Desired oververbosity 5.
Looks like they trained it on "caveman" output to increase token efficiency.
Cache TTL will get you anyway. For Claude subscriptions, subagents still have a 5m TTL and before I discovered that, their constant waits for ~10 min tests or similar tasks would destroy my 5h usage. And for my API subscription at work, Claude has a 5m TTL for all stuff, and that's nowhere near long enough. You have to be monitoring this unless you want to absolutely torch your company's budget.
$100/mo. is basically the cost of a cable TV package. Often $120-170, actually.
If you're the kind of person who gets enjoyment being able to code all sorts of things you were never able to before, or 20x faster than you were able to before, then $100/mo. is extremely reasonable in comparison.
I'd argue it's money much better spent than on cable.
Also, if a hobby of yours runs you $100/mo., that's actually very much on the cheap side for a lot of hobbies.
Also, $20/mo. is for hobbyists using it for 2 hours at a time a few times a week. $100/mo. is for people working on a side project every evening and every full weekend.
1) What kind of guardrails are you using? Do you do models like prompt injection checkers? Or what guardrails are useful that do not know the environment the agent lives in?
2) I heard Bifrost also works for this. Did you try that as well, pros/cons?
I'm not the person you asked but yes pretty much. I've found the prompt injection protection models pretty limited, but get good results from gpt-oss-safeguard with a policy describing various prompt injection techniques. More expensive but more flexible and reliable and I only run it on web fetch and web search results. Costs me like 5-20 cents a day depending on what I'm working on.
Still working on deploying it and setting it up with cloudflare sandboxes so I don't have deep expertise and need to experiment first before I can give advice
Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?
This general phenomenon is not new, but it has been so obvious during 2026. Everyone want to consume everyone else hard work, to create their own slop. Just pure egoism.
Hi, I'm sorry you feel this way about LiteLLM. We're actively working on improving code quality / reducing bugs. Were there any specific pain points you faced with LiteLLM?
If you're open to a convo, I'd love to hear what specifically about the product was not great and improve it for you :)
Beacon is a harness-agnostic, environment-agnostic framework for capturing, normalizing, and storing the full session history of AI agents. We extend the OpenTelemetry GenAI standard and have done the heavy lifting of integrating with 25+ agent harnesses through a combination of OTel instrumentation, hooks, and local session database capture.
This gives you a unified view of agent activity across environments, including local agents, cloud agents, browser-based chat, and agent SDKs, with the ability to track and retain complete session histories across all of them.
OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.
How do you ditch Claude Code without paying for API tokens? (Assuming from the way you phrased it that you still use Claude, only you didn't want the system prompts that came with Claude Code.)
Last I looked into it, using other harnesses with a subscription was a TOS violation that Anthropic seemed to take quite seriously (with automatic rate limiting etc.)
Mostly using GLM/zai and pi agent these days. GLM also allows hermes on their subscription which is very useful.
If I do need Opus/whatever to look at a specific issue I just use openrouter API.
Can't say I particularly care for the US providers walled garden ToS restricted handholding suprise fallback nonsense even if their models are the strongest. Most of my tasks work just fine with lagging behind absolute SOTA so don't feel the need to tolerate it.
Until they quit subsidizing my personal tokens I don't think I want this. I do love having insight when I run qwen locally through Claude Code Router so I do agree it's so valuable to have, just not valuable enough to pay API rates lol
I don't do this but I do have a system that imports all my conversations with AI (claude code, chatgpt, claude app, etc.) to a storage system I call Chatacombs: a unified storage of most conversations I've had with AI, as detailed as I can easily get.
I agree its so important to proxy and capture all the data. I've been very happy with bifrost as a proxy. It's a quality piece of software, unlike LiteLLM. You also get this one interface to access all your models, across all services, including self-hosted, STT models, everything. There are ways to use subscriptions with proxies too, but they are discouraged because of grey-hat uses..
prob a noob question; but what kind of data are people looking to get to help with what kind of decisions with these tools?
i can see some IT/infra team using these tools to manage their LLM costs, but this blog post seems to hint at some kind of product decision making? what am i missing here?
It is a not proxy. It can clearly show whats going on across all agents. The proxy's are generally inconvenient and yet another point of failure. I think the author got to the resort option quickly
Just one info for many: if you’re a big company and have enterprise needs, subscriptions aren’t a solution. OpenAI, Anthropic, Google (ain’t sure about them tbh), GitHub Copilot aren’t offering subsidized subscription prices in their enterprise plans; discounts are calculated differently and these services are much more expensive.
For example, Github Copilot has these AI credits that are insane in terms of costs with the usual token-based pricing.
Enterprise plans are almost never subsidized. You are paying a premium to get what others are missing. Guarantee of data residency, SLA agreements, fast support etc.
Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing.
Don't really understand why other agents to give those insights but well found my solution.
what proxy is he using? i support not trying to support a specific product in a post but not even mentioning what you use after writing this seems pretty odd.
Umm, observability ? I think the article means to say how do you know the agents arent talking nonsense to each other if you use a multi-agent system. Here again a thin layer of o11y would do the trick and also hooking into the wider ecosystem. Thats without having to bother with a proxy.
Anyone know an easy, out of the box method of doing this inspection on DGXOS (the OS for DGX Spark devices)?
I'm a little miffed that this isn't just an immediate OOBE, but I'm prepared to do some nc/ssh tunnel magic if that'll work properly .. just wondering what other DGX users have vis a vis best practices ..
Because the details we care about right now already travelling over the network and the industry is adopting OTEL, where you can export and collate kernel things if you need to
there's the kind of people who care and the kind who doesn't ...
your local politicians and landlords do all kinds of normal, non-funky stuff you don't know about ... as do your local ISP-techies and hobbyists with access to supply chains and sub-distributing units ...
or your local school teachers and some parents and their widdle vintage schemes and circumambient tales they tell some kids but not others ... Lord, some guy even caught teachers and parents running schemes on teens to prep them for gay homo fag stuff, despair, frustration ... running for years ...
and you wonder how peeps don't what is going on xD?<
even those hackies and hackers and programmers and brogrammers and AI superusers who think they know and care and wield some power really really don't
I use GoModel and its amazing. My favorite features are virtual model, the support for arbitrary request labels, and the nice dashboads / ui. Creator has more sophisticated report generation on the near term roadmap. He's super responsive to feedback and ships fast. (except for his recent trip to the US west coast to meet founders and investors, he's EU based)
You can use and control all.your subscriptions or aoikeys through a gateway like this, so you don't get pwned like what just happened with OpenCode (or an OpenRouter like vendor). GoModel has some ambitions like this, but is open source so you can run it on you own infra, with plans for self hosted enterprise as well (there are a few features already like SSO). We are rolling out the free version at work right now.
I run GoModel at home and have hyper detailed history of every agent interaction, across models running on my sparks and OpenCode sub. You can call me a committed superfan of this project.
All that posturing for the purpose to be manually monitoring cache hits and misses. This can't be serious. Buy a larger plan or reduce some instructions and you don't have to deal with this waste of time.
With just a 12gb VRAM GPU + 64gb of CPU RAM, and you can actually own your own infrastructure. I can't wait for hardware prices to fall so that becomes even more accessible.
I am boggled how the first step of an llm after you make an orchestrator isn't some kind of proxy that records your personal internet surfing to be used by your personal AI later. Every college should be running in house AI with their own in house search engines and in house any service. Don't go to any college that wants you to use chatgpt or other corporate nonsense. If they can't make local LLMs work, they are not a real college.
You don't need a proxy to do this. I would argue you should not use a proxy to do this because the end result is a single point of failure direct in the line of your productivity.
Both Codex[0] and Claude Code[1] output OTEL traces that you can ingest and process.
Other harnesses like OpenCode, Pi, etc. you can hook into their plugin architecture and implement your own OTEL emission from a plugin (or use an off the shelf plugin[2]).
Host an OpenTelemetry collector[3][4] or rely on your existing collector infra (e.g. Datadog (no affiliation), AppInsights, etc.)
The OTEL traces will have exactly what you need. Tool calls, full prompts, token usage/cost, etc. It is less intrusive and does not become a single-point-of failure since the OTEL emission and collection is best-effort whereas a failing/overloaded proxy is going to strand your team until resolved. It's OK if you lose some OTEL traces; no productivity lost.
The configuration to enable this for Codex and CC is very straightforward (see [0][1] + [5])
--
[0] https://learn.chatgpt.com/docs/config-file/config-advanced#o...
[1] https://code.claude.com/docs/en/monitoring-usage
[2] https://github.com/DEVtheOPS/opencode-plugin-otel
[3] Open source collector: https://opentelemetry.io/docs/collector/
[4] Sample configuration for filtering noise: https://github.com/zeeq-ai/zeeq-app/blob/main/build/otel-col...
[5] Concrete examples of .codex/config.toml and .claude/settings.json: https://zeeq.ai/docs/key-features/session-telemetry#configur...
That shows what you do, not what the proxy does with your queries
If you are using a Datadog, Langfuse, Langchain, or other OTEL target, you do nothing. Just query it. Build alerts, dashboards, and datasets; whatever you normally do with your OTEL collector. A forward OTEL collector deployment is still useful for filtering but the Datadog Agent, for example, already has buffering and filtering built into it.
If you want to deploy your own collector and write your own handler (e.g. put it into a database), you can generate the stubs for the OTEL payloads and convert to domain records.
1. Get the protos here: https://github.com/open-telemetry/opentelemetry-proto
2. Generate the type/class stubs and endpoints for your platform.
3. I strongly recommend pre-filtering with the open source `opentelemetry-collector` before your custom endpoint handles the inbound telemetry payload. The collector is optimized for performing the filtering and I put it upstream of my custom ingest endpoint on a very small VM just to pre-filter and perform auth checks[0][1] on the inbound channel.
4. Do what you need to with the inbound payloads. Examples below:
- Ingest endpoint (Codex/Claude -> OTEL collector -> API): https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...
- Codex telemetry adapter: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...
- CC telemetry adapter: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...
I run my backend on Postgres with partitioned tables. You can forward to Elastic or other OLAP-type database targets for querying. You can see examples of what I do with the session telemetry here: https://zeeq.ai/docs/key-features/session-telemetry
[0] https://github.com/zeeq-ai/zeeq-app/blob/main/build/otel-col...
[1] https://github.com/open-telemetry/opentelemetry-collector-co...
Disclosure: I work on Origin [1]. We built a platform for enterprises that consumes OTEL, and other traces without a proxy or an LLM gateway. We even support collecting traces from local inference workloads.
For supported tools, we collect prompts, responses, and tool calls, then tie them to the user, organizational context and what actually happened on the machine. We can then use this data to detect when agents (or the people piloting them) do bad things [2].
--
[1] https://www.originhq.com/
[2] https://www.originhq.com/investigations
Wouldn't a fair criticism of OTEL be that you only see what these harnesses are willing to show you?
There have been numerous scandals of AI companies offering CC/cursor alternatives, then indexing data they never should have; specifically Z.ai [0], ByteDance Trae [1] and Alibaba Qwen [2]. Maybe the argument should be that all harnesses should be sandboxed and monitored for all networking.
If it were me, all AI harnesses would be running on a VM with network logging enabled. Beyond not being able to trust what data they send via the harness, you should audit what data they access.
[0] https://www.chosun.com/english/industry-en/2026/09/22/ABYLC3...
[1] https://github.com/segmentationf4u1t/trae_telemetry_research
[2] https://github.com/QwenLM/qwen-code/issues/11198
Well if you own the harness ( and you should ) you see everything you want.
But I’ve seen approaches using eBPF to collect telemetry to OTEL
Sometimes, it gets to very hacky solutions, such as what we did at Laminar with Claude Agent SDK. Basically we spin up a tiny in-process proxy that intercepts requests and responses and emits the telemetry as OTEL traces. We repoint the Python / JS SDK to convince the harness that our proxy is the anthropic URL, and the proxy then forward the request downstream. I wrote about it some time ago [1]. This is definitely not elegant, but it generally works.
[1] https://laminar.sh/blog/2025-12-03-claude-agent-sdk-instrume...
Otel is only a quarter of what the llm proxies solve for you. What happens when an agent goes down some random rabbit hole and burns through $100 before you even notice? It's nice to have post-disaster visibility with Otel, but even self hosting an LLM proxy prevents the disaster.
Shameless plug, I've been building Relay for this exact reason. You get real-time visibility into what your agents are doing, how much they're spending, and can set hard limits so things don't spiral out of control. Plus a lot of other features they bring to the table if you want.
It's open source and self-hostable too.
https://github.com/anchorshell/relay https://anchorshell.com/
You can do that with MCP or hooks as well.
You can build a proxy setup that is not a SPOF.
Sure, but OTEL has all of the right qualities for the majority of use cases here.
Standards based, many off-the-shelf providers, out-of-process + buffered by default, many standard processors for traces + spans, many tools for storing and querying.
OTEL is pretty widely supported and mostly "just works" at this stage of maturity. A failing/mis-configured OTEL endpoint/collector isn't going to stop you from getting work done in the same way that a failing proxy is. You'll drop some traces, but that's probably OK in most cases.
There is a place for proxies, but those places should share the same failure boundary mode. Meaning if the proxy fails, it should necessarily stop forward agent activity. Example: proxy injection of secrets instead of local acquisition.
I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription.
I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.
The current crop of flash models are more comparable to sonnet than opus but they are very affordable.
The reality is that the subscriptions are heavily subsidized. Pretty much every sub that's not run by one of the money-burning frontier labs has had to drop it or dramatically increase prices and lower limits in the last few months. Cheap tokens are an acquisition tool, not a real product. It sounds dramatic but it really is the drug dealer sales funnel.
At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.
>At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less. Better to get a handle on it now.
So just wait until that happens? What's the point of scrimping and saving now? Not to mention you're paying for a premium in the meantime.
I don’t disagree really and it is mostly what I am doing. I put probably like 20% of my work through open weights.
It does make me sharper I think. If you only ever use claude or whatever you don’t even really know exactly how tied to it you are.
Funny enough the drug dealer thing still holds. A deal’s a deal if you’re going to buy it anyway.
It's not obvious the subsidization will end. Loss leaders are a thing. Most enterprise at this point are paying API pricing
To better predict what they'll do with it we'd need to know the proportion of token usage via API vs subscription. Keeping an affordable, slightly unprofitable subscription to get people in isn't a bad idea. They also get marketing benefits from the word of mouth
Also, some subset of developers on subscriptions are going to develop products that use the API, and some of those products are going to be successful.
In the optimistic case you describe, what smart customers want to watch out for is being lured into vibing your projects into such byzantine slop-complexity that subscription plans and trailing models are no longer sufficient to make progress on them.
Vibe-coders talk about surrendering understanding to the models, trusting that there will always be a model able to overcome and work through any technical debt they happen to accumulate, but the danger that still lurks there (at least for the naive) is that you'll need increasingly large contexts, increasingly sophisticated models, and increasingly large token spends just to keep that ball rolling.
Between profitable enterprise accounts being ineligible for subscription, eager but busy people paying for underutilized subscriptions, and naive vibe-coders falling into their pit trap, you can indeed imagine subsidized subscription accounts being available for a long time and remaining an effective value hack for people careful enough not to fall into the pit trap. But only insofar as they are that careful.
Also they can always find another way to make money. Make subscription users product and sell some analytics like "show most trending project being built this week".
You cannot predict the future, but you can identify risks and attribute a likelihood. A risk is that the subsidized inference will either end or be reduced significantly in the next 1-2y. If you think the likelihood is high enough it makes sense to ensure your systems are prepared for it.
That’s risk management 101
> It's not obvious the subsidization will end. Loss leaders are a thing.
Loss leaders are a thing so long as they actually do lead somewhere. If your primary value proposition is conveyed by the loss leader itself, where do you go from there?
(meta commentary) really gnarly word choice: "drop" in "they're going to drop the real prices in your lab"?
I took your meaning as "they're gonna drop [a bad surprise on you by raising] the real prices"... so at least it's not quite as tortured as the ambiguous "support for version N dropped today" (where you often can't even infer the direction from context).
this isn't just a tangent, I'm having trouble understanding the point you're trying to make, which hinges entirely on deciding if you meant "drop" literally
They meant "they're going to drop a bad surprise by charging you API prices", so yes, a better wording would be "they're going to give you a bad surprise in the form of forced API pricing".
The term "drop" becoming a synonym for "release", "put out" or "put into effect" for things outside of music/albums is one of the most confusing linguistic innovations of the last decade.
“Lab” is a typo, it should be “lap”.
“Drop [an unwelcome surprise] in your lap” is a common English idiom.
> At some point they're going to drop the real prices in your lab
The real price for tokens at a specific level of intelligence is dropping rapidly. Between models improving and new hardware getting faster, I think prices will be going down, not up.
> Better to get a handle on it now.
I experiment with a lot of providers and models. I strongly disagree that anyone needs to start migrating now in preparation for prices maybe going up in the future.
It doesn’t make sense. Deal with a price increase when it happens, if it does.
Switching models and providers isn’t some huge lift. I can jump to a new model or even hardness and get a feel for it in a day or two. It’s not some huge undertaking that requires preparation and planning.
If prices go up, try something at that point in time. Don’t worry about it until then.
When Claude Code subscriptions really started to catch on, there was a popular idea on HN that Anthropic's end game was that basically every dev adjacent role would need to pay for a $200/mo coding plan.
Then, both Anthropic and OpenAI in lockstep developed a enterprise pricing structure that openly coerced large businesses to pay API prices where it was feasible to spend $200/day rather than the comparatively cheap $200/mo.
Even if token pricing relative to intelligence does trend down over time, that doesn't mean the pricing structure will actually translate to lower costs, particularly in the cash cow B2B market.
Its insane, I don't possibly understand how they convinced companies to pay 30x
What was the alternative? It takes time for the open weight ecosystem to develop.
Companies still believe they have IP to protect. Even when all they do are basic CRUD systems or on the verge of collapsing if they don’t renew some contracts.
From experience, if you work somewhere with very disconnected higher ups a sales person shows what your token usage is vs spend on the monthly subs for whatever set of users you have and then show an API price with a sample bill for even “more users” at that same number of tokens that is 30% lower. The crux of course being that your existing user base is not going to use fewer toke a just because you signed more people up and have a discount for the status quo.
Then when you consolidate all users to a single provider you find out you have a 2x to 3x larger bill because you… logically have more users at the same average token spend than the quote.
At my previous job, all while managing a full team, I did a lot of porting of legacy apps to modern standards to maintain various obligations. It was AI gen work in between meetings and paperwork. It was seen as such a success that I was asked to move to an IC role to focus on it within the “Elite new products team” that was supposed to be all AI and the latest tech. I showed up to find that they were pushing out low quality manually written code in 12 hour days six days per week to appear like LLMs were in use to save face because they lacked a token budget or the skills to properly manage one under the APIs. They had dismissed someone who had spent multiple thousands per month on Agent swarms. It was the perfect situation to make their 3x budget issue a 10-30x budget spend issue.
For many many reasons I left that job and got a new one but that was the cherry on top.
I see you are not familiar with enterprise IT hardware and software purchasing...
The API/subscription split as always there. I don’t know why you’re talking about it as a bait-and-switch development.
That doesn’t change anything about what I said, though. The relative price per task or price per unit intelligence is also going down in the APIs too.
> Even if token pricing relative to intelligence does trend down over time, that doesn't mean the pricing structure will actually translate to lower costs, particularly in the cash cow B2B market.
What are you talking about? The APIs serve the same models as the subscriptions. Unless you think the price of APIs has been going up (it has not) this argument doesn’t make sense
> At some point they're going to drop the real prices in your lab and you're going to have to figure out how to pay that much or pay someone else a lot less.
This only applies under the assumption that everyone tokenmaxxes their subscription.
I have visibility across a few different Claude team subscriptions. I'd say roughly 10% of people max their subscriptions.
I think I've only seen one organization where the under-used subs don't more than pay for the tokenmaxxed ones.
And that is ignoring the 90% margin that API inference runs at.
At some point companies will need to have the “scary” conversation about the value proposition of adding another human to the team, or adding a comparable amount of token spend to the team.
$1k/person sounds like a lot, but a ten person team at $1k token burn / head / month is spending $120k. How much does a new hire cost, tco? Does velocity from one more person compare with velocity from one more person’s worth of tokens?
Sure, learning to be efficient matters. So does taking the time to consider cost vs alternative. You might be surprised by what you find, in either direction.
Counter point - models get better, ways to run them gets optimized and hardware will eventually come down in price.
The share of tasks where "cheaper" models are enough also increases for same reason. For majority of use cases it will quickly become race to the bottom. And that means part of the pie for the "premium" models of frontier AI labs will quickly become smaller and smaller.
The drug dealer gives you a free hit because it is profitable for him in the long term for you to be addicted to his product, not because he is your friend.
Another way to look at it is they are running a business. It's not even close to gambling for example.
Running businesses at massive losses with the expectation of making that money back by being _the_ monopoly on this technology.
It's weird to me how often this analogy gets used because:
1. In most cases, like this one, the product isn't heroin.
2. Drug dealers don't actually do that.
What? (1) Yes.. it's never that [stupid economics, they don't want their purchasers to die.. might as well sell fentanyl], (2) it's always that [hardly any way to propose alternatives.. within the scope of not killing the customer]
I'm not sure exactly what you're implying but if you're curious, there's a lot of writing about the persistence of this particular urban legend. A lot of it starts with 1920s newspapers painting pictures of dastardly drug peddlers luring in children with free drugs to get them hooked.
Turns out that drug dealers by and large prefer the straightforward approach of selling drugs to the huge market of people who want to take them.
Are you sure the product isn't heroin? If we take it away, there's a bunch of people around here at least that will definitely experience withdrawal.
...yes? It doesn't literally kill you if use more tokens than usual?
And we're talking about subscription plans from OpenAI/Anthropic here. Switching to a different model and/or harness if the price changes isn't a "the world has run out of opioids and I'm going to need a detox center" level crisis.
You're describing the impact of overuse or moving to another dealer, not withdrawing from it altogether ;)
The original question was "are you sure an OpenAI subscription isn't heroin?" Different impacts of overuse (death vs slop code) feels like a valid point of difference.
This is a weirdly melodramatic analogy.
If you want to flatten everything to "all things habit forming are definitionally equivalent", you'll get some pretty funky results.
You're equivocating, they were talking about addiction, you're talking about overuse/overdose which is where the analogy breaks down for pretty obvious reasons.
> "1. In most cases, like this one, the product isn't heroin."
I'm not sure about that. Someone using AI for productivity every day will experience massive issues if you suddenly take it away.
It's even the whole business model of MS Copilot: They don't work with companies to have their data structured in the past or going forward.
They don't help you to structure your business "life", they offer a tool you consume to make sense of it while you use it.
The moment you stop using it, you'll struggle to even make sense of your own past actions.
> "2. Drug dealers don't actually do that."
It would be a weird business if they only WAIT for people to demand the more profitable drugs...
If you're genuinely curious, I'd recommend searching around about the history of this persistent urban legend. It's really interesting.
But the short version is that there are lots of people who want to buy any given drug and drug dealers sell them. I don't know exactly what you mean by "WAIT for people to demand the more profitable drugs", but there typically isn't a whole lot of waiting. There's a lot of demand out there. Giving away a bunch of free fentanyl to random people in the hopes that one of them instantly gets addicted would be a pretty wasteful and high risk sales tactic, especially given the number of people that want to buy it already.
They are betting on people don't max out their subscription.
Someone who pays $20 a month may only use 10% of their max budget.
Personally at home I use the free github copilot and then openrouter when that runs out. Last month I used $0.90 of openrouter. Zero so far this month. When my bill starts hitting double digits I'll consider options.
Different matter at work but even there we are paying for API costs and have a monthly allowance (starts at $200/month but increases fairly easy to get)
This is true for pretty much every fixed price subscription service in existence
I honestly feel a bit worried Anthropic/OpenAI are going to drop me since I consistently hit my limits
You might not have a choice. For example, if you want to get a guarantee from Anthropic that they don't train bases on your prompts, you will need an enterprise deal, which is pay-by-token.
I'm with you there. I'm on expensive plans with everyone because I run out of tokens and rotate/use the models that are best in their niche...
CLI Model Pct Remaining Reset Date claude all models 15% 3d claude Fable 50% 3d grok all models 0% 4d agy Gemini 36% 5d agy Claude and GPT models 0% 5d codex codex 0% 6d
And I also just used two of my three get-another-week-free tokens for codex.
If you've got an agentic swarm, or are working on multiple big projects at once, or have agents parsing data... you burn through tokens fast. The last four days in API would have cost me well over $3k.
I'm always curious, what kind of software exactly are people building that requires multiple parallel agents and thousands of dollars per month in AI tokens per developer!
I think in many cases the token bloat comes from using a fleet of autonomous agents that "collaborate" over a problem. Using one coherent agent loop with a human involved might not feel as futuristic but it's more effective and efficient (right now).
We have two wildly different cohorts of people. Those who see these tools as a technological curiosity and those who see them as a means to a specific end. I know that if I finish a task with a certain set of constraints that I will get paid a certain amount of money. I'd much rather be HITL on one very good agent loop if there is real money on the table. If you are mostly having fun, being curious, or exploring (which is fine!) then the cost of subscriptions might feel safer. Running an IRL version of factorio can get very expensive.
I also think the API tokens perform better but I have zero evidence other than observing the quality of my work product relative to that of my peers. I have almost a religious fervor about using prepaid tokens now. I am still using GPT5.5 in my copilot with custom model configuration for most heavy lifting in visual studio. It is more than sufficient. My average monthly token cost is <$100. Paying for a subscription would be more expensive because I have a predictable and controllable amount of demand.
This is analogous to variable vs fixed rate electricity. Paying the real time wholesale price is a massive economic advantage if you can figure out how to control the demand side of the house.
> This is analogous to variable vs fixed rate electricity. Paying the real time wholesale price is a massive economic advantage if you can figure out how to control the demand side of the house.
Wholesale electricity is almost always cheaper.
Sorry, I have to be that guy... https://xkcd.com/386/
Aren’t you both saying the same thing?
I didn't express myself correctly.
The analogy is false.
Domestic electricity is priced per kWh, wholesale electricity is priced per kWh, the latter is, as a rule, cheaper. It is not fixed vs variable rate.
> Domestic electricity is priced per kWh
> wholesale electricity is priced per kWh
> the latter is, as a rule, cheaper.
I think you’re missing some detail. Why do two things that are “priced per kilowatt hour” have to have different prices?
Possibly we are talking about different things. I am not prescribing or postulating anything. This is just reality.
Source: https://www.eia.gov/electricity/monthly/update/end-use.php
The reason is mostly higher consumption = lower average price.
Isn't there a time of day variable as well? That would further drive down the average for e.g, Industrial sites that run continuously.
Well that's going to be obscured by an average.
Depends on the industrial or commercial contract, but yes.
Also my state recently mandated all residents be placed on time-of-use plans of some form.
rrr_oh_man is claiming that wholesale fixed rate electricity is generally cheaper than varible rate domestic electricity. Or that the wholesale-bit is higher order than the variable rate-bit. Not that it's a big issue, bob1029's point doesn't actually care about one situation being wholesale. It's just a somewhat narrow, (in this case) inconsequential, but also valid, nitpick.
I often read opinions where people say their particular method of more hands-on working with AI is the most efficient and reasonable way.
From 500 hours working on my mobile app project outside of work, I can confidently say that using anywhere between 2 to 10 conversations to parallelize work is much more effective in my project.
The models being served in the API are the same as via the subscription, there are no shenanigans here, particularly not concerning quality.
And GPT 5.5 was great at the time, I made heavy use of it for my project, but it's long outdated and already being retired.
You also mentioned using it in copilot, which is a third-rate harness that is frustratingly buggy and inefficient.
There's no comparison to Opus 5.5 in Claude Code.
Use whatever works for you, but frankly, I think you should try something new.
The models are the same but the system prompt isn't.
Just not finding this to hold true. The difference in work quality for peers isn’t a matter of how many threads they run, but how much attention they give each. Which is a skill.
> I know that if I finish a task with a certain set of constraints that I will get paid a certain amount of money.
So not salary… ok. That explains some of it. Getting paid by the task might flavor one’s approach. But also one could reason, more tasks, more pay… somewhat confusing.
> I also think the API tokens perform better but I have zero evidence other than observing the quality of my work product relative to that of my peers. I have almost a religious fervor about using prepaid tokens now
And this appears to be the other half. It looks no less futuristic than the more wireheaded approach, just different in form.
I think businesses are footing the majority of that bill right now due to regulatory requirements, but for individuals that price increase is def coming. Maybe I'm too pessimistic, but enjoy it while it lasts :')
For a lot of large scale day to day work, deepseek / zai models (the latter comes with a subscription) can be very cost effective and very competitive actually.
Another thing of note is that when coding at 'scale' to my surprise over time with memory aggregation across users/sessions these models can be better than the evals would have you believe because they seem to be differently/less (?) RL'ed (aka actually listen to instructions), which results in them being more fastidious about writing memories and benefiting from them, especially over time. These are of course mostly subjective opinions but they were surprising, and not what we were expecting.
Relevant: the proxy we wrote and use: https://yapp.latchlabs.dev is partially open sourced: still working on releasing more of our memories stuff.
People using API keys heavily, are either making enough money from whatever they're doing to afford Anthropic's high prices, or are using a mix of locally hosted and cloud-based models. E.g. using a cloud model as a planner/orchestrator, with a local model as a subagent that follows the orchestrator's designs/instructions.
I had a subscription for a few months. I found that at the end of every week I would succumb to the sunk cost fallacy and try to squeeze all the usage out of my subscription. I'd rather pay a little extra in LLM cost to not have that issue.
Asked Pi what I used last 3 days.
I am on $100/m sub.
Oct 16 I switched machine
Excludes work done on local and Claude Chrome plugin.
> I am on $100/m sub.
You spent $720 in four days on a $100 per month subscription? So that table isn't correct?
Do you mean the table shows what it'd have cost you if you weren't on the $100/m subscription?
I understand this as: he has a fixed rate $100/m subscription. Prices are what he would pay if he paid API rates.
> Do you mean the table shows what it'd have cost you if you weren't on the $100/m subscription?
Yep, that. There's tools out there that will add up your token usage and then estimate what it would be at API pricing. That's likely what they used.
Any org with more than 150 employees needs to pay API prices for the enterprise subscription.
> I still don't understand how people justify the cost of using API keys vs. subscription
Like a big white truck or a 600mm lens, expensive gear should be purchased by an employer. I think most folks talking about API access are getting it through work/self-employment.
Anthropic needs to retune their personal plans. Opus is the Amelia Bedelia of models. Maddening! I shelled out $100 for Fable to drive Opus and finish a personal project, but that's probably my last time loading up on usage credits. I put a second $100 towards API credits so I could dink around with API access and understand conversations like these.
I am on the 25 dollar plan and I never even come close to hitting the weekly limit. Still manage to ship quite a bit. Not sure how folks hit their Claude limits. I have handle on the codebase though.
There are tools like Paseo that call the existing CLIs meaning you can keep using the subsidized subscriptions while still having access to the json stream.
Check out Lattis, getlattis.ai
We (some friends and I) built it because we wanted to understand how these things worked and also had max accounts. It has since grown into a ton more, but is still in essence a very full featured local proxy.
Proxies are good if you pay API prices.
For subscriptions, it is easier to collect the JSONL files from ~/.claude/projects/ and similar directories. They are better organized (e.g., subagents) than what the proxy sees.
However, the best value comes from collecting agent sessions across the whole organization. You'll need to scrub credentials and handle security.
What’s the best way to pull (ie archive) and search these across a few harnesses? I remember seeing a project or two trying to standardize them, but can’t find anything that seems to be a good sensible solution
I've used https://github.com/kenn-io/agentsview and found it to be quite sensible. Main friction point for me is I want it to be easier to extend.
Thanks will check it out
Don't install a second harness. You only need one to begin with!
I mean Claude and codex both have their uses for me and I’m one of the deranged running my own inferencing box with Qwen Next so I need pi for it.
Ask claude to do it end to end, then write a solution that fits your needs from it's experience with you driving it.
I did exactly this - probably the single most valuable piece of software I've "written". It's the backbone of every piece of code coming out of any agent under my control (via MCP toolset).
That’s like half a day to get to sensible useful defaults that everybody needs. Any chance you’d be up to push what you got to git?
I think that T3 Code tries to do something like this, though I do not use it and have not used it, and I have heard good things about this.
I've been using https://getmaestro.ai/ for 3 months now and I like it. Good founders.
No archival, but simply for searching https://github.com/raine/claude-history
Mlflow does it pretty nice including session gathering and optional data enrichment (eg what files were touched, what model was used, git remote). We use these session IDs for commenting on issues and stories touched by AI for context tracing.
I just wrapped around the Mlflow setup to make this simple CLI helper to set it up. Ofcourse the name suggests only Claude but it’s Codex, CC, GHCP and Agy. https://pypi.org/project/claudetracing/
https://github.com/CauchyIO/claudetracing
I'm the author of Pharos, a tool for just this use case! I've been using it extensively for personal use
https://grady.dev/projects/pharos/
If you're in the EU, be aware that the chats might contain PII. Even if someone says something like "Sorry I have dyslexia" in a chat, that's PII.
So id be very careful with chats and treat them like chats, not like code.
Out of curiosity, what if someone says “I have dyslexia” into their dictation app in the next desk over?
Is my brain now subject to GDPR regulations?
Not an attack on you, but my word am I thankful I don’t participate in the European regulatory regime.
GDPR doesn't cover unrecorded human memory, so you're fine to overhear whatever in a day-to-day setting. If you then do anything that looks like digital processing or cataloguing that information though, then you're making it a GDPR issue.
Unless you are a robot, you'll be fine. GDPR is about digitally recorded information. If you are a robot, it's probably best to delete that information.
So you don't think if employers collect private data about employees there should be limits on what they can do with that?
not an attack on you, but I am thankful to have guardrails for greedy scrupulous corporations
And I write that I have ADHD it's suddenly medical information?
That's actually one of my skills lol
Yes...? Is it not? If you write that you're gay, or muslim, or christian, or even just straight, all of that is especially sensitive under GDPR.
You’re conflating PII with personal data.
this guy GDPRs
If you have dyslexia and I know it, that brings it down to 20% of people. Combined with some other data, this is a pretty good factor to narrow it down to "who you are", right?
Claude often addresses me by name in chats.
Isn't any personal data a PII? You just need a couple of personal o enable t identify individual. And if this is in an org then amount of individuals is quite small, so sometimes it needs just one personal detail.
Read Article 4. "PII" is not a thing in the GDPR.
Is a medical condition not PII? Together with the fact that it's linked to a specific person, their tasks, responsibilities, mannerisms. Even an IP address is PII, not sure how a full chat log isn't, even if it's not text-book PII.
PII is not a thing under the GDPR. Like, at all. Personal data is. Personal data and PII are not the same thing.
Is dyslexia PII or PHI?
Thank god I'm not in the EU where regulation is shoved down my throat by the government in order to make me feel like they're protecting me.
That's not what GDPR is. I'm assuming you don't have the time or energy to research it, so the quick summary for you is that it does actually protect your privacy and gives you easy and fast avenues to acquire the data a company has about you (in a portable format), request deletion of your personal data, and requires that you explicitly consent to any processing of your data (in detail).
If a company doesn't follow these laws, the fines are proportional to the revenue(?) of the company, so the fines can get quite existential if a company fucks up.
In practice, this is not "shoved down" my throat at all. In practice, this means that I can request companies delete all my data, and they respect this, in practice. I can also request a .zip file with all data they have of me, and I will get it within a couple of days max.
It also means that e.g. your family doctor has to have you sign a document if they want to enter your data into a system, or if that system might be accessed by a 3rd party (such as IT support).
This isn't theoretical; this works. I've requested data deletion from various companies and requested data exports from various companies (and it worked without any issues every time).
It also means that, at any company I work at that handles PII, everyone is VERY careful touching anything with PII. The fact that you have to implement data retention periods, limit the amount of data you collect and retain, etc. forces companies to respect your privacy even if they don't want to or it's inconvenient.
It's incredible how much regulations the EU has to stifle the smallest of projects. No wonder they're so behind in the AI race
Technically you can still run Claude-Code through proxies though, even with the subscription.
Right now I run my Claude-Code sessions through a very simple mitmproxy, logging everything that goes in & out. It's very low level, sure, but it works.
Are there mamy organizations that have subscriptions? Anything above 150 people is forced into API pricing.
Something like 99% of US businesses have less than 100 employees.
Tailscale makes Aperture available for free, and while it's not something you own, it is in your tailnet, so at least you have a certain amount of control over how things access it. It gives a nice view of what agents are doing, with lots of configurability and extensions, and the team behind it has been updating pretty steadily.
I happen to be working on an extension to Aperture, so I have my own reason to promote it. But I do think it's a nice easy solution, and I promoted it to people before I was extending it.
Aperture supports extensions? What sort are you working on?
They have "connectors" you can configure for a variety of services, and both specific external tools and custom webhooks: https://tailscale.com/docs/aperture/integrate
I'm building a "bring your own data" layer for agentic work of all sorts, using the webhook method for now.
In the context of Aperture, I watch for agents looking for certain types/sources of data, and instead provide them with an in-tailnet API for accessing that data.
In a personal context, think email, calendar events, music playlists, fitness app records, transactions. I've also found it useful in a professional context, but I think I need to explore that a fair bit more.
It keeps credentials out of agents hands, and provides some relatively standard ways for controlling what agents can see and change.
I need a fair bit of hardening work before I try to get random folk to use it, but I might post something more here in the next week. :)
Posting in case it is useful to others - I've built a OSS Mac App that handles this across various providers and lets you examine usage in exacting detail.
The App: https://github.com/Pythia-Software/pharos Blog Post: https://grady.dev/projects/pharos/
One man’s fun is another man’s “idk what any of these words mean, I pay $20/mo for some company to abstract this complexity away”
For clarity, I do like blog posts like these.
LLMs work over a JSON API. You can read the JSON to see what's going on.
Using other people's harnesses I always got frustrated by not being able to see what the model sees. What are we actually sending it?
Context is pretty important when it comes to LLMs. Arguably, context is everything!
Usually the most straightforward way to see it, was to intercept the HTTP requests and dump them to a file.
This is usually trivial, just change the OPENAI_API_URL (or equivalent) to http://localhost:8080/ or whatever, which can be running a 20 line Python script, that just forwards the HTTP requests, and dumps them to a file.
It's pretty straightforward, although it really shouldn't be necessary in the first place.
I’m not sure what people are doing with these subscriptions to feel the need to monitor what’s going on this closely. A $100 OpenAI subscription feels effectively infinite for my usage at home
Some people are fine with an ISP-provided router that just works, and some would rather run pfSense, if even just to look at the internals.
>ISP-provided router that just works
I don't remember the last time this was true. These things don't have a 'just works' mode.
I've never touched my ISP-provided router. But maybe it's just how it works in my country (Malaysia), we pretty much only have 1 ISP with monopoly, so maybe that helps.
Maybe. In general though I think the speeds have improved and needs have remained the same giving the illusion that these things are good enough.
Unifi don't have a monopoly.
I let them install the free ISP router a year back when they installed fiber in the neighborhood. I assumed I would swap it out the next day, but I haven't had any issues like back in the day. I love shopping for a nice router as much as the next tech nerd, but decent drop-in home networking seems to be pretty commodity now days.
Maybe things have improved and our needs have remained the same giving the illusion that the ISP supplied routers are good enough.
If things have improved and needs have remained the same then it’s not an illusion that the routers are good enough.
Exactly how I feel about the two major AI compute dealers.
In similar spirit, the biggest motivation for me to set up DeepSeek Flash over Opus 5.5 would be seeing the reasoning traces, which is a mere curiosity.
They're pretty funny, actually:
> We need answer explain LLM reasoning traces. Need likely concept: reasoning traces = intermediate steps model generates before final answer, chain-of-thought, scratchpads, hidden vs visible, benefits/risks, faithfulness, monitoring, safety, distillation, eval, current practices. Need explain clearly. Desired oververbosity 5.
Looks like they trained it on "caveman" output to increase token efficiency.
From what I saw they look like this for all models.
Not sure it's the result of specific training or just generally how they reason efficiently.
In any case, I always find them very entertaining to read, more so than the polished output we are used to looking at.
It was also interesting in the HF incident write-up or other research like "Hacker Opus", which said this gem:
Cache TTL will get you anyway. For Claude subscriptions, subagents still have a 5m TTL and before I discovered that, their constant waits for ~10 min tests or similar tasks would destroy my 5h usage. And for my API subscription at work, Claude has a 5m TTL for all stuff, and that's nowhere near long enough. You have to be monitoring this unless you want to absolutely torch your company's budget.
I think people who monitor their usage like this are probably running $20-30 subscriptions or PAYG API pricing.
$100 a month on an LLM subscription for hobby usage is an absurd amount of money IMO.
$100/mo. is basically the cost of a cable TV package. Often $120-170, actually.
If you're the kind of person who gets enjoyment being able to code all sorts of things you were never able to before, or 20x faster than you were able to before, then $100/mo. is extremely reasonable in comparison.
I'd argue it's money much better spent than on cable.
Also, if a hobby of yours runs you $100/mo., that's actually very much on the cheap side for a lot of hobbies.
Also, $20/mo. is for hobbyists using it for 2 hours at a time a few times a week. $100/mo. is for people working on a side project every evening and every full weekend.
Quality and troubleshooting. This is a complex setup and there are plenty of places where things can go wrong.
"hacker" news… Even in the AI era, some people still enjoy understanding the technicalities of things
I think it was a great blog post.
Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around.
I can't orchestrate agents in sandboxes without it.
1. It can issue virtual keys, so provider API keys never go near the sandbox
2. I can see everything
3. Guardrails
And much more
Looking forward to it.
1) What kind of guardrails are you using? Do you do models like prompt injection checkers? Or what guardrails are useful that do not know the environment the agent lives in?
2) I heard Bifrost also works for this. Did you try that as well, pros/cons?
I'm not the person you asked but yes pretty much. I've found the prompt injection protection models pretty limited, but get good results from gpt-oss-safeguard with a policy describing various prompt injection techniques. More expensive but more flexible and reliable and I only run it on web fetch and web search results. Costs me like 5-20 cents a day depending on what I'm working on.
Still working on deploying it and setting it up with cloudflare sandboxes so I don't have deep expertise and need to experiment first before I can give advice
Hi i'm a LiteLLM maintainer thanks for using us, I'd love to connect to hear how we can improve for you. Here's my Linkedin https://www.linkedin.com/in/reffajnaahsi/ / Twitter (https://x.com/ishaan_jaff) if you're open to a convo
Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?
Bifrost has been decent, despite being partially shareware
Imagine complaining about “buggy vibecoded messes” that you want to use to produce your own buggy vibe coded mess.
This general phenomenon is not new, but it has been so obvious during 2026. Everyone want to consume everyone else hard work, to create their own slop. Just pure egoism.
Hi, I'm sorry you feel this way about LiteLLM. We're actively working on improving code quality / reducing bugs. Were there any specific pain points you faced with LiteLLM?
If you're open to a convo, I'd love to hear what specifically about the product was not great and improve it for you :)
My [LinkedIn](https://www.linkedin.com/in/reffajnaahsi/) if you're open to a convo.
We've tackled the core problem of agent telemetry normalization with our open-source project Beacon! https://github.com/Asymptote-Labs/agent-beacon
Beacon is a harness-agnostic, environment-agnostic framework for capturing, normalizing, and storing the full session history of AI agents. We extend the OpenTelemetry GenAI standard and have done the heavy lifting of integrating with 25+ agent harnesses through a combination of OTel instrumentation, hooks, and local session database capture.
This gives you a unified view of agent activity across environments, including local agents, cloud agents, browser-based chat, and agent SDKs, with the ability to track and retain complete session histories across all of them.
Would love your feedback!
Want to shamelessly self promote my self hosted proxy, supports subs and API usage.
Primaily designed as a local LLM ops project, but will definitely do proxying for claude-code, opencode, codex, etc
https://llama-dash.dev
OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.
My org is using https://visiqlabs.com/ for this. not a proxy, agent integration with rules engine
Toyed around with it back when claude code showed up, but never really got any useful actionable insights from it so stopped
(Aside from ditching CC & it's bloated sys prompts that is)
we use it to assign cost/project level analytics which is actually quite useful!
How do you ditch Claude Code without paying for API tokens? (Assuming from the way you phrased it that you still use Claude, only you didn't want the system prompts that came with Claude Code.)
Last I looked into it, using other harnesses with a subscription was a TOS violation that Anthropic seemed to take quite seriously (with automatic rate limiting etc.)
Mostly using GLM/zai and pi agent these days. GLM also allows hermes on their subscription which is very useful.
If I do need Opus/whatever to look at a specific issue I just use openrouter API.
Can't say I particularly care for the US providers walled garden ToS restricted handholding suprise fallback nonsense even if their models are the strongest. Most of my tasks work just fine with lagging behind absolute SOTA so don't feel the need to tolerate it.
Until they quit subsidizing my personal tokens I don't think I want this. I do love having insight when I run qwen locally through Claude Code Router so I do agree it's so valuable to have, just not valuable enough to pay API rates lol
I do know checking both agent session and logging at the inference side...
I don't do this but I do have a system that imports all my conversations with AI (claude code, chatgpt, claude app, etc.) to a storage system I call Chatacombs: a unified storage of most conversations I've had with AI, as detailed as I can easily get.
“Chatacombs” is a fantastic name.
Is was one of Claude’s suggestions. A thumping good one
Do you have it all scrapped via. on-device apps or does it scrape the browser (for chat convos, not for agent/harnesses)
A browser extension gets the web stuff.
I like the content but I have to say the Mark Manson writing style is off putting.
Even worse if he had an LLM write this
I agree its so important to proxy and capture all the data. I've been very happy with bifrost as a proxy. It's a quality piece of software, unlike LiteLLM. You also get this one interface to access all your models, across all services, including self-hosted, STT models, everything. There are ways to use subscriptions with proxies too, but they are discouraged because of grey-hat uses..
prob a noob question; but what kind of data are people looking to get to help with what kind of decisions with these tools?
i can see some IT/infra team using these tools to manage their LLM costs, but this blog post seems to hint at some kind of product decision making? what am i missing here?
As long the harness is not open source a proxy won't help, it's easy to phone home on a different subdomain also.
https://github.com/gumieri/nenya
my own ai gateway/proxy
This is the best one I found https://github.com/kenn-io/agentsview
It is a not proxy. It can clearly show whats going on across all agents. The proxy's are generally inconvenient and yet another point of failure. I think the author got to the resort option quickly
Just one info for many: if you’re a big company and have enterprise needs, subscriptions aren’t a solution. OpenAI, Anthropic, Google (ain’t sure about them tbh), GitHub Copilot aren’t offering subsidized subscription prices in their enterprise plans; discounts are calculated differently and these services are much more expensive.
For example, Github Copilot has these AI credits that are insane in terms of costs with the usual token-based pricing.
Feel free to prove me wrong, I really wish I was.
Enterprise plans are almost never subsidized. You are paying a premium to get what others are missing. Guarantee of data residency, SLA agreements, fast support etc.
Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing.
Don't really understand why other agents to give those insights but well found my solution.
what proxy is he using? i support not trying to support a specific product in a post but not even mentioning what you use after writing this seems pretty odd.
welcome to the new way of software distribution, blogs and tweets are the prompts you use
Seeing some good stuff in the comments, but how are people doing this?
I am not, but I can tell you how:
homelab running liteLLM or gomodel
It's LLM, of course they don't know what's going on.
Some don't need to. Not yet anyway. There's config, there's control, and then there's setup-theatre as a hobby.
Agents can log traces, which can be super useful for RL, prompt optimization, tool improvements, etc.
Check out Calibra for example: https://calibra.swival.dev which turns traces from Swival into nice web pages, allows diff'ing traces, etc.
Umm, observability ? I think the article means to say how do you know the agents arent talking nonsense to each other if you use a multi-agent system. Here again a thin layer of o11y would do the trick and also hooking into the wider ecosystem. Thats without having to bother with a proxy.
By running my own agent harness that provides detailed API metrics in the GUI and logs very verbosely.
What is the use case for these sorts of things?
Anyone know an easy, out of the box method of doing this inspection on DGXOS (the OS for DGX Spark devices)?
I'm a little miffed that this isn't just an immediate OOBE, but I'm prepared to do some nc/ssh tunnel magic if that'll work properly .. just wondering what other DGX users have vis a vis best practices ..
Why not just grab it from the kernel?
Because the details we care about right now already travelling over the network and the industry is adopting OTEL, where you can export and collate kernel things if you need to
> how do you know what's going on
there's the kind of people who care and the kind who doesn't ...
your local politicians and landlords do all kinds of normal, non-funky stuff you don't know about ... as do your local ISP-techies and hobbyists with access to supply chains and sub-distributing units ...
or your local school teachers and some parents and their widdle vintage schemes and circumambient tales they tell some kids but not others ... Lord, some guy even caught teachers and parents running schemes on teens to prep them for gay homo fag stuff, despair, frustration ... running for years ...
and you wonder how peeps don't what is going on xD?<
even those hackies and hackers and programmers and brogrammers and AI superusers who think they know and care and wield some power really really don't
but there you go, AI augmented diluted delusions
Does anyone here use LiteLLM for this? How well does it do? Can I use it with my Claude subscription?
I use GoModel and its amazing. My favorite features are virtual model, the support for arbitrary request labels, and the nice dashboads / ui. Creator has more sophisticated report generation on the near term roadmap. He's super responsive to feedback and ships fast. (except for his recent trip to the US west coast to meet founders and investors, he's EU based)
You can use and control all.your subscriptions or aoikeys through a gateway like this, so you don't get pwned like what just happened with OpenCode (or an OpenRouter like vendor). GoModel has some ambitions like this, but is open source so you can run it on you own infra, with plans for self hosted enterprise as well (there are a few features already like SSO). We are rolling out the free version at work right now.
I run GoModel at home and have hyper detailed history of every agent interaction, across models running on my sparks and OpenCode sub. You can call me a committed superfan of this project.
https://github.com/ENTERPILOT/GOModel
https://docs.litellm.ai/docs/tutorials/claude_code_max_subsc...
Hi i'm a LiteLLM maintainer here, yes you can use it with a Claude Sub: https://docs.litellm.ai/docs/tutorials/claude_code_max_subsc...
Let me know if any issues, happy to help
vibes
All that posturing for the purpose to be manually monitoring cache hits and misses. This can't be serious. Buy a larger plan or reduce some instructions and you don't have to deal with this waste of time.
> Own your infrastructure
...proceeds to use codex.
With just a 12gb VRAM GPU + 64gb of CPU RAM, and you can actually own your own infrastructure. I can't wait for hardware prices to fall so that becomes even more accessible.
I am boggled how the first step of an llm after you make an orchestrator isn't some kind of proxy that records your personal internet surfing to be used by your personal AI later. Every college should be running in house AI with their own in house search engines and in house any service. Don't go to any college that wants you to use chatgpt or other corporate nonsense. If they can't make local LLMs work, they are not a real college.
Easy. I don’t use any of the AI shit.