Llmledger

List the highest rated models for a kind of work within price, prompt size, ability and maker limits, each at its cheapest listing with its rating; or, with `name`, look one model up and list every host that offers it. Use when browsing or filtering the catalogue, or to find a model's exact name. Not for choosing between named models (compare_models), for a prompt you have in hand (suggest_for_project), or for a budget (estimate_cost).

Compare 2 to 6 named models for a kind of work: rating, prices, cost per task (or how fast each uses a Claude or ChatGPT plan), the pick, and for each model the share of answer tokens at which it would beat the pick. Use when the user already has candidate models in mind. Not for finding candidates (find_models), for reading a prompt to guess the work (suggest_for_project), or for a monthly budget (estimate_cost).

Price a workload on 1 to 10 models at each one's cheapest listing: dollars per call, per day and per 30 days, and with cache tokens the saving the cache makes. Use when the question is a budget or a bill: token counts and calls a day are known. Not for deciding which model is better (compare_models) or for pricing past usage from an export (price_my_usage).

Show one model's price history: every price and retirement date change since 2023 on every host, each with the LiteLLM change behind it, the chance of a price change, retirement or removal in the next 90 days and year, and with `date`, the prices on that day. Use when the question is about one model over time, a past price, or how likely a price is to move. Not for today's cheapest price of several models (find_models or estimate_cost).

Suggest a model and effort level for a prompt or a described project: it guesses the kind of work and names the best value (the cheapest model the votes cannot tell apart from the top), the top rated and the cheapest rated, each with its cost per task. Use when the user asks "which model should I use for this" and the work itself is in hand. Not for named candidates (compare_models) or for browsing by limits (find_models).

List the models Arena has rated that LiteLLM's catalogue does not price yet, with how soon the catalogue usually prices a maker's models after Arena first shows them. Use when a brand-new model is missing from the other tools' answers, or to ask what is coming. Not for models that already have a price (find_models).

Estimate how long one call and a whole job take, as a usual time and a slow day, from the user's own timed runs of that model called that way (kept by log_run). Use when the question is time, not money, and some runs have been logged; it says so when none have. Not for cost (estimate_cost) or for a job you have a promptfoo sample of (forecast_job gives time and cost together).

Record a timed run (model, tokens, seconds, how it was called), or every result of a promptfoo results file, in a file on this computer so estimate_time can learn from it. Use right after a call or a promptfoo run whose time is known. Not a query: it stores and returns what it stored. Nothing is sent anywhere.

Estimate a whole job's time, cost and accuracy from a promptfoo run of a sample of it: for each model in the sample, the time as a usual time and a slow day, the cost at its cheapest listing next to the listing the sample used, the 90-day chance of a price change, how often it was right (95% range) and the break-even against the pick. Use when a promptfoo results file of a sample exists and the whole job's size is known. Not for a job with no sample (estimate_cost for money, estimate_time for time).

Compare running an open model on a computer you buy with paying a cloud model per call: the cost a month each way, the month the computer pays for itself, and whether it keeps up with the load. Use when the question is whether to buy hardware for a local model. Needs timed runs of the local model (log_run) and the computer's price; electricity comes from EIA or the user's own rate. Not for cloud-only comparisons (compare_models or estimate_cost).

Price the user's own past coding agent usage, from a ccusage or codeburn export, on every rated model: each session at the price of its day, the models that keep the rating and cost less, a 95% range for the cost per session, and whether two models differ for real or the data cannot tell. Use when a usage export (or a list of token mixes) exists and the question is what the same work would have cost elsewhere. Not for a future workload (estimate_cost) or for a prompt (suggest_for_project). Reads token counts, names and dates only; nothing leaves the computer.