This performance-metric glossary for the token platform explains each metric, common measurement conventions, and appropriate use cases so you can compare models and providers and evaluate real application performance.
Ongoing collection and evaluation supply the provider data in the Model Market and model details. The table below describes the methodology for common metrics, test conditions, and result locations. Use this data to inform model selection, not as a permanent guarantee of future availability or performance.
| Metric | Measurement | Presentation |
|---|---|---|
Latency | Text generation uses time to first token; Embedding and Reranker use full-response latency. | Provider data uses P90 over the last 6 hours; trend charts use P90 over the last 7 days. |
Throughput | For streaming responses, the number of output tokens received per unit of time after the first token. | Provider data uses the average over the last 6 hours; trend charts use the average over the last 7 days. |
Reliability | Sampled real-request success rate × 0.8 + Benchmark test success rate × 0.2; use the Benchmark result when there are too few real samples. | Provider data in model details uses the reliability result from the last 6 hours. |
Price and Capabilities | Providers supply prices, context length, and similar values whenever possible. Boundary requests can evaluate missing maximum input or output lengths. | Shown in model details and provider data cards. Use the current model data as the source of truth. |
Test Conditions | Tests do not use caching. They change the Prompt prefix between rounds, send requests from nodes in multiple regions, and run providers in the same batch concurrently. | Sample results vary by model, provider, measurement window, network, and request conditions. |
Each metric answers a different question: request rate and TPM measure capacity, TTFT and end-to-end latency measure responsiveness, throughput measures sustained output speed, and reliability and error rate measure stability. Keep the model, provider, request content, concurrency, and measurement window consistent when comparing results.
| Metric | Name | Definition and Unit | How to Interpret It |
|---|---|---|---|
QPS / RPM | Request Rate | Requests completed per second or per minute | Measures the request frequency an endpoint can handle. RPM is the per-minute request limit. |
TPM | token Rate | Total input tokens and output tokens processed per minute | Reflects actual LLM gateway processing volume more accurately than request count alone. Long-context requests consume more TPM. |
Concurrency | Concurrent Requests | Number of requests the system processes at the same time | Shows pressure from connections, queues, and upstream processing. High concurrency does not mean every request receives the same throughput. |
TTFT | Time to First token | Time from sending a request to receiving the first token | Primarily affects how quickly a streaming conversation appears to begin responding. Usually measured in seconds; lower is better. |
Latency / E2E | End-to-End Latency | Time from sending a request to receiving the complete response | Useful for measuring non-streaming calls or total task duration. Compare it together with output length. |
Throughput | Output Throughput | Output tokens received per unit of time in a streaming response | Usually measured in tokens/s. Useful for comparing sustained output speed during long-form generation. |
Success rate | Success Rate | Successful requests ÷ total requests | Reflects request-path stability. Define the measurement window, status-code range, and whether the metric counts upstream failures. |
Availability | Availability | Percentage of requests the service can accept and complete during the measurement window | Measures service continuity. Evaluate it alongside outage duration, error rate, and recovery time. |
P50 / P90 / P95 / P99 | Latency Percentiles | Value at the corresponding position after sorting latency from lowest to highest | P50 represents the typical experience; P90/P95/P99 expose the slow tail. Lower percentile values are better. |
Context window | Context Window | Maximum combined input and output tokens allowed in one request | Determines how much context a single request can carry. It is not the platform-wide TPM. |
Max output tokens | Maximum Output Length | Maximum output tokens a single request can generate | Affects whether long-form tasks can finish and how long they take. Actual limits depend on the model and provider. |
Error rate | Error Rate | Failed requests ÷ total requests | Classify errors by authentication, rate limits, balance or quota, parameters, and upstream failures. A single aggregate value is not enough for diagnosis. |
A unified API, connection management, and routing layer consolidate calls to different models and providers behind one integration. This supports a gradual path from low-traffic validation to high-concurrency production workloads.
The platform can process more than one hundred million TPM overall. This describes aggregate gateway capacity; it does not mean that every model or provider channel has the same limit.
Filter, rank, or automatically fall back across providers using latency, throughput, price, input length, and availability so each workload can strike the right balance among cost, speed, and reliability.
Model details, provider data, and request logs expose latency, throughput, reliability, token usage, and error information, making it easier to compare channels, diagnose slow requests, and tune routing policies.
