Metrics
Prometheus-compatible metrics are made available on the default port, on the /metrics
endpoint.
Below is a list of the metrics that are exposed:
| Metric Name | Type |
| -------------------------------------------- | --------- |
| lorax_request_count
| Counter |
| lorax_request_success
| Counter |
| lorax_request_failure
| Counter |
| lorax_request_duration
| Histogram |
| lorax_request_queue_duration
| Histogram |
| lorax_request_validation_duration
| Histogram |
| lorax_request_inference_duration
| Histogram |
| lorax_request_mean_time_per_token_duration
| Histogram |
| lorax_request_generated_tokens
| Histogram |
| lorax_request_input_length
| Histogram |
For all histograms, there are metrics that are autogenerated which are the metric name + _sum
and _count
, which are the sum of all values for that histogram, and the count of all instances of that histogram respectively.