Skip to content

Metrics

Prometheus-compatible metrics are made available on the default port, on the /metrics endpoint.

Below is a list of the metrics that are exposed: | Metric Name | Type | | -------------------------------------------- | --------- | | lorax_request_count | Counter | | lorax_request_success | Counter | | lorax_request_failure | Counter | | lorax_request_duration | Histogram | | lorax_request_queue_duration | Histogram | | lorax_request_validation_duration | Histogram | | lorax_request_inference_duration | Histogram | | lorax_request_mean_time_per_token_duration | Histogram | | lorax_request_generated_tokens | Histogram | | lorax_request_input_length | Histogram |

For all histograms, there are metrics that are autogenerated which are the metric name + _sum and _count, which are the sum of all values for that histogram, and the count of all instances of that histogram respectively.