Prometheus¶

Connecting to Prometheus¶

We specify a Prometheus URL through the PROMETHEUS_QUERY_URL environment variable.

Header provider: Customize authentication to access prometheus¶

Tamland supports a header provider to customize the requests sent to prometheus, e.g. to add authentication headers. This is useful in situations where prometheus sits behind an authentication proxy (e.g. Google IAP).

A header provider is an external program that generates a JSON with header information:

{
  "headers": {
    "Authentication": "Bearer my-token"
  }
}

The external program needs to be executable.

In order to enable using the header provider, specify the path to the executable in the PROMETHEUS_HEADER_PROVIDER environment variable.

Tamland will execute the header provider regularly (every 30 minutes per default) to update the headers used for sending requests to prometheus. This is useful e.g. when tokens expire over time.

The interval between header freshes can be configured by setting the PROMETHEUS_HEADER_PROVIDER_TTL environment variable to the desired interval in seconds.

Caching Prometheus data¶

S3/GCS support and multi-level caching¶

We’ve introduced support for PyFilesystem based caches, which allows us to store cache data locally but also in S3 or GCS, for example. The default cache can be changed by specifying a PyFilesystem compatible URL through the PROM_QUERY_CACHE_FS environment variable.

Optionally, a second-level cache can be used to store the cache longer-term, e.g. in S3 or GCS. This can be configured through the PROM_QUERY_SECOND_CACHE_FS, which also expects a PyFilesystem compatible URL.

Cache misses will be backfilled either from a higher-level cache or from querying prometheus directly.

For example, an advanced use case is having a local on-disk cache and using GCS as a second-level longer-term cache:

PROM_QUERY_CACHE_FS=./path/to/local/cache/directory
PROM_QUERY_SECOND_CACHE_FS=gs://your-tamland-bucket-here/cache