Prometheus

Prev Next

Prometheus setup guide

Overview

This document provides instructions for setting up Prometheus monitoring for Lobster Data Platform application.

Requirements

  • Prometheus server installed

  • Access to target applications for metrics scraping

  • Valid authentication tokens for secured endpoints

Configuration file structure

The Prometheus configuration is defined in prometheus.yml (https://prometheus.io/docs/prometheus/latest/configuration/configuration/) with the following key components:

Global settings

global:
  scrape_interval: 15s      # Default interval for scraping metrics
  evaluation_interval: 15s  # Default interval for evaluating rules

Scrape configurations

External system monitoring

Monitors an external system with the following configuration:

  • Target: <server name> like: localhost, 10.xx.yy.zz

  • Metrics path: /system/metrics

  • Scrape interval: 25 seconds

  • Timeout: 10 seconds

  • Authentication: Bearer token authentication

Security configuration

Authentication methods

The configuration supports multiple authentication methods but LDP only uses one:

  • Bearer token: Currently active authentication method

Token management

  • Store tokens securely and avoid committing them to version control

  • Each environment should have its own set of authentication credentials

Sample configuration file

Below is a complete sample prometheus.yml configuration file that you can use as a starting point:


# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'lobster-data-app'
    static_configs:
      - targets: [ 'localhost:8080' ]
    scrape_interval: 25s
    scrape_timeout: 10s
    metrics_path: /system/metrics
    bearer_token: 'your-application-token-here'

Configuration customization

To customize this configuration for your environment:

  • Replace target addresses:

    • Change localhost:8080 to your application's host and port

  • Update authentication tokens:

    • Replace your-application-token-here with your application's token

  • Adjust timing settings:

    • Modify scrape_interval values based on your monitoring frequency needs

    • Adjust scrape_timeout if your endpoints need more time to respond

Deployment steps

Prepare configuration

  • Update target addresses for your environment

  • Replace authentication tokens with valid credentials

  • Adjust scrape intervals based on your monitoring requirements

Validate configuration

promtool check config prometheus.yml

Start Prometheus

prometheus --config.file=prometheus.yml

Verify setup

  • Access Prometheus web interface (default: http://localhost:9090)

  • Check targets status under Status → Targets

  • Verify metrics are being collected  

Troubleshooting

Common issues

  1. Authentication failures: Verify bearer tokens are valid and not expired

  2. Connection timeouts: Check network connectivity and firewall rules

  3. Missing metrics: Ensure target applications expose metrics at the configured path

Monitoring health

  • Check Prometheus logs for scraping errors

  • Monitor target status in Prometheus UI

  • Verify metrics are being collected with expected labels

Best practices

  • Security

    • Use environment-specific tokens

    • Implement proper access controls

  • Performance

    • Adjust scrape intervals based on requirements

    • Monitor Prometheus resource usage

    • Consider metric retention policies

  • Maintenance

    • Regular configuration reviews

    • Monitor target health

    • Update authentication credentials as needed

Related articles