Custom Metrics

Beyond system metrics, Probestyx can collect custom metrics from your applications, APIs, and external sources. This guide shows you how to create custom metric collectors.

Overview

Custom metrics are defined in the scrapers section of your configuration:

yaml

scrapers:
  - name: my_custom_metrics
    source:
      type: url|file|command
      url: "..."
      format: json|prometheus|raw|log
    metrics:
      - path: "..."
        name: "output_name"

Scraping from Applications

JSON API Metrics

Most modern applications expose metrics as JSON. Here's how to collect them:

Your application endpoint (http://localhost:8080/metrics):

json

{
  "http": {
    "requests": {
      "total": 150000,
      "errors": 42
    },
    "response_time_ms": 125.5
  },
  "database": {
    "queries": {
      "total": 200000,
      "slow": 15
    },
    "pool": {
      "active": 8,
      "idle": 2,
      "max": 10
    }
  },
  "cache": {
    "hits": 45000,
    "misses": 5000
  }
}

Probestyx configuration:

yaml

scrapers:
  - name: app_metrics
    source:
      type: url
      url: "http://localhost:8080/metrics"
      format: json
    metrics:
      - path: "http.requests.total"
        name: "http_requests_total"
      
      - path: "http.requests.errors"
        name: "http_errors_total"
      
      - path: "http.response_time_ms"
        name: "http_response_time_ms"
      
      - path: "database.pool.active"
        name: "db_connections_active"
      
      - path: "database.pool.idle"
        name: "db_connections_idle"
      
      - path: "cache.hits"
        name: "cache_hits"
      
      - path: "cache.misses"
        name: "cache_misses"

Calculate cache hit rate:

yaml

metrics:
  - path: "cache.hits"
    name: "cache_hits"
  
  - path: "cache.misses"
    name: "cache_misses"
  
  # Note: Complex calculations must be done in post-processing
  # Probestyx only supports simple value transformations

Nested JSON Objects

For deeply nested JSON, use dot notation:

Source:

json

{
  "services": {
    "api": {
      "v1": {
        "health": "healthy",
        "requests": 1000
      },
      "v2": {
        "health": "healthy",
        "requests": 5000
      }
    }
  }
}

Configuration:

yaml

metrics:
  - path: "services.api.v1.requests"
    name: "api_v1_requests"
  
  - path: "services.api.v2.requests"
    name: "api_v2_requests"

JSON Arrays

Extract values from arrays using index notation:

Source:

json

{
  "servers": [
    {"name": "web-01", "load": 0.5},
    {"name": "web-02", "load": 0.7}
  ]
}

Configuration:

yaml

metrics:
  - path: "servers.0.load"
    name: "web01_load"
  
  - path: "servers.1.load"
    name: "web02_load"

Scraping from Files

Local JSON Files

Read metrics from files written by your application:

File: /var/app/stats.json

json

{
  "timestamp": 1704067200,
  "metrics": {
    "queue_size": 42,
    "processed_items": 1500,
    "error_count": 3
  }
}

Configuration:

yaml

scrapers:
  - name: app_stats
    source:
      type: file
      path: "/var/app/stats.json"
      format: json
    metrics:
      - path: "metrics.queue_size"
        name: "queue_size"
      
      - path: "metrics.processed_items"
        name: "processed_items"
      
      - path: "metrics.error_count"
        name: "error_count"

Prometheus Format Files

Many exporters write metrics to files:

File: /var/node_exporter/metrics.prom

# TYPE node_memory_MemTotal_bytes gauge
node_memory_MemTotal_bytes 16777216000

# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 8388608000

# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67

Configuration:

yaml

scrapers:
  - name: node_metrics
    source:
      type: file
      path: "/var/node_exporter/metrics.prom"
      format: prometheus
    metrics:
      - match: "node_memory_MemTotal_bytes"
        name: "memory_total_mb"
        calculate: "value / 1024 / 1024"
      
      - match: "node_memory_MemAvailable_bytes"
        name: "memory_available_mb"
        calculate: "value / 1024 / 1024"

Plain Text Files

Parse custom text formats using regex:

File: /var/sensors/readings.txt

temperature=23.5
humidity=65.2
pressure=1013.25
battery_voltage=3.7
signal_strength=-45

Configuration:

yaml

scrapers:
  - name: sensor_data
    source:
      type: file
      path: "/var/sensors/readings.txt"
      format: raw
      pattern: '(\w+)=([0-9.-]+)'  # key=value pattern
    metrics:
      - match: "temperature"
        name: "temperature_celsius"
      
      - match: "humidity"
        name: "humidity_percent"
      
      - match: "pressure"
        name: "pressure_hpa"
      
      - match: "battery_voltage"
        name: "battery_volts"

Custom Patterns

Define custom regex patterns for specific formats:

File: /var/log/app.log (log-based metrics)

[2024-01-01 12:00:00] Requests processed: 1500
[2024-01-01 12:00:00] Average latency: 125ms
[2024-01-01 12:00:00] Active users: 42

Configuration:

yaml

scrapers:
  - name: log_metrics
    source:
      type: file
      path: "/var/log/app.log"
      format: raw
      pattern: '(\w+\s\w+):\s+([0-9]+)'  # "Label: 123" pattern
    metrics:
      - match: "Requests processed"
        name: "requests_processed"
      
      - match: "Active users"
        name: "active_users"

Scraping from External Services

Prometheus Exporters

Collect from standard Prometheus exporters:

yaml

scrapers:
  # Node Exporter
  - name: node_exporter
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    metrics:
      - match: "node_memory_MemAvailable_bytes"
        name: "memory_available_mb"
        calculate: "value / 1024 / 1024"
    filter:
      include:
        - "^node_memory_.*"
        - "^node_cpu_.*"
  
  # Redis Exporter
  - name: redis_exporter
    source:
      type: url
      url: "http://localhost:9121/metrics"
      format: prometheus
    metrics:
      - match: "redis_connected_clients"
        name: "redis_clients"
      - match: "redis_memory_used_bytes"
        name: "redis_memory_mb"
        calculate: "value / 1024 / 1024"
    filter:
      include:
        - "^redis_.*"

Database Metrics

PostgreSQL stats endpoint:

yaml

scrapers:
  - name: postgres_stats
    source:
      type: url
      url: "http://localhost:9187/metrics"
      format: prometheus
    metrics:
      - match: "pg_stat_database_numbackends"
        name: "pg_active_connections"
      
      - match: "pg_stat_database_tup_fetched"
        name: "pg_rows_fetched"

Message Queue Metrics

RabbitMQ management API:

yaml

scrapers:
  - name: rabbitmq
    source:
      type: url
      url: "http://localhost:15672/api/overview"
      format: json
    metrics:
      - path: "queue_totals.messages"
        name: "rabbitmq_total_messages"
      
      - path: "queue_totals.messages_ready"
        name: "rabbitmq_messages_ready"
      
      - path: "object_totals.connections"
        name: "rabbitmq_connections"

Container Metrics

Docker stats (requires wrapper script):

Create a script that outputs JSON:

bash

#!/bin/bash
# /usr/local/bin/docker-stats.sh

docker stats --no-stream --format '{"container":"{{.Name}}","cpu":"{{.CPUPerc}}","mem":"{{.MemPerc}}"}' | jq -s '.'

Configuration:

yaml

scrapers:
  - name: container_stats
    source:
      type: file
      path: "/tmp/docker-stats.json"  # Output of script
      format: json
    metrics:
      - path: "0.cpu"
        name: "container_1_cpu"
      
      - path: "0.mem"
        name: "container_1_mem"

TIP

The wrapper-script-plus-cron-plus-file dance above exists because there was no way to feed a command's output straight into probestyx. type: command (see Scrapers → Command Sources) removes that need for sources like this — probestyx runs the command itself on every scrape.

Container Logs (Docker error lines)

To list error lines out of a container's logs instead of resource stats, skip the wrapper script and point a command source directly at docker logs, using format: log:

yaml

scrapers:
  - name: docker_errors
    source:
      type: command
      command: "docker logs --since {{since}} --tail 1000 my-container 2>&1"
      format: log
      max_lines: 50
    metrics:
      - match: "count"
        name: "docker_error_count"
      - match: "lines"
        name: "docker_error_lines"

is substituted with the timestamp of this scraper's last successful run, so each scrape only asks Docker for what's new — see Scrapers → Log Parser for the full tailing/ behavior, and how to give multiple independent monitoring backends their own view via server.consumers + ?callerId=.

Metric Transformations

Unit Conversions

Convert between units using calculations:

yaml

metrics:
  # Bytes to Megabytes
  - path: "memory.bytes"
    name: "memory_mb"
    calculate: "value / 1024 / 1024"
  
  # Bytes to Gigabytes
  - path: "disk.bytes"
    name: "disk_gb"
    calculate: "value / 1024 / 1024 / 1024"
  
  # Seconds to Milliseconds
  - path: "response.time"
    name: "response_time_ms"
    calculate: "value * 1000"
  
  # Milliseconds to Seconds
  - path: "duration.ms"
    name: "duration_seconds"
    calculate: "value / 1000"
  
  # Celsius to Fahrenheit
  - path: "temperature.celsius"
    name: "temperature_fahrenheit"
    calculate: "value * 1.8 + 32"
  
  # Ratio to Percentage
  - path: "cache.hit_rate"
    name: "cache_hit_percent"
    calculate: "value * 100"

Negative Values

Handle metrics that can be negative:

yaml

metrics:
  # Temperature (can be negative)
  - path: "outdoor.temperature"
    name: "outdoor_temp_celsius"
  
  # Profit/Loss
  - path: "financials.profit"
    name: "profit_usd"

Scaling

Scale values up or down:

yaml

metrics:
  # Scale down large numbers
  - path: "requests.total"
    name: "requests_thousands"
    calculate: "value / 1000"
  
  # Normalize percentage already in 0-100 range
  - path: "cpu.usage"  # Already 0-100
    name: "cpu_percent"

Filtering Metrics

Include Patterns

Only collect specific metrics:

yaml

scrapers:
  - name: selective_metrics
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      include:
        - "^node_memory_.*"     # Only memory metrics
        - "^node_cpu_.*"        # Only CPU metrics
        - "^node_network_.*"    # Only network metrics

Exclude Patterns

Remove unwanted metrics:

yaml

scrapers:
  - name: filtered_metrics
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      exclude:
        - ".*_bucket$"          # Remove histogram buckets
        - ".*_sum$"             # Remove histogram sums
        - ".*_count$"           # Remove histogram counts
        - ".*internal.*"        # Remove internal metrics
        - ".*debug.*"           # Remove debug metrics

Combined Filtering

Use both include and exclude:

yaml

scrapers:
  - name: precise_filtering
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      include:
        - "^node_.*"            # Only node_exporter metrics
      exclude:
        - ".*_bucket$"          # But not histograms
        - ".*_info$"            # Or info metrics

Advanced Examples

Multi-Endpoint Application

Collect from multiple endpoints of the same app:

yaml

scrapers:
  # Main API metrics
  - name: api_metrics
    source:
      type: url
      url: "http://localhost:8080/metrics"
      format: json
    metrics:
      - path: "requests.total"
        name: "api_requests_total"
  
  # Health check endpoint
  - name: api_health
    source:
      type: url
      url: "http://localhost:8080/health"
      format: json
    metrics:
      - path: "database.connected"
        name: "db_connected"
      - path: "cache.connected"
        name: "cache_connected"
  
  # Admin endpoint
  - name: api_admin
    source:
      type: url
      url: "http://localhost:8080/admin/stats"
      format: json
    metrics:
      - path: "worker_pool.active"
        name: "workers_active"

Microservices

Monitor multiple services:

yaml

scrapers:
  # Auth Service
  - name: auth_service
    source:
      type: url
      url: "http://auth:8080/metrics"
      format: json
    metrics:
      - path: "auth.logins.total"
        name: "auth_logins_total"
      - path: "auth.tokens.active"
        name: "auth_tokens_active"
  
  # User Service
  - name: user_service
    source:
      type: url
      url: "http://user:8080/metrics"
      format: json
    metrics:
      - path: "users.active"
        name: "users_active"
      - path: "users.sessions"
        name: "user_sessions"
  
  # Order Service
  - name: order_service
    source:
      type: url
      url: "http://order:8080/metrics"
      format: json
    metrics:
      - path: "orders.pending"
        name: "orders_pending"
      - path: "orders.processed"
        name: "orders_processed"

Hybrid Sources

Mix URLs and files:

yaml

scrapers:
  # Remote API
  - name: remote_api
    source:
      type: url
      url: "https://api.example.com/metrics"
      format: json
    metrics:
      - path: "status.active_users"
        name: "remote_active_users"
  
  # Local file from cron job
  - name: batch_job_stats
    source:
      type: file
      path: "/var/batch/last_run.json"
      format: json
    metrics:
      - path: "processed_records"
        name: "batch_records_processed"
      - path: "duration_seconds"
        name: "batch_duration"
  
  # Local Prometheus exporter
  - name: local_exporter
    source:
      type: url
      url: "http://localhost:9200/metrics"
      format: prometheus
    metrics:
      - match: "custom_metric.*"

Best Practices

Naming Conventions

Use clear, consistent metric names:

yaml

# Good
metrics:
  - path: "requests.total"
    name: "http_requests_total"           # Clear and specific
  
  - path: "memory.used"
    name: "app_memory_used_bytes"         # Includes unit
  
  - path: "queue.size"
    name: "rabbitmq_queue_messages"       # Includes source

# Avoid
metrics:
  - path: "requests.total"
    name: "total"                         # Too vague
  
  - path: "memory.used"
    name: "mem"                           # Ambiguous

Keep It Simple

Don't over-collect:

yaml

# Good - focused collection
metrics:
  - path: "cache.hit_rate"
    name: "cache_hit_rate"
  - path: "cache.total_keys"
    name: "cache_total_keys"

# Avoid - collecting everything
metrics:
  - path: "cache.hit_rate"
    name: "cache_hit_rate"
  - path: "cache.miss_rate"
    name: "cache_miss_rate"
  - path: "cache.total_hits"
    name: "cache_total_hits"
  - path: "cache.total_misses"
    name: "cache_total_misses"
  # ... and 20 more metrics you don't need

Use Filtering

Reduce payload size:

yaml

scrapers:
  - name: focused_metrics
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      include:
        - "^node_cpu_.*"        # Only what you need
        - "^node_memory_.*"
      exclude:
        - ".*_bucket$"          # Remove histogram overhead

Document Your Metrics

Add comments to your config:

yaml

scrapers:
  # Application performance metrics
  # Updated every 60 seconds by the app
  - name: app_performance
    source:
      type: file
      path: "/var/app/perf.json"
      format: json
    metrics:
      # Request latency in milliseconds
      - path: "latency.p95"
        name: "app_latency_p95_ms"
      
      # Active database connections
      - path: "database.connections.active"
        name: "app_db_connections_active"

Troubleshooting

Metric Not Appearing

Check the source is accessible
Verify the path/match is correct
Test the source manually
Check Probestyx logs

bash

# Test JSON endpoint
curl http://localhost:8080/metrics | jq '.your.metric.path'

# Test file exists and is readable
cat /var/app/stats.json | jq '.your.metric.path'

# Check Probestyx logs
sudo journalctl -u probestyx -f

Wrong Values

Verify calculation is correct
Check source value type (string vs number)
Ensure units are what you expect

Performance Issues

Reduce number of scrapers
Use filtering to limit metrics
Scrapers aren't cached (cache_ttl only applies to system metrics) — poll /metrics less often if a scraper's source is slow
Check source response times

Next Steps

Scrapers - Advanced scraper configuration
Integration - Send metrics to Logstyx

Custom Metrics ​

Overview ​

Scraping from Applications ​

JSON API Metrics ​

Nested JSON Objects ​

JSON Arrays ​

Scraping from Files ​

Local JSON Files ​

Prometheus Format Files ​

Plain Text Files ​

Custom Patterns ​

Scraping from External Services ​

Prometheus Exporters ​

Database Metrics ​

Message Queue Metrics ​

Container Metrics ​

Container Logs (Docker error lines) ​

Metric Transformations ​

Unit Conversions ​

Negative Values ​

Scaling ​

Filtering Metrics ​

Include Patterns ​

Exclude Patterns ​

Combined Filtering ​

Advanced Examples ​

Multi-Endpoint Application ​

Microservices ​

Hybrid Sources ​

Best Practices ​

Naming Conventions ​

Keep It Simple ​

Use Filtering ​

Document Your Metrics ​

Troubleshooting ​

Metric Not Appearing ​

Wrong Values ​

Performance Issues ​

Next Steps ​

Custom Metrics

Overview

Scraping from Applications

JSON API Metrics

Nested JSON Objects

JSON Arrays

Scraping from Files

Local JSON Files

Prometheus Format Files

Plain Text Files

Custom Patterns

Scraping from External Services

Prometheus Exporters

Database Metrics

Message Queue Metrics

Container Metrics

Container Logs (Docker error lines)

Metric Transformations

Unit Conversions

Negative Values

Scaling

Filtering Metrics

Include Patterns

Exclude Patterns

Combined Filtering

Advanced Examples

Multi-Endpoint Application

Microservices

Hybrid Sources

Best Practices

Naming Conventions

Keep It Simple

Use Filtering

Document Your Metrics

Troubleshooting

Metric Not Appearing

Wrong Values

Performance Issues

Next Steps