Scrapers

Scrapers are Probestyx's mechanism for collecting metrics from external sources. This guide covers advanced scraper configuration, patterns, and optimization techniques.

Scraper Architecture

Each scraper runs independently and in parallel:

┌─────────────────────────────────────────────┐
│          Probestyx Scraper Engine           │
│                                             │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐    │
│  │Scraper 1│  │Scraper 2│  │Scraper 3│    │
│  │ (URL)   │  │ (File)  │  │ (URL)   │    │
│  └────┬────┘  └────┬────┘  └────┬────┘    │
│       │            │            │          │
│       ▼            ▼            ▼          │
│  ┌────────────────────────────────┐       │
│  │      Metric Aggregator         │       │
│  │  • Merge results               │       │
│  │  • Apply cache                 │       │
│  └───────────┬────────────────────┘       │
│              ▼                             │
│       HTTP Server :9100                    │
│         GET /metrics                       │
└─────────────────────────────────────────────┘

Source Types

URL Sources

Fetch metrics from HTTP endpoints.

Basic URL scraper:

scrapers:
  - name: api_metrics
    source:
      type: url
      url: "http://localhost:8080/metrics"
      format: json
    metrics:
      - path: "requests.total"
        name: "total_requests"

With HTTPS:

scrapers:
  - name: secure_api
    source:
      type: url
      url: "https://api.example.com/metrics"
      format: json
    metrics:
      - path: "data.value"
        name: "api_value"

Multiple endpoints:

scrapers:
  - name: service_a
    source:
      type: url
      url: "http://service-a:8080/metrics"
      format: json
    metrics:
      - path: "requests"
        name: "service_a_requests"
  
  - name: service_b
    source:
      type: url
      url: "http://service-b:8080/metrics"
      format: json
    metrics:
      - path: "requests"
        name: "service_b_requests"

File Sources

Read metrics from local files.

Static file path:

scrapers:
  - name: app_stats
    source:
      type: file
      path: "/var/app/stats.json"
      format: json
    metrics:
      - path: "processed"
        name: "items_processed"

Log file (latest metrics):

scrapers:
  - name: log_metrics
    source:
      type: file
      path: "/var/log/app/metrics.log"
      format: raw
      pattern: '(\w+)=(\d+)'
    metrics:
      - match: "requests"
        name: "log_requests"

Cron-generated file:

scrapers:
  - name: daily_stats
    source:
      type: file
      path: "/var/batch/daily-stats.json"
      format: json
    metrics:
      - path: "total_processed"
        name: "daily_processed"

Command Sources

Run a shell command and capture its combined stdout+stderr as the source text. This is the way to reach sources that aren't a file or an HTTP endpoint — most commonly, docker logs.

scrapers:
  - name: docker_errors
    source:
      type: command
      command: "docker logs --since {{since}} --tail 1000 my-container 2>&1"
      format: log
    metrics:
      - match: "count"
        name: "docker_error_count"
      - match: "lines"
        name: "docker_error_lines"

• Runs via sh -c on Linux/macOS, cmd /C on Windows.
• The command string comes from config.yaml, an operator-controlled file — not from request input — so shell interpretation here isn't an injection surface exposed to callers of /metrics. Still, only put trusted commands in it.
• is a special placeholder recognized only for format: log — see Log Parser below for what it does and why you'd want it.

Format Parsers

JSON Parser

Extracts values using dot notation paths.

Simple object:

{
  "requests": 1000,
  "errors": 5
}

metrics:
  - path: "requests"
    name: "total_requests"
  - path: "errors"
    name: "total_errors"

Nested object:

{
  "database": {
    "queries": {
      "total": 5000,
      "slow": 12
    }
  }
}

metrics:
  - path: "database.queries.total"
    name: "db_queries_total"
  - path: "database.queries.slow"
    name: "db_slow_queries"

Array access:

{
  "servers": [
    {"name": "web-01", "requests": 1000},
    {"name": "web-02", "requests": 1500}
  ]
}

metrics:
  - path: "servers.0.requests"
    name: "web01_requests"
  - path: "servers.1.requests"
    name: "web02_requests"

Complex nested structures:

{
  "environments": {
    "production": {
      "regions": {
        "us-east": {
          "instances": 10,
          "requests": 50000
        }
      }
    }
  }
}

metrics:
  - path: "environments.production.regions.us-east.instances"
    name: "prod_us_east_instances"
  - path: "environments.production.regions.us-east.requests"
    name: "prod_us_east_requests"

Prometheus Parser

Parses Prometheus exposition format.

Basic gauge metric:

# TYPE http_requests_total counter
http_requests_total 1500

metrics:
  - match: "http_requests_total"
    name: "http_requests"

Metric with labels:

# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1200
http_requests_total{method="POST",status="200"} 300
http_requests_total{method="GET",status="404"} 50

metrics:
  # Matches first occurrence
  - match: "http_requests_total"
    name: "http_requests_total"

Multiple metrics:

node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
node_memory_MemTotal_bytes 16777216000
node_memory_MemAvailable_bytes 8388608000

metrics:
  - match: "node_cpu_seconds_total"
    name: "cpu_seconds"
  
  - match: "node_memory_MemTotal_bytes"
    name: "memory_total_mb"
    calculate: "value / 1024 / 1024"
  
  - match: "node_memory_MemAvailable_bytes"
    name: "memory_available_mb"
    calculate: "value / 1024 / 1024"

Using wildcards with filters:

scrapers:
  - name: node_exporter
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      include:
        - "^node_memory_.*"  # All memory metrics
    metrics:
      - match: "node_memory_MemTotal_bytes"
        name: "memory_total"
      - match: "node_memory_MemAvailable_bytes"
        name: "memory_available"

Raw Text Parser

Uses regex to extract key-value pairs.

Default pattern ((\w+)=(\S+)):

Source:

cpu_usage=45.2
memory_usage=67.8
disk_usage=52.3

source:
  format: raw
  # pattern is optional, defaults to (\w+)=(\S+)
metrics:
  - match: "cpu_usage"
    name: "cpu_percent"
  - match: "memory_usage"
    name: "memory_percent"

Custom pattern:

Source:

[CPU] 45.2%
[Memory] 67.8%
[Disk] 52.3%

source:
  format: raw
  pattern: '\[(\w+)\]\s+([0-9.]+)%'
metrics:
  - match: "CPU"
    name: "cpu_percent"
  - match: "Memory"
    name: "memory_percent"

Key-value with units:

Source:

temperature: 23.5 C
humidity: 65.2 %
pressure: 1013.25 hPa

source:
  format: raw
  pattern: '(\w+):\s+([0-9.]+)'
metrics:
  - match: "temperature"
    name: "temperature_celsius"
  - match: "humidity"
    name: "humidity_percent"
  - match: "pressure"
    name: "pressure_hpa"

Log-style parsing:

Source:

2024-01-01 12:00:00 INFO requests_processed=1500
2024-01-01 12:00:00 INFO active_users=42
2024-01-01 12:00:00 INFO cache_hit_rate=0.95

source:
  format: raw
  pattern: '(\w+)=([0-9.]+)'
metrics:
  - match: "requests_processed"
    name: "requests_processed"
  - match: "active_users"
    name: "active_users"
  - match: "cache_hit_rate"
    name: "cache_hit_rate"
    calculate: "value * 100"

Log Parser

Scans a text source line-by-line and returns the lines that look like errors, plus a count. Unlike raw, which collapses regex matches into single key/value pairs (a repeated match just overwrites the last one), log is built to return a list — the right tool for "show me the error lines," not "extract one value."

• If pattern is set, it's used as a regex to test each line.
• If pattern is omitted, lines are matched case-insensitively against the word error (covers nginx's [error] and typical Error: ... app/Docker output).
• max_lines caps how many matching lines are returned per scrape (default 100).

Produces two values you can map: count and lines.

nginx error log (file source, tailed):

scrapers:
  - name: nginx_errors
    source:
      type: file
      path: "/var/log/nginx/error.log"
      format: log
      max_lines: 50
    metrics:
      - match: "count"
        name: "nginx_error_count"
      - match: "lines"
        name: "nginx_error_lines"

Docker container logs (command source, -windowed):

scrapers:
  - name: docker_errors
    source:
      type: command
      command: "docker logs --since {{since}} --tail 1000 my-container 2>&1"
      format: log
      max_lines: 50
    metrics:
      - match: "count"
        name: "docker_error_count"
      - match: "lines"
        name: "docker_error_lines"

Only new lines are reported each scrape — this is the important difference from every other format on this page. If it scanned the whole file/output every time, a resolved error sitting in the log would keep showing up in every response forever, since nothing removes old lines from an nginx log or a docker tail window.

• type: file sources are tailed: probestyx remembers how far it has already read. The first scrape after startup seeds that position at the current end of the file (reports nothing, so it doesn't dump history on boot); every scrape after only sees newly appended lines. A shrinking file (log rotation/truncation) is detected and read from the start.
• type: command sources need the placeholder to get this behavior — it's substituted with the timestamp of that scraper's last successful run. Without in the command string, it re-scans the full output every time.
• This state is in-memory only, per scraper: a restart resets tailing to "start from now."

Output (on a scrape where one new error appeared in each):

{
  "nginx_errors": {
    "nginx_error_count": 1,
    "nginx_error_lines": [
      "2026/07/18 10:00:05 [error] 123#0: *2 open() \"/var/www/missing.html\" failed (2: No such file or directory)"
    ]
  },
  "docker_errors": {
    "docker_error_count": 1,
    "docker_error_lines": [
      "something failed with an Error: boom"
    ]
  }
}

Nothing in the matched lines is redacted — if your logs contain IPs, paths, or tokens, they're shipped as-is. Enable server.secret on any deployment using log format.

Per-Consumer Tailing

If more than one independent monitoring backend polls the same probestyx instance for a log scraper, they'd race over that single shared tailing position — whichever polls first after a new error line appears "consumes" it, and the other never sees it.

server:
  consumers:
    - logstyx-prod
    - staging-poller

curl "http://localhost:9100/metrics?callerId=logstyx-prod"
curl "http://localhost:9100/metrics?callerId=staging-poller"

Each declared callerId gets its own independent position per scraper. No callerId (or no server.consumers configured at all) falls back to a shared "default" bucket — unchanged single-consumer behavior. An unlisted callerId gets 400 Bad Request rather than being silently tracked, since /metrics is often unauthenticated and an unbounded ID namespace would let a caller grow the in-memory state indefinitely just by making up new IDs.

Pull mode still means whoever's polling /metrics only sees a new error on their next scrape — up to a full poll interval of latency. See Push Mode below if that's too slow.

Push Mode

Push flips the direction for format: log scrapers specifically: instead of waiting for /metrics to be polled, probestyx detects new lines itself and sends them straight to Logstyx's ingestion API.

Triggering

• type: file scrapers are event-driven. Probestyx watches the log's parent directory rather than the file itself — a directory watch survives log rotation (including the rename-then-recreate scheme, where the original file is renamed away and a new one created at the same path) since it isn't tied to the old file's inode. New writes trigger a push after a ~200ms debounce, so a burst of many lines in one write collapses into one push, not one per line.
• type: command scrapers (e.g. docker logs) have no file to watch, so they're re-run on a ticker instead: push.command_poll_interval_seconds (default 15).
• Each push carries one scraper's new lines since the last trigger — never one push per line, since the ingestion API has no batch endpoint and this keeps well under its rate limit during a burst.
• Push keeps its own tailing cursor, fully independent of /metrics polling. A scraper that's both pulled and pushed reports the same new lines through both channels independently — enabling one doesn't change what the other sees.

Configuration

push:
  enabled: true
  endpoint: "https://api.logstyx.com/api/v1/logs"
  project_id: "your-project-id"
  secret: "your-project-secret"   # distinct from server.secret - see below
  level: "error"
  context:
    environment: "production"

scrapers:
  - name: nginx_errors
    source:
      type: file
      path: "/var/log/nginx/error.log"
      format: log
    push: true

push.secret authenticates probestyx to Logstyx and is a different credential from server.secret, which authenticates callers of probestyx's /metrics. push: true only does anything on a format: log scraper when push.enabled is also true; on any other format it's ignored with a startup warning.

What to know before relying on it

• The retry queue is in-memory only — a restart during a Logstyx outage loses whatever was queued, with no disk persistence.
• A 401 (bad secret/project_id) is treated as fatal: push logs one warning, drops the queue, and stops for that process's lifetime rather than retrying forever into a black hole.
• A 200 response doesn't guarantee the log was stored — a wrong project_id can be silently accepted and discarded server-side with no client-visible error.

See the Configuration guide's Push Mode section for the full config reference.

Metric Mapping

Path-based (JSON)

Use dot notation to navigate JSON structure:

metrics:
  # Top-level
  - path: "total"
    name: "total_count"
  
  # Nested
  - path: "stats.requests.total"
    name: "request_count"
  
  # Array index
  - path: "servers.0.cpu"
    name: "server0_cpu"
  
  # Deep nesting
  - path: "env.prod.region.us.instances"
    name: "prod_us_instances"

Match-based (Prometheus/Raw)

Use metric names or regex patterns:

metrics:
  # Exact match
  - match: "http_requests_total"
    name: "http_requests"
  
  # Matches first occurrence
  - match: "node_cpu_seconds_total"
    name: "cpu_seconds"
  
  # For raw format, matches the captured key
  - match: "temperature"
    name: "temp_celsius"

Output Naming

Choose clear, consistent names:

metrics:
  # Good - descriptive and specific
  - path: "database.connections.active"
    name: "db_connections_active"
  
  # Good - includes unit
  - path: "response.time.milliseconds"
    name: "response_time_ms"
  
  # Good - includes source
  - path: "cache.hit_rate"
    name: "redis_cache_hit_rate"
  
  # Avoid - too generic
  - path: "total"
    name: "total"
  
  # Avoid - ambiguous
  - path: "time"
    name: "time"

Transformations

Simple Calculations

Apply mathematical operations:

metrics:
  # Division
  - path: "bytes_used"
    name: "mb_used"
    calculate: "value / 1024 / 1024"
  
  # Multiplication
  - path: "hit_rate"
    name: "hit_percent"
    calculate: "value * 100"
  
  # Addition
  - path: "temp_celsius"
    name: "temp_kelvin"
    calculate: "value + 273.15"
  
  # Subtraction
  - path: "max_connections"
    name: "available_connections"
    calculate: "value - 10"  # Reserve 10

Common Conversions

Bytes to human-readable:

metrics:
  # Bytes to KB
  - path: "size_bytes"
    name: "size_kb"
    calculate: "value / 1024"
  
  # Bytes to MB
  - path: "size_bytes"
    name: "size_mb"
    calculate: "value / 1024 / 1024"
  
  # Bytes to GB
  - path: "size_bytes"
    name: "size_gb"
    calculate: "value / 1024 / 1024 / 1024"

Time conversions:

metrics:
  # Seconds to milliseconds
  - path: "duration_seconds"
    name: "duration_ms"
    calculate: "value * 1000"
  
  # Milliseconds to seconds
  - path: "latency_ms"
    name: "latency_seconds"
    calculate: "value / 1000"
  
  # Minutes to seconds
  - path: "uptime_minutes"
    name: "uptime_seconds"
    calculate: "value * 60"

Percentage conversions:

metrics:
  # Ratio (0-1) to percentage
  - path: "success_rate"
    name: "success_percent"
    calculate: "value * 100"
  
  # Fraction to percentage
  - path: "cache.hits"  # Assuming this is hits/total
    name: "cache_hit_percent"
    calculate: "value * 100"

Limitations

Calculations support only simple expressions:

Supported:

calculate: "value * 100"
calculate: "value / 1024"
calculate: "value + 273.15"
calculate: "value - 10"

Not supported:

# Multiple operations
calculate: "value * 100 / 1024"

# Functions
calculate: "sqrt(value)"

# Multiple values
calculate: "hits / (hits + misses)"

For complex calculations, perform them in your application or in post-processing.

Filtering

Include Filters

Only collect metrics matching patterns:

filter:
  include:
    - "^node_cpu_.*"        # CPU metrics only
    - "^node_memory_.*"     # Memory metrics only

Multiple patterns:

filter:
  include:
    - "^http_.*"            # All HTTP metrics
    - "^database_.*"        # All database metrics
    - ".*_total$"           # All counters ending in _total

Exclude Filters

Remove unwanted metrics:

filter:
  exclude:
    - ".*_bucket$"          # Remove histograms
    - ".*_debug$"           # Remove debug metrics
    - ".*internal.*"        # Remove internal metrics

Combined Filtering

Use both include and exclude:

filter:
  # First, only keep node metrics
  include:
    - "^node_.*"
  
  # Then, remove histograms and info
  exclude:
    - ".*_bucket$"
    - ".*_sum$"
    - ".*_count$"
    - ".*_info$"

Filter Order

Filters are applied in this order:

1. Include patterns are checked first
   • If any include patterns exist, only matching metrics proceed
   • If no include patterns exist, all metrics proceed
2. Exclude patterns are checked second
   • Metrics matching exclude patterns are removed

Example:

filter:
  include:
    - "^node_.*"      # Only node_ metrics
  exclude:
    - ".*_total$"     # But not counters

Process:

1. node_cpu_usage → Matches include → Passes → Doesn't match exclude → Kept
2. node_requests_total → Matches include → Passes → Matches exclude → Removed
3. app_requests_total → Doesn't match include → Removed (never checked against exclude)

Caching

Scrapers are not cached. Every scraper in the scrapers: section — URL, file, or command — runs fresh on every single /metrics request, regardless of how often it's polled.

cache_ttl exists, but it lives under system: and applies only to the built-in system metrics (CPU/RAM/disk/network/etc.), not to scrapers:

system:
  enabled: true
  cache_ttl: 60  # Caches system metrics only, for 60 seconds
  metrics:
    - cpu_usage_percent

scrapers:
  - name: slow_api
    source:
      type: url
      url: "http://slow-api:8080/metrics"
      format: json
    metrics:
      - path: "data"
        name: "api_data"
    # No cache_ttl here - this scraper hits slow-api on every /metrics request,
    # no matter how frequently probestyx itself is polled.

If a scraper's source is slow or rate-limited, there's no built-in throttle for it today — the mitigations are to reduce how often you poll /metrics itself (fewer, less-frequent Logstyx probes), or to put your own caching layer in front of the scraped source.

The one scraper-level exception is format: log, which has its own separate mechanism for "don't redo work" — it tracks how much of the source it's already read (a file offset, or a timestamp) rather than caching a result. See Log Parser.

Performance Optimization

Parallel Execution

Scrapers run in parallel automatically:

scrapers:
  # These all execute simultaneously
  - name: service_a
    source:
      type: url
      url: "http://service-a:8080/metrics"
      format: json
  
  - name: service_b
    source:
      type: url
      url: "http://service-b:8080/metrics"
      format: json
  
  - name: service_c
    source:
      type: url
      url: "http://service-c:8080/metrics"
      format: json

Total time ≈ slowest scraper, not sum of all scrapers.

Reduce Payload

Use filtering to minimize data transfer:

scrapers:
  - name: node_exporter
    source:
      type: url
      url: "http://localhost:9100/metrics"
      format: prometheus
    filter:
      include:
        - "^node_cpu_.*"
        - "^node_memory_.*"
    # Only collect specific metrics instead of all
    metrics:
      - match: "node_cpu_usage"
        name: "cpu_usage"
      - match: "node_memory_usage"
        name: "memory_usage"

Limit Scrapers

Only collect what you need:

# Good - focused collection
scrapers:
  - name: critical_metrics
    source:
      type: url
      url: "http://app:8080/metrics"
      format: json
    metrics:
      - path: "requests.total"
        name: "requests"
      - path: "errors.total"
        name: "errors"

# Avoid - over-collection
scrapers:
  # 10+ scrapers for different endpoints
  # Hundreds of metrics
  # Slow sources

Optimize Sources

Fast sources:

• Local files
• Localhost URLs
• Cached responses

Slow sources:

• Remote URLs
• Large files
• Complex parsing

Error Handling

Source Unavailable

If a source fails:

• Error is logged
• Other scrapers continue
• Failed scraper returns no metrics
• Service remains healthy

Invalid Format

If parsing fails:

• Error is logged
• Scraper returns no metrics
• Other scrapers unaffected

Partial Failures

If some metrics fail:

• Successfully parsed metrics are returned
• Failed metrics are logged
• Service continues normally

Viewing Errors

Check logs for scraper errors:

# Linux
sudo journalctl -u probestyx -f | grep ERROR

# Look for messages like:
# ERROR scraper "api_metrics" failed: connection refused
# ERROR scraper "file_metrics" failed: file not found
# ERROR scraper "prom_metrics" failed: parse error

Advanced Patterns

Dynamic File Paths

Use consistent file names updated by your app:

scrapers:
  - name: latest_stats
    source:
      type: file
      path: "/var/app/current-stats.json"  # Symlink or overwritten file
      format: json
    metrics:
      - path: "data"
        name: "current_data"

Conditional Collection

Collect different metrics based on environment:

Production config:

scrapers:
  - name: prod_metrics
    source:
      type: url
      url: "http://prod-api:8080/metrics"
      format: json
    metrics:
      - path: "production.requests"
        name: "prod_requests"

Development config:

scrapers:
  - name: dev_metrics
    source:
      type: url
      url: "http://localhost:8080/metrics"
      format: json
    metrics:
      - path: "debug.requests"
        name: "dev_requests"

Multi-Format Sources

Some sources expose multiple formats:

scrapers:
  # JSON endpoint
  - name: app_json
    source:
      type: url
      url: "http://app:8080/metrics.json"
      format: json
    metrics:
      - path: "custom.metric"
        name: "custom_value"
  
  # Prometheus endpoint
  - name: app_prom
    source:
      type: url
      url: "http://app:8080/metrics"
      format: prometheus
    metrics:
      - match: "standard_metric"
        name: "standard_value"

Best Practices

1. Name Scrapers Clearly

# Good
scrapers:
  - name: postgres_metrics
  - name: redis_cache_stats
  - name: app_performance

# Avoid
scrapers:
  - name: scraper1
  - name: data
  - name: metrics

2. Use Filtering

# Reduce payload and processing
filter:
  include:
    - "^needed_.*"
  exclude:
    - ".*_debug$"

3. Document Intent

scrapers:
  # Production database metrics
  # Collected every 60 seconds
  - name: postgres_production
    source:
      type: url
      url: "http://postgres-exporter:9187/metrics"
      format: prometheus
    metrics:
      # Active connections to monitor capacity
      - match: "pg_stat_database_numbackends"
        name: "pg_active_connections"

4. Test Sources Manually

Before adding a scraper:

# Test URL
curl http://localhost:8080/metrics

# Test file
cat /var/app/stats.json

# Verify format
curl http://localhost:8080/metrics | jq .

5. Start Small, Scale Up

# Start with minimal config
scrapers:
  - name: app_basics
    metrics:
      - path: "requests"
        name: "requests"

# Add more as needed
scrapers:
  - name: app_basics
    metrics:
      - path: "requests"
        name: "requests"
      - path: "errors"
        name: "errors"
      - path: "latency"
        name: "latency_ms"

Troubleshooting

Scraper Not Collecting

1. Check source is accessible
2. Verify URL/path is correct
3. Test manually with curl/cat
4. Check Probestyx logs

Wrong Values

1. Verify path/match is correct
2. Check calculation syntax
3. Ensure source format matches config
4. Test with simple metric first

Performance Issues

1. Reduce number of scrapers
2. Add filtering to limit metrics
3. Poll /metrics less frequently — scrapers aren't cached and re-run on every request (see Caching)
4. Check source response times
5. Use local files instead of remote URLs

Next Steps

• Integration - Connect Probestyx to Logstyx
• Custom Metrics - More custom metric examples