Production Deployment

This guide covers advanced deployment scenarios, custom configurations, and production hardening for Logstyx.

Make sure you read the Security Checklist beforehand

Understanding the Docker Architecture

Logstyx uses a multi-file Docker Compose setup for flexibility:

docker-compose.yml       # Infrastructure services (MongoDB, Redis, RabbitMQ)
docker-compose.prod.yml  # Application services (Auth, Core, Cache, etc.)
docker-compose.dev.yml   # Development overrides (hot-reload, debugging)

This separation allows you to:

Use bundled infrastructure or connect to external services
Switch between development and production easily
Customize configurations without modifying base files

Deployment Modes

Bundled Services (Default)

Use both compose files to run everything in Docker:

bash

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

What this includes:

MongoDB (with replica set)
Redis (with persistence)
RabbitMQ (with management UI)
All Logstyx microservices

Best for:

Quick start and testing
Small to medium deployments
Development environments
Single-server deployments

External Services

Use only the prod compose file when connecting to external infrastructure:

bash

docker compose -f docker-compose.prod.yml up -d

Configure external services in .env.encrypted:

bash

ENC_MONGODB_HOST=mongodb://your-mongo-cluster:27017
ENC_REDIS_URL=redis://your-redis-server:6379
ENC_AMQP_HOST=amqp://user:pass@your-rabbitmq:5672

Best for:

Production deployments
Using managed services (MongoDB Atlas, AWS ElastiCache, CloudAMQP)
Shared infrastructure across multiple applications
High-availability setups

Infrastructure Services Deep Dive

MongoDB Configuration

The bundled MongoDB is configured as a replica set, which is required for some Logstyx features:

yaml

logstyx_mongodb:
  image: mongo:latest
  command: ["--replSet", "rs0", "--bind_ip_all"]
  volumes:
    - mongo-data:/data/db
  healthcheck:
    test: |
      mongosh --quiet --eval "
      try { 
        rs.status(); 
      } catch(e) { 
        if (e.codeName === 'NotYetInitialized') {
          rs.initiate({
            _id: 'rs0',
            members: [{ _id: 0, host: 'logstyx_mongodb:27017' }]
          });
        }
      }" || exit 1

Key features:

Automatic replica set initialization
Health checks ensure readiness
Persistent storage via named volume
Internal port (not exposed by default)

Custom MongoDB configuration:

Create a custom compose override:

yaml

# docker-compose.override.yml
services:
  logstyx_mongodb:
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: secure_password
    ports:
      - "27017:27017"  # Expose if needed for external access
    command: 
      - --replSet
      - rs0
      - --bind_ip_all
      - --auth  # Enable authentication

Redis Configuration

Redis is configured with persistence and resource limits:

yaml

logstyx_redis:
  image: redis:7
  volumes:
    - redis-data:/data
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
  deploy:
    resources:
      limits:
        cpus: "0.5"
        memory: "256M"

Custom Redis configuration:

yaml

# docker-compose.override.yml
services:
  logstyx_redis:
    command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: "512M"

RabbitMQ Configuration

RabbitMQ includes the management UI for monitoring:

yaml

logstyx_rabbitmq:
  image: rabbitmq:3-management
  volumes:
    - rabbitmq-data:/var/lib/rabbitmq
  healthcheck:
    test: rabbitmq-diagnostics -q ping

Access RabbitMQ Management UI:

yaml

# docker-compose.override.yml
services:
  logstyx_rabbitmq:
    ports:
      - "15672:15672"  # Management UI
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: secure_password

Then access at http://localhost:15672

Application Services

All Logstyx microservices share a common base configuration:

yaml

x-microservice-base: &microservice-base
  env_file: .env.encrypted
  environment: &microservice-base-env
    AUTH_WSHOST: ws://logstyx_auth_service:5001/rpc
    CACHE_WSHOST: ws://logstyx_cache_service:5002/rpc
    UTILITY_WSHOST: ws://logstyx_utility_service:5003/rpc
    CORE_WSHOST: ws://logstyx_core_service:5004/rpc
    REDIS_URL: redis://logstyx_redis:6379
    MONGODB_HOST: mongodb://logstyx_mongodb:27017
    AMQP_HOST: amqp://logstyx_rabbitmq:5672
  logging:
    driver: "json-file"
    options:
      max-file: "3"
      max-size: "500k"
  restart: unless-stopped
  deploy:
    resources:
      limits:
        cpus: "1"
        memory: "512M"

Key features:

Shared environment variables via YAML anchors
Log rotation (3 files, 500KB each)
Auto-restart on failure
Resource limits per service
Health check dependencies

Production Hardening

1. Resource Limits

Adjust resource limits based on your workload:

yaml

# docker-compose.override.yml
services:
  logstyx_core_service:
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: "1G"
        reservations:
          cpus: "1"
          memory: "512M"

2. Logging Configuration

Prevent disk space issues with proper log rotation:

yaml

services:
  logstyx_core_service:
    logging:
      driver: "json-file"
      options:
        max-file: "10"
        max-size: "10m"
        compress: "true"

Or use a centralized logging driver:

yaml

services:
  logstyx_core_service:
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://logs.example.com:514"

3. Network Isolation

Create separate networks for different layers:

yaml

# docker-compose.override.yml
networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true  # No external access

services:
  logstyx_gateway_service:
    networks:
      - frontend
      - backend
  
  logstyx_mongodb:
    networks:
      - backend  # Only accessible from backend

4. Health Checks

Add custom health checks for application services:

yaml

services:
  logstyx_core_service:
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:5004/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

5. Security Hardening

Don't expose unnecessary ports:

yaml

# Only expose the gateway
services:
  logstyx_gateway_service:
    ports:
      - "127.0.0.1:5000:80"  # Only localhost access

Use secrets for sensitive data:

yaml

# docker-compose.override.yml
services:
  logstyx_mongodb:
    secrets:
      - mongo_root_password
    environment:
      MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo_root_password

secrets:
  mongo_root_password:
    file: ./secrets/mongo_root_password.txt

Run as non-root user (if images support it):

yaml

services:
  logstyx_redis:
    user: "999:999"  # redis user

Scaling Services

Logstyx microservices are stateless and can be horizontally scaled. The RPC client supports multiple endpoints with round-robin load balancing using comma-separated URLs.

How Scaling Works

Logstyx leverages Docker’s internal DNS load balancing. When you scale a service, Docker assigns multiple IP addresses to a single service name. Other microservices simply connect to that service name, and Docker distributes the traffic automatically.

Example: Scaling to Multiple Replicas

Step 1: Start services with scaling:

Run the scale command directly. Docker will spin up the requested number of containers for each service.

bash

docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.scale.yml up -d \
  --scale logstyx_auth_service=2 \
  --scale logstyx_cache_service=2 \
  --scale logstyx_utility_service=2 \
  --scale logstyx_core_service=3

Step 2: Verify scaling:

bash

# Check all replicas are running
docker compose ps

# Should show multiple replicas for each scaled service:
# logstyx_auth_service_1
# logstyx_auth_service_2
# logstyx_cache_service_1
# logstyx_cache_service_2
# ...

Step 3: Restart gateway to discover scaled instances:

bash

docker compose restart logstyx_gateway_service

This allows nginx to re-resolve DNS and load balance across all replicas.

Custom Scaling Configurations

You can create different scaling profiles for different scenarios:

Light load (2x core):

bash

docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.scale-light.yml up -d \
  --scale logstyx_core_service=2

Heavy load (5x core, 3x cache):

bash

docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.scale-heavy.yml up -d \
  --scale logstyx_core_service=5 \
  --scale logstyx_cache_service=3 \
  --scale logstyx_auth_service=2

Monitoring Scaled Services

Check resource usage across all replicas:

bash

docker stats

View logs from all replicas:

bash

# All replicas of a service
docker compose logs -f logstyx_core_service

# Specific replica
docker compose logs -f logstyx_core_service_1

Check individual replica health:

bash

# View all container statuses
docker compose ps

# Restart a specific replica
docker compose restart logstyx_core_service_2

Important Notes

Replica Naming Convention

When you scale a service, Docker Compose names replicas as:

service_name_1
service_name_2
service_name_3

Your RPC endpoint URLs must match this exact naming pattern. The number suffix starts at _1, not _0.

Persistent Scaling

To make scaling persistent across restarts, always include your scaling compose file:

bash

# Your standard startup command with scaling
docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.scale.yml up -d

Without the scaling file, services will revert to single instances.

Gateway Service

The logstyx_gateway_service doesn't inherit from x-microservice-base, so it doesn't need to be scaled. It acts as the single entry point and routes to the scaled backend services.

Backup and Restore

Backup MongoDB Data

bash

# Create backup
docker exec logstyx_mongodb mongodump --out=/backup --gzip
docker cp logstyx_mongodb:/backup ./mongodb-backup-$(date +%Y%m%d)

# Or use volumes directly
docker run --rm -v logstyx_mongo-data:/data -v $(pwd):/backup \
  mongo:latest tar czf /backup/mongo-backup.tar.gz /data

Backup Redis Data

bash

# Trigger save
docker exec logstyx_redis redis-cli BGSAVE

# Copy RDB file
docker cp logstyx_redis:/data/dump.rdb ./redis-backup-$(date +%Y%m%d).rdb

Restore from Backup

bash

# Stop services
docker compose down

# Restore MongoDB volume
docker run --rm -v logstyx_mongo-data:/data -v $(pwd):/backup \
  mongo:latest tar xzf /backup/mongo-backup.tar.gz -C /

# Restore Redis
docker cp ./redis-backup.rdb logstyx_redis:/data/dump.rdb

# Start services
docker compose up -d

Monitoring and Observability

Service Metrics

Check resource usage:

bash

docker stats logstyx_core_service logstyx_mongodb logstyx_redis

Container Logs

bash

# Follow all logs
docker compose logs -f

# Specific service
docker compose logs -f logstyx_core_service

# Last 100 lines
docker compose logs --tail=100 logstyx_core_service

# Since timestamp
docker compose logs --since="2024-01-18T10:00:00" logstyx_core_service

Export Logs

bash

# Export to file
docker compose logs --no-color > logstyx-logs-$(date +%Y%m%d).log

Custom Compose Override

Create docker-compose.override.yml for local customizations:

yaml

# This file is automatically loaded by docker compose
# Add it to .gitignore to keep local configs private

services:
  logstyx_gateway_service:
    ports:
      - "8080:80"  # Custom port
  
  logstyx_mongodb:
    ports:
      - "27017:27017"  # Expose for debugging
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: devpassword
  
  logstyx_rabbitmq:
    ports:
      - "15672:15672"  # Management UI

Environment-Specific Deployments

Development

bash

# If docker-compose.dev.yml exists
docker compose -f docker-compose.yml -f docker-compose.dev.yml up

Staging

yaml

# docker-compose.staging.yml
services:
  logstyx_core_service:
    image: devatlogstyx/logstyx-core-service:staging
    environment:
      LOG_LEVEL: debug

bash

docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d

Production

bash

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Troubleshooting

Service Won't Start

bash

# Check service status
docker compose ps

# View logs for specific service
docker compose logs logstyx_core_service

# Inspect container
docker inspect logstyx_core_service

MongoDB Replica Set Issues

bash

# Check replica set status
docker exec logstyx_mongodb mongosh --eval "rs.status()"

# Re-initialize if needed
docker exec logstyx_mongodb mongosh --eval "rs.initiate()"

Network Issues

bash

# List networks
docker network ls

# Inspect network
docker network inspect logstyx_default

# Test connectivity
docker exec logstyx_core_service ping -c 3 logstyx_mongodb

Volume Issues

bash

# List volumes
docker volume ls

# Inspect volume
docker volume inspect logstyx_mongo-data

# Check volume size
docker system df -v

Complete Cleanup

bash

# Stop and remove containers
docker compose down

# Remove volumes (DELETES DATA!)
docker compose down -v

# Remove networks
docker compose down --remove-orphans

# Remove images
docker compose down --rmi all

Production Deployment ​

Understanding the Docker Architecture ​

Deployment Modes ​

Bundled Services (Default) ​

External Services ​

Infrastructure Services Deep Dive ​

MongoDB Configuration ​

Redis Configuration ​

RabbitMQ Configuration ​

Application Services ​

Production Hardening ​

1. Resource Limits ​

2. Logging Configuration ​

3. Network Isolation ​

4. Health Checks ​

5. Security Hardening ​

Scaling Services ​

How Scaling Works ​

Example: Scaling to Multiple Replicas ​

Custom Scaling Configurations ​

Monitoring Scaled Services ​

Important Notes ​

Backup and Restore ​

Backup MongoDB Data ​

Backup Redis Data ​

Restore from Backup ​

Monitoring and Observability ​

Service Metrics ​

Container Logs ​

Export Logs ​

Custom Compose Override ​

Environment-Specific Deployments ​

Development ​

Staging ​

Production ​

Troubleshooting ​

Service Won't Start ​

MongoDB Replica Set Issues ​

Network Issues ​

Volume Issues ​

Complete Cleanup ​

Production Deployment

Understanding the Docker Architecture

Deployment Modes

Bundled Services (Default)

External Services

Infrastructure Services Deep Dive

MongoDB Configuration

Redis Configuration

RabbitMQ Configuration

Application Services

Production Hardening

1. Resource Limits

2. Logging Configuration

3. Network Isolation

4. Health Checks

5. Security Hardening

Scaling Services

How Scaling Works

Example: Scaling to Multiple Replicas

Custom Scaling Configurations

Monitoring Scaled Services

Important Notes

Backup and Restore

Backup MongoDB Data

Backup Redis Data

Restore from Backup

Monitoring and Observability

Service Metrics

Container Logs

Export Logs

Custom Compose Override

Environment-Specific Deployments

Development

Staging

Production

Troubleshooting

Service Won't Start

MongoDB Replica Set Issues

Network Issues

Volume Issues

Complete Cleanup