Centralize Docker Logs With Loki and Grafana

Before you begin

• A Linux host running Docker and Docker Compose.
• Comfort editing a few YAML files.
• Enough disk space for log storage, even if it is only a few gigabytes to start.
• A clear idea of which logs are for internal admin use and should not be publicly exposed.

docker logs is fine when one container failed five minutes ago. It becomes painful when you need to compare multiple services, review what happened overnight, or search for one request ID across an app and reverse proxy. Loki and Grafana give you a practical, open-source-first way to centralize logs without jumping straight into a heavyweight enterprise logging stack.

Step 1: Understand the moving parts

The stack here has three pieces:

• Loki stores and indexes logs efficiently.
• Promtail reads logs from the host and ships them to Loki.
• Grafana lets you search, filter, and visualize the logs.

This is a good fit for small operators because it is cheaper and simpler than a full ELK stack, but still much more usable than manual SSH sessions and ad hoc text filtering.

For a single Docker VPS, Promtail usually reads the Docker JSON log files from /var/lib/docker/containers. That works well as long as you understand where the files live and keep log retention under control.

Step 2: Create the stack directory and configuration files

Create a dedicated project directory:

mkdir -p ~/apps/logging/{loki,promtail,grafana}
cd ~/apps/logging

Create compose.yml:

services:
  loki:
    image: grafana/loki:3.0.0
    command: -config.file=/etc/loki/config.yml
    ports:
      - "127.0.0.1:3100:3100"
    volumes:
      - ./loki/config.yml:/etc/loki/config.yml:ro
      - loki-data:/loki
    restart: unless-stopped

  promtail:
    image: grafana/promtail:3.0.0
    command: -config.file=/etc/promtail/config.yml
    volumes:
      - ./promtail/config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "127.0.0.1:3001:3000"
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: change-this-now
      GF_SERVER_ROOT_URL: http://localhost:3001
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

Create loki/config.yml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/chunks

limits_config:
  retention_period: 168h

Create promtail/config.yml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*-json.log

This basic Promtail config reads all Docker JSON logs on the host. It is intentionally simple so you can get to a working system first.

Step 3: Start the stack and open Grafana safely

Start the services:

docker compose up -d

Check status and logs:

docker compose ps
docker compose logs --tail=50 loki promtail grafana

Because the ports are bound to 127.0.0.1, Grafana and Loki are not publicly exposed by default. Reach Grafana over SSH tunneling from your laptop:

ssh -L 3001:127.0.0.1:3001 user@your-vps

Then open http://localhost:3001 in your browser and sign in with the admin credentials you set.

Immediately change the default password after first login. Then add Loki as a data source:

1. Go to Connections then Data sources.
2. Choose Loki.
3. Set the URL to http://loki:3100.
4. Save and test.

If the test succeeds, Grafana can query Loki from inside the Compose network.

Step 4: Explore logs and improve labels

Open Grafana Explore and query:

{job="docker"}

You should start seeing container logs. At this stage, they may not yet be labeled with friendly container names. The fast path is proving ingestion first.

Once basic ingestion works, inspect a raw Docker log entry:

sudo head -n 5 /var/lib/docker/containers/$(docker ps -q | head -n 1)/*-json.log

If you want better labels, extend Promtail using Docker service discovery or pipeline stages later. For many beginners, a practical next step is simply identifying which containers are chatty and which errors appear repeatedly after deploys.

Also set Docker log rotation so the JSON files do not grow forever. In /etc/docker/daemon.json you can use:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then restart Docker during a maintenance window:

sudo systemctl restart docker

This does not replace centralized logging. It prevents local log files from consuming the whole disk.

Rollback and recovery notes

The logging stack is usually low-risk if you bind it to localhost and keep it separate from production apps. If problems appear:

• Stop the logging stack without touching the rest of your applications: docker compose down
• If Promtail creates too much disk activity, stop only that service first.
• If Grafana credentials were exposed, reset them before reopening access.

If Docker becomes unstable after changing daemon log rotation, restore the previous /etc/docker/daemon.json from backup or remove the new block and restart Docker again during a controlled window.

Step 5: Verify that the stack is operationally useful

By the end of this guide, you should have:

• Loki receiving Docker logs
• Grafana connected as a query interface
• A private access path to Grafana
• Basic retention and local Docker log rotation choices in place

The important outcome is not just “the containers are running.” It is being able to answer real questions faster when something breaks.

Troubleshooting common Loki and Promtail issues

Grafana cannot connect to Loki.
Make sure the data source URL is http://loki:3100 from inside Grafana, not localhost. In Compose, service-to-service traffic uses service names.

No logs appear in Explore.
Check docker compose logs promtail and verify the Docker JSON log path exists on the host.

Permission denied reading container logs.
Confirm the bind mount to /var/lib/docker/containers is present and read-only. Some hardened setups may need extra group or host permissions.

Disk usage keeps growing.
Reduce Loki retention, add Docker JSON log rotation, and review especially noisy containers.

I only see raw lines and poor labels.
That is normal for a first pass. Prove ingestion first, then improve scrape configs and labels once you know which metadata you actually need.

What to do next

Once logs are easier to search, the next useful habit is separating environment-specific Compose changes cleanly. Continue with Use Multiple Docker Compose Files for Dev, Staging, and Prod.