Docker Compose log aggregation using Grafana Loki and Fluent Bit

How to use Grafana Loki for docker compose log aggregation using Fluent Bit.

Recently Alex, my co-maintainer of MediathekView, asked me to build a Docker Compose file to view the MServer logs with Grafana Loki. I chose Fluent Bit as log proccessor and forwarder becuase Docker Compose can use the fluentd logging driver out of the box.

This does not seem particularly complex at first. However, there are some pitfalls hidden due to missing or outdated information.

The solution

Loki, Grafana and Fluent Bit

The Docker Compose part for Loki and Grafana is straightforward:

docker-compose-loki.yml

  version: "3.3"
  networks:
    loki:
      driver: bridge

  services:
    loki:
      image: grafana/loki:2.0.0
      container_name: loki
      ports:
        - "3100:3100"
      command: -config.file=/etc/loki/local-config.yaml
      networks:
        - loki

    grafana:
      image: grafana/grafana:latest
      container_name: grafana
      ports:
        - "3000:3000"
      networks:
        - loki

The fluent bit part is also straightforward, but unlike the sample Fluent Bit Docker Compose file, port 24224 is exposed. The port is required by the fluentd docker logging driver.

docker-compose-loki.yml

  fluent-bit:
    image: fluent/fluent-bit
    container_name: fluentbit
    ports:
        - "24224:24224"
        - "24224:24224/udp"
    volumes:
      - .docker/logging/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    networks:
      - loki

The two compose configuration can then be combined into one single compose file:

docker-compose-loki.yml

version: "3.3"
networks:
  loki:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.0.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    networks:
      - loki

  fluent-bit:
    image: fluent/fluent-bit
    container_name: fluentbit
    ports:
        - "24224:24224"
        - "24224:24224/udp"
    volumes:
      - .docker/logging/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    networks:
      - loki

Fluent Bit configuration

The configuration of Fluent Bit is almost identically to the example in the Fluent Bit documentation, but one important change must be made.

The label $sub['stream'] of the example isn’t applicable for Docker Compose logs. An example for a log line looks like this:

{"container_id":"05070542e00a616589a3671404d34fe7374f3b6f08ef6f71431acc42fbfa8675","container_name":"/mserver_mserver_1","source":"stdout","log":"[INFO ] [EtmMonitor] Shutting down JETM."}

Instead of $sub['stream'] I defined $container_id, $container_name and $source as labels.

fluent-bit.conf

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[Output]
    Name loki
    Match *
    host loki
    port 3100
    labels job=mserver,$container_id,$container_name,$source
    line_format json

Using the fluentd docker logging driver

To use the fluentd docker logging driver in conjunction with other Docker Compose this snippet can be used:

docker-compose.yml

    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

Sending the logs of nginx to Loki may look like this: .docker-compose.yml

version: "3.3"
services:
  nginx:
    image: nginx
    volumes:
    - ./templates:/etc/nginx/templates
    ports:
    - "8080:80"
    environment:
    - NGINX_HOST=foobar.com
    - NGINX_PORT=80
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

Grafana Loki show only Log message

When viewing the logs in the Grafana exploring view or with the Dashboard Log Panel, the individual lines are quite ugly. The entire json message is displayed, which quickly becomes very confusing. In order to get the log messages into a more readable format, one can use a new feature of Loki version 2.0: line_format

An example query for the container mserver_mserver_1:

{container_name="/mserver_mserver_1"} | json | line_format "{{.log}}"

Links

Some useful Links:

Thanks to Daniel for his review of this post