Elastic Stack with Fluent Bit Helm for Kubernetes container logs

How to install Elastic Stack with Helm and get all Kubernetes container logs in to it with Fluent Bit.

The problem

The helm chart for Elastic Stack (stable/elastic-stack) seems to be pretty easy right? Yeah it is but if you install it with Fluent Bit activated it tries to find Fluentd and even if you install Fluentd too it can’t find it. And to get it even worse Fluentd can’t find Elasticsearch. But why?

Fluent Bit uses the wrong hostname of Fluentd and Fluentd uses the wrong hostname of Elasticsearch.

The solution

Fortunally Fluent Bit don’t need Fluentd to send its logs to Elasticsearch. So we just need to configure Fluent Bit to communicate with Elasticsearch and use the right hostname. The default helm values already include the right Elasticsearch hostname. Why it isn’t already set for Fluentd and Fluent Bit? I don’t know.

Here is my values file to get it working:

elastic-stack-fluentbit-values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
elasticsearch:
  enabled: true

kibana:
  enabled: true
  env:
    ELASTICSEARCH_HOSTS: http://{{ .Release.Name }}-elasticsearch-client:9200

logstash:
  enabled: false
  # elasticsearch:
  #   host: elastic-stack-elasticsearch-client

filebeat:
  enabled: false
  # config:
  #   output.file.enabled: false
  #   output.logstash:
  #     hosts: ["{{ .Release.Name }}-logstash:5044"]
  # indexTemplateLoad:
  #   - {{ .Release.Name }}-elasticsearch-client:9200

fluentd:
  enabled: false

fluent-bit:
  enabled: true
  backend:
    type: es
    es:
      host: {{ .Release.Name }}-elasticsearch-client

fluentd-elasticsearch:
  enabled: false

nginx-ldapauth-proxy:
  enabled: false
elasticsearch-curator:
  enabled: false

elasticsearch-exporter:
  enabled: false