Skip to main content
Mapagam

Implementing Centralized Logging

1. Installing Fluentd DaemonSet

Property Description Value
fluentd.image Container image for Fluentd fluent/fluentd:v1.15
fluentd.config Fluentd configuration file path /fluentd/etc/fluent.conf
fluentd.log.level Log level for Fluentd info

Example: Deploying Fluentd DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.15
        ports:
        - containerPort: 24224
          name: fluentd
        volumeMounts:
        - name: config-volume
          mountPath: /fluentd/etc/fluent.conf
          subPath: fluent.conf
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-config
Note: Ensure the Fluentd DaemonSet has appropriate RBAC permissions to access pod logs.

2. Configuring Elasticsearch

Property Value Description
elasticsearch.host localhost Hostname or IP address of the Elasticsearch instance
elasticsearch.port 9200 Port number for the Elasticsearch HTTP API
elasticsearch.index.prefix k8s-logs- Prefix for log indices in Elasticsearch
elasticsearch.index.retention.days 7 Number of days to retain log indices before deletion

Example: Elasticsearch Configuration in Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
data:
  elasticsearch.yml: |
    cluster.name: "kubernetes-logs"
    network.host: 0.0.0.0
    http.port: 9200
    index.prefix: "k8s-logs-"
    retention.days: 7
Note: Ensure that the Elasticsearch instance is accessible from the Kubernetes cluster and that proper network policies are in place.

3. Configuring Kibana

Configuration Option Description Default Value Notes
server.host Host address for Kibana to bind to "localhost" Change to "0.0.0.0" to allow external access
server.port Port number for Kibana server 5601 Ensure this port is open in firewall rules
elasticsearch.hosts Array of Elasticsearch host addresses ["http://localhost:9200"] Must point to the correct Elasticsearch instance
logging.level Log level for Kibana logs "info" Options: error, warn, info, debug
monitoring.ui.container.enabled Enable Kibana monitoring UI false Set to true for UI-based monitoring

Example: Basic Kibana Configuration

server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["http://elasticsearch-service:9200"]
logging.level: "info"
Note: Ensure that the Elasticsearch service is running and accessible from Kibana before starting Kibana.
Warning: Do not expose Kibana directly to the internet without authentication and TLS encryption.

Key Configuration Points

  • Kibana must connect to a valid Elasticsearch instance.
  • Server binding and port configuration should align with network policies.
  • Log levels can be adjusted for debugging or production environments.

4. Setting Up Log Forwarding

Component Description Configuration
Fluentd Open-source data collector for logs 
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
    </source>
    <match kubernetes.**>
      @type forward
      host logging-server.example.com
      port 24224
    </match>
Logstash Centralized log processing and forwarding 
input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
  }
}
Vector High-performance log collector and forwarder 
sources:
  kubernetes_logs:
    type: kubernetes_logs
    paths:
      - /var/log/containers/*.log
sinks:
  forward_to_elasticsearch:
    type: elasticsearch
    inputs:
      - kubernetes_logs
    endpoints:
      - http://elasticsearch:9200

Example: Forwarding Logs to Elasticsearch

apiVersion: v1
kind: ConfigMap
metadata:
  name: log-forwarder-config
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
    </source>
    <match kubernetes.**>
      @type elasticsearch
      host elasticsearch.example.com
      port 9200
      logstash_format true
    </match>
Note: Ensure the log forwarder has appropriate RBAC permissions to access Kubernetes logs.
Warning: Forwarding logs to external systems may introduce latency or data loss if not configured properly.

Key Considerations

  • Choose a log forwarder that supports Kubernetes log formats
  • Ensure network connectivity between nodes and the logging system
  • Configure log retention and rotation policies

5. Using Loki for Logging

Example: Deploying Loki with Grafana

apiVersion: v1
kind: Namespace
metadata:
  name: logging

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-deployment
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
      - name: loki
        image: grafana/loki:2.9.1
        ports:
        - containerPort: 3100
        args:
        - -config.file=/etc/loki/local-config.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: loki-service
  namespace: logging
spec:
  selector:
    app: loki
  ports:
  - port: 3100
    targetPort: 3100
Component Description Version
Loki Log aggregation system v2.9.1
Grafana Visualization and dashboarding tool v10.3.0
Note: Loki is designed to be lightweight and scalable, making it ideal for containerized environments like Kubernetes.

Example: Configuring Fluentd to Send Logs to Loki

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_key time
        time_format %Y-%m-%dT%H:%M:%S.%LZ
      </parse>
    </source>

    <match kubernetes.**>
      @type loki
      url http://loki-service.logging.svc.cluster.local:3100
      labels
        job <%= record["job"] %>
        namespace <%= record["namespace"] %>
        pod <%= record["pod"] %>
    </match>

Key Features of Loki

  • Highly scalable log aggregation system
  • Designed for containerized environments
  • Efficient storage and querying capabilities
  • Easy integration with Grafana for visualization

6. Installing Promtail

Component Description Notes
Promtail Log agent for Kubernetes that collects logs from pods and sends them to Loki Part of the Grafana stack for centralized logging

Example: Installing Promtail via Helm

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install promtail grafana/promtail

Example: Installing Promtail via Manifests

apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      serviceAccountName: promtail
      containers:
      - name: promtail
        image: grafana/promtail:latest
        args:
        - -config.file=/etc/promtail/promtail.yml
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: run
          mountPath: /run
      volumes:
      - name: config
        configMap:
          name: promtail-config
      - name: run
        hostPath:
          path: /run
Note: Ensure that the Loki instance is accessible from the cluster where Promtail is installed.

7. Querying Logs with LogQL

Operator Description Example
match Matches logs with specific labels {job="nginx"}
logql Used to filter and aggregate logs count by (level) (log)
count Counts occurrences of logs matching a pattern count by (level) (log)
rate Calculates the rate of log entries over time rate(log[5m])

Example: Basic Log Query

{job="nginx"} |= "error"

Example: Aggregating Logs by Level

count by (level) (log)

Example: Rate of Log Entries

rate(log[5m])

8. Setting Log Retention Policies

Policy Description Configuration Notes
log.retention.days Number of days to retain logs 30 Default is 7 days in some systems
log.retention.bytes Maximum size of logs before rotation 1GB Applies to each log file individually
log.rotation.count Number of rotated log files to keep 5 Used in conjunction with size or time-based rotation

Example: Configuring Log Retention in Fluentd

<source>
  @type tail
  path /var/log/pods/**/*.log
  pos_file /var/log/fluentd.pos
  tag kubernetes.*
  read_from_head true
</source>

<match kubernetes.**>
  @type file
  path /var/log/containers/${tag}
  rotate_age 30d
  rotate_size 1g
</match>
Note: Ensure that log retention policies align with compliance requirements and storage capacity constraints.

9. Configuring Structured Logging

Component Configuration Description
logFormat json Specifies the structured logging format for logs
logLevel info Set the minimum severity level for logs
outputPath /var/log/structured.log File path where structured logs are written

Example: Structured Logging Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: logging-config
data:
  logFormat: json
  logLevel: info
  outputPath: /var/log/structured.log
Note: Structured logs should be formatted as JSON for easy parsing and analysis by logging systems.

10. Troubleshooting Logging Stack

Common Logging Issues and Solutions

  1. Logs Not Appearing in Centralized System
    • Check if the logging agent (e.g., Fluentd, Logstash) is running and healthy.
    • Verify that the logging agent has proper permissions to read pod logs.
    • Ensure log paths are correctly configured in the logging agent's configuration.
  2. High Latency in Log Delivery
    • Monitor network connectivity between logging agents and the central logging system.
    • Check if the logging system is under heavy load or experiencing resource constraints.
    • Review the logging agent's buffer size and flush interval settings.
  3. Log Loss During Pod Restart
    • Ensure logging agents are configured to capture logs from the beginning of pod lifecycle.
    • Use persistent storage for log buffers if applicable.
  4. Incorrect Log Format or Parsing Errors
    • Validate log format in the logging agent configuration matches actual pod logs.
    • Use structured logging (e.g., JSON) to reduce parsing errors.
Issue Potential Cause Solution
Logs Missing in Centralized System Logging agent not running or misconfigured Check logging agent pods and logs for errors; verify configuration files
High Log Delivery Latency Network or logging system bottleneck Monitor network and logging system resources; adjust agent buffer settings
Log Loss on Pod Restart Logging agent not capturing from start Ensure logging agent is configured to capture logs from the beginning of pod lifecycle
Log Parsing Failures Incorrect or inconsistent log format Use structured logging (JSON) and validate log format in agent config

Example: Check Logging Agent Status

kubectl get pods -n logging
kubectl logs -n logging <logging-agent-pod-name>

Example: Validate Log Format in Agent Config

input:
  type: tail
  path: /var/log/pods/*/*/containers/*.log
  pos_file: /var/log/pos.log
  read_from_head: true
  format:
    type: json

Was this article helpful?

Related Cheat Sheets