Configure Spring Logging To Elasticsearch

Configure Spring Logging To Elasticsearch

by

Overview

In this quick tutorial, I will show you how to configure Spring logging to Elasticsearch so you can search, visualize and do many useful things with your log. Things that you cannot do by looking at the console output.

Let’s get started.

Create ELK stack using Docker Compose

If you don’t have Elasticsearch running already, you can use the following file to create a running ELK stack on your docker environment in just a few minutes:

version: '3'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - cluster.name=elasticsearch-cluster
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:7.9.3
    container_name: kibana
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    volumes:
      - ./data/kibana:/usr/share/kibana/data
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

  logstash:
    image: docker.elastic.co/logstash/logstash:7.9.3
    container_name: logstash
    volumes:
      - ./data/logstash:/usr/share/logstash/data
    ports:
      - 9600:9600
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    depends_on:
      - elasticsearch

This starts Elasticsearch, Kibana, and Logstash on the default ports. If you have ports conflict, change to different ones.

Now the ELK stack is ready, let’s create a Spring Boot application and configure it to log into Elasticsearch.

Create an index to store logging messages on Elasticsearch

To store your logs on Elasticsearch, you need to create an index first. There are various ways to accomplish this task. However, I prefer to use Kibana because it’s visual.

Head to your Kibana URL (if you follow my setup, the URL should be http://localhost:5601/app/dev_tools#/console) and execute the following request to create an index named spring-app-index:

Query to create a new index in Kibana
Query to create a new index in Kibana

Configure Spring logging to Elasticsearch

Here is the pom file of the Spring boot application:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.datmt.spring</groupId>
    <artifactId>spring-logging-elasticsearch</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>spring-logging-elasticsearch</name>
    <description>spring-logging-elasticsearch</description>
    <properties>
        <java.version>11</java.version>
    </properties>
    <dependencies>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
        <dependency>
            <groupId>org.appenders.log4j</groupId>
            <artifactId>log4j2-elasticsearch-jest</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

There are some interesting points in this file:

  • Line 21 to 30 exclude the default logging dependency of Spring (which is Logback).
  • Line 32 to 35 add log4j2 as the logging framework for this app
  • Line 36 to 40 is the meat of this application. It’s a library that helps you write log to Elasticsearch.

The next step is to configure logging in log4j2.

Configure log4j2.xml

Create a file named log4j2.xml in the src/main/resources directory with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <Elasticsearch name="Elasticsearch">
            <IndexName indexName="spring-app-index" />
            <JacksonJsonLayout />
            <AsyncBatchDelivery>
                <IndexTemplate name="spring-app-index" path="classpath:indexTemplate.json" />
                <JestHttp serverUris="http://localhost:9200" />
            </AsyncBatchDelivery>
        </Elasticsearch>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="Elasticsearch"/>
        </Root>
    </Loggers>
</Configuration>

Some important things you need to know about this file:

  • On line 8, you need to put the name of the index (created earlier in this post)
  • On line 11, the path option points to a file name indexTemplate.json (also inside the resources folder). You can use this file to configure the mapping between the log messages and fields inside the index
  • On line 12, this points to the address of your Elasticsearch instance.

As mentioned, here is the content of indexTemplate.json

{
  "index_patterns": [
    "spring-app-index-*"
  ],
  "template": {
    "mappings": {
      "properties": {
        "loggerName": {
          "type": "keyword",
          "index": false
        },
        "message": {
          "type": "keyword",
          "index": true
        },
        "timeMillis": {
          "type": "date",
          "format": "epoch_millis"
        },
        "thread": {
          "type": "keyword",
          "index": false
        },
        "level": {
          "type": "keyword",
          "index": false
        }
      }
    }
  }
}

Write log and view logs in Kibana

Now let’s add some simple logging messages to the application.

@SpringBootApplication
public class SpringLoggingElasticsearchApplication {

    public static void main(String[] args) {
        Logger logger = LoggerFactory.getLogger(SpringLoggingElasticsearchApplication.class);
        
        SpringApplication.run(SpringLoggingElasticsearchApplication.class, args);
        logger.info("Starting application");
    }
}

Let’s run the app.

Now, if you execute the following query in Kibana, you should see the log message appear:

GET /spring-app-index/_search?size=100
{
  "query": {
    "match_all": {}
  }
}
Log message successfully delivered to Elasticsearch
Log message successfully delivered to Elasticsearch

Conclusion

This post shows you how to quickly set up Spring logging to Elasticsearch and view the messages in Kibana. There could be many other tweaks you need to do to suit your application needs but this would give you a quick start.

As usual, the code is available on GitHub.

Leave a Comment