Quarkus logging splunk

Introduction

Splunk is a middleware solution that receives, stores, indexes and finally allows to exploit the logs of an application.

This Quarkus extension provides the support of the official Splunk client library to index log events through the HTTP Event collection, provided by Splunk enterprise solution.

  • The official client is an opensource library available here.

  • The documentation of HTTP Event collection can be found here.

Installation

If you want to use this extension, you need to add the quarkus-logging-splunk extension first. In your pom.xml file, add:

<dependency>
    <groupId>io.quarkiverse.logging.splunk</groupId>
    <artifactId>quarkus-logging-splunk</artifactId>
</dependency>

Features

The extension can be used transparently with any log frontend used by Quarkus (Log4j, SLF4J, …​ ).

Structured vs raw log events

Unless quarkus.log.handler.splunk.raw is enabled, the extension uses structured logging.

The log message format is aligned with the one of Quarkus console handler:

quarkus.log.handler.splunk.format="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n"

By default, only few structured metadata computed by the HEC client library are sent. Extra metadata can be added via configuration (quarkus.log.handler.splunk.include-thread-name, etc.).

The extension also provides the support of the resolution of MDC scoped properties, as defined in JBoss supported formatters. MDC key/values are sent as indexed fields in the event.

A structured log event looks like:

{
  "time": "timestamp",
  "host": "hostname",
  "source": "mysource",
  "sourcetype": "mysourcetype",
  "index": "myindex",
  "event": {
    "message": "2021-10-07 ERROR The log message",
    "logger": "com.acme.MyClass",
    "severity": "ERROR",
    "exception": "java.lang.NullPointerException",
    "properties": {
      "mdc-key": "mdc-value"
    }
  }
}

Connectivity failures

Batched events that cannot be sent to the Splunk indexer will be logged to stdout:

  • Formatted using console handler settings if the console handler is enabled

  • Formatted using splunk handler settings otherwise

In any case, the root cause of the failure is always logged to stderr.

Asynchronous handler

By default, the log handler is synchronous and only the HTTP requests to HEC endpoint are done asynchronously:

sync

This can be an issue because the Splunk library #send is synchronized, so any preprocessing of the batch HTTP request itself happens on the application thread of the log event that triggered the batch to be full (either by reaching quarkus.log.handler.splunk.batch-size-count or quarkus.log.handler.splunk.batch-size-bytes)

By enabling quarkus.log.handler.splunk.async=true, an intermediate event queue is used, which decouples the flushing of the batch from any application thread:

async

By default quarkus.log.handler.splunk.async.queue-length=block, so applicative threads will block once the queue limit has reached quarkus.log.handler.splunk.async.queue-length.

There’s no link between quarkus.log.handler.splunk.async.queue-length and quarkus.log.handler.splunk.batch-size-count.

Sequential and parallel modes

The number of events kept in memory for batching purposes is not limited. After tuning quarkus.log.handler.splunk.batch-size-count and quarkus.log.handler.splunk.batch-size-bytes, in case the HEC endpoint cannot keep up with the batch throughput, using multiple HTTP connections might help to reduce memory usage on the client.

By setting quarkus.log.handler.splunk.send-mode=parallel multiple batches will be sent over the wire in parallel, potentially increasing throughput with the HEC endpoint.

Extension Configuration Reference

This extension follows the log handlers configuration domain that is defined by Quarkus, every configuration property of this extension will belong to the following configuration root : quarkus.log.handler.splunk

When present this extension is enabled by default, meaning the client would expect a valid connection to a Splunk indexer and would print an error message for every log created by the application.

So in local environment, the log handler can be disabled with the following property :

quarkus.log.handler.splunk.enabled=false

Every configuration property of the extension is overridable at runtime.

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property

Type

Default

Determine whether to enable the handler

boolean

true

The splunk handler log level. By default it is not more strict than the root handler level.

Level

ALL

Splunk HEC endpoint base url. The actual endpoint is expected at path /services/collector/events/1.0 for JSON events or /services/collector/raw for raw events.

string

https://localhost:8088/

Disable TLS certificate validation with HEC endpoint

boolean

false

The application token to authenticate with HEC, the token is mandatory if the extension is enabled https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/FormateventsforHTTPEventCollector#HEC_token

string

The strategy to send events to HEC. In sequential mode, there is only one HTTP connection to HEC and the order of events is preserved, but performance is lower. In parallel mode, event batches are sent asynchronously over multiple HTTP connections, and events with the same timestamp (that has 1 millisecond resolution) may be indexed out of order by Splunk.

sequential, parallel

sequential

A GUID to identify an HEC client and guarantee isolation at HEC level in case of slow clients. https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/AboutHECIDXAck#About_channels_and_sending_data

string

Batching delay before sending a group of events. If 0, the events are sent immediately.

Duration

10S

Maximum number of events in a batch. By default 10, if 0 no batching.

long

10

Maximum total size in bytes of events in a batch. By default 10KB, if 0 no batching.

long

10

Maximum number of retries in case of I/O exceptions with HEC connection.

long

0

The log format, defining which metadata are inlined inside the log main payload. Specific metadata (hostname, category, thread name, …​), as well as MDC key/value map, can also be sent in a structured way.

string

%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n

Whether to send the thrown exception message as a structured metadata of the log event (as opposed to %e in a formatted message, it does not include the exception name or stacktrace)

boolean

false

Whether to send the logger name as a structured metadata of the log event (equivalent of %c in a formatted message)

boolean

false

Whether to send the thread name as a structured metadata of the log event (equivalent of %t in a formatted message)

boolean

false

Overrides the host name metadata value.

string

The equivalent of %h in a formatted message

The source value to assign to the event data. For example, if you’re sending data from an app you’re developing, you could set this key to the name of the app. https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/FormateventsforHTTPEventCollector#Event_metadata

string

The source type value to assign to the event data https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/FormateventsforHTTPEventCollector#Event_metadata A given source type may have indexed fields extraction enabled, which is the case of the default built-in _json.

string

_json

The optional name of the index by which the event data is to be stored. If set, it must be within the list of allowed indexes of the token (if it has the indexes parameter set). https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/FormateventsforHTTPEventCollector#Event_metadata

string

Determines whether the events are sent in raw mode. In case the raw event (i.e. the actual log message) is not a JSON object you need to explicitly set a source type or Splunk will reject the event (the default source type, _json, assumes that the incoming event can be parsed as JSON)

boolean

false

Indicates whether to log asynchronously

boolean

false

The queue length to use before flushing writing

int

512

Determine whether to block the publisher (rather than drop the message) when the queue is full

block, discard

block

Optional collection of key/value pairs to populate the "fields" key of event metadata. This key isn’t applicable to raw data. https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/FormateventsforHTTPEventCollector#Event_metadata

Map<String,String>

About the Duration format

The format for durations uses the standard java.time.Duration format. You can learn more about it in the Duration#parse() javadoc.

You can also provide duration values starting with a number. In this case, if the value consists only of a number, the converter treats the value as seconds. Otherwise, PT is implicitly prepended to the value to obtain a standard java.time.Duration format.