Ask A Question

Questions

[Syslog-ng] How to handle and manage multi-line events like Java stack traces?

Posted by Nils Bunge 5 months ago

You want to aggregate Java stack traces and multi-line logs as one event, and not splitting them into several events.
For instance, your logs are formatted as follow:

2014-11-23 23:25:22,119 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2014-11-23 23:25:22,222 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2014-11-23 23:25:22,481 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.NullPointerException
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:970)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:673)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
2014-11-23 23:25:22,485 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Syslog-ng has a Rsyslog similar functionality for multi-line management.
it is called the regexp multi-line mode.

So you can have the following configuration:

#For logmatic platform bis
template LogmaticFormat_bis { template("YOUR_API_KEY <${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $(indent-multi-line ${MESSAGE})\n"); };
destination d_logmatic_bis { tcp("api.logmatic.io" port(10514) template(LogmaticFormat_bis));};
rewrite escape {
    subst("\n","\\n", value("MESSAGE"), flags("global"));
};
source s_source1 {
  file("/var/log/toto.log",flags(no-parse),follow_freq(1),program_override("myprogram"), multi-line-mode(regexp), multi-line-prefix(""^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}""));};
log {source(s_source1 );rewrite(escape); destination(d_logmatic_bis); };

It is important to notice the rewrite.
It was added because any \n in the stack trace is interpreted by TCP as the end of a message. So in order to make sure that our stack strace is correctly included in the message we need
to replace all \n by another sign that will then be used in the parser.

Loading comments...