Ask A Question

Questions

1
ANSWERED

[Syslog-ng] How to handle and manage multi-line events like Java stack traces?

You want to aggregate Java stack traces and multi-line logs as one event, and not splitting them into several events. For instance, your logs are formatted as follow: ``` 2014-11-23 23:25:22,119 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2014-11-23 23:25:22,222 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2014-11-23 23:25:22,481 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:970) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:673) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) 2014-11-23 23:25:22,485 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task ``` Syslog-ng has a Rsyslog similar functionality for multi-line management. it is called the regexp multi-line mode. So you can have the following configuration: ``` #For logmatic platform bis template LogmaticFormat_bis { template("YOUR_API_KEY <${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $(indent-multi-line ${MESSAGE})\n"); }; destination d_logmatic_bis { tcp("api.logmatic.io" port(10514) template(LogmaticFormat_bis));}; rewrite escape { subst("\n","\\n", value("MESSAGE"), flags("global")); }; source s_source1 { file("/var/log/toto.log",flags(no-parse),follow_freq(1),program_override("myprogram"), multi-line-mode(regexp), multi-line-prefix(""^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}""));}; log {source(s_source1 );rewrite(escape); destination(d_logmatic_bis); }; ``` It is important to notice the `rewrite`. It was added because any `\n` in the stack trace is interpreted by TCP as the end of a message. So in order to make sure that our stack strace is correctly included in the message we need to replace all `\n` by another sign that will then be used in the parser.

Posted by Nils Bunge about a month ago