Ask A Question

Questions

1
ANSWERED

[Nxlog] How can I forward my logs on a local central server with Nxlog ?

If you want to centralize all your logs from several **originator servers** to a **collector server** before sending them to [logmatic.io](http://logmatic.io/), here is what you need to do: ##On the **Collector server** Add an input module into your collector server *nxlog.conf* file to listen to any tcp connections on a choosen port: ``` <Input input_collector> Module im_tcp Host 0.0.0.0 Port <your_choosen_port> </Input> ``` **The host field:** - This specifies the IP address or a DNS hostname which the module should listen on **to accept connections**. Because of security reasons the default listen address is localhost if this directive is not specified (the localhost loopback address is not accessible from the outside). You will most probably want to send logs from remote hosts, so make sure that the address specified here is accessible. **The any address 0.0.0.0 is commonly used here** Then edit your Route module of your collector server *nxlog.conf* file to forward all your log entries to logmatic.io: ``` ############ ROUTES TO CHOOSE ##### <Route 1> Path syslog,input_collector => out </Route> ``` ##On the **Originator server** Change the output module of your originator *nxlog.conf* file into: ``` <Output out> Module om_tcp Host <your_collector_server> Port <your_choosen_port> </Output> ``` **The host field:** This specifies the IP address or a DNS hostname to which the module should send the log entries

Posted by Pierre Guceski 3 years ago

1
ANSWERED

[Syslog-ng] How to handle and manage multi-line events like Java stack traces?

You want to aggregate Java stack traces and multi-line logs as one event, and not splitting them into several events. For instance, your logs are formatted as follow: ``` 2014-11-23 23:25:22,119 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2014-11-23 23:25:22,222 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2014-11-23 23:25:22,481 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:970) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:673) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) 2014-11-23 23:25:22,485 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task ``` Syslog-ng has a Rsyslog similar functionality for multi-line management. it is called the regexp multi-line mode. So you can have the following configuration: ``` #For logmatic platform bis template LogmaticFormat_bis { template("YOUR_API_KEY <${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $(indent-multi-line ${MESSAGE})\n"); }; destination d_logmatic_bis { tcp("api.logmatic.io" port(10514) template(LogmaticFormat_bis));}; rewrite escape { subst("\n","\\n", value("MESSAGE"), flags("global")); }; source s_source1 { file("/var/log/toto.log",flags(no-parse),follow_freq(1),program_override("myprogram"), multi-line-mode(regexp), multi-line-prefix(""^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}""));}; log {source(s_source1 );rewrite(escape); destination(d_logmatic_bis); }; ``` It is important to notice the `rewrite`. It was added because any `\n` in the stack trace is interpreted by TCP as the end of a message. So in order to make sure that our stack strace is correctly included in the message we need to replace all `\n` by another sign that will then be used in the parser.

Posted by Nils Bunge about a year ago

1
ANSWERED

[Syslog-ng] How can I replace part of a log message with syslog-ng?

To replace a part of the log message with Syslog-ng, you have to: - Define a string or regular expression to find the text to replace. - Define a string to replace the original text (macros work as well). - Select the field of the message that the rewrite rule should process. You can rewrite the structured-data fields of messages complying to the [RFC5424](https://tools.ietf.org/html/rfc5424) message format. **Substitution rules use the following syntax:** ``` rewrite <name_of_the_rule> { subst("<string or regular expression to find>", "<replacement string>", value(<field name>), flags() ); }; ``` The `type()` and `flags()` options are optional: - `type()` specifies the type of regular expression to use - `flags()` are the [flags](http://doc.logmatic.io/discuss/568cd56313c5ad0d00b34ea3) of the regular expressions. **The following example replace every occurence of IP in the text of the message with the string IP-Address:** ``` rewrite r_rewrite_subst{ subst("IP", "IP-Address", value("MESSAGE"), flags("global")); }; ``` A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect. **The following rules replace the first occurrence of the string IP with the string IP-Addresses:** ``` rewrite r_rewrite_subst{ subst("IP", "IP-Address", value("MESSAGE")); subst("Address", "Addresses", value("MESSAGE")); }; ```

Posted by Pierre Guceski 3 years ago

0

IIS to JSON

Attempting to ship IIS logs to Logmatic using NXlog. I’ve gone through countless articles trying multiple methods. Seems the data is getting into Logmatic but its just the raw message. We are trying to add fields to it to make it readable (Prettify) as we have done with other integrations. Any tips or ticks would be appreciated at this point. ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #To change for your own system if necessary define ROOT C:\Program Files (x86)\nxlog #define ROOT_STRING C:\Program Files (x86)\nxlog #define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log ##Extension to JSON <Extension json> Module xm_json </Extension> <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, \ $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, \ $sc-version, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, \ $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken, $ClientSourceIP FieldTypes integer, integer, string, string, integer, string, \ string, string, integer, string, integer, \ string, string, string, string, integer, \ integer, integer, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> ########## INPUTS ########### #Convert the IIS logs to JSON and use the original event time <Input iis_w3c> Module im_file File 'C:\ALB-WEB-05\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $SourceName= "IIS"; $Message = to_json(); } </Exec> </Input> ############ OUTPUTS ############## ##TCP output module <Output out> Module om_tcp Host api.logmatic.io Port 10514 Exec $raw_event="HIDDEN "+$raw_event; </Output> ############ ROUTES TO CHOOSE ##### <Route 1> Path iis_w3c => out </Route>

Posted by Joshua Jacobs about a year ago