Ask A Question

Questions

0

Can any one please help me to extract stack trace from following log

[Thu, 08 Mar 2018 07:28:27 GMT] [ERROR] default - POST : PostWorkflow Error :column "delpoymentStatus" of relation "workflows" does not exist [Thu, 08 Mar 2018 07:28:27 GMT] [ERROR] default - POST : PostWorkflow Error STACKTRACE:SequelizeDatabaseError: column "delpoymentStatus" of relation "workflows" does not exist at Query.formatError (C:\repo\Postgres\plan\services\node_modules\sequelize\lib\dialects\postgres\query.js:363:16) at query.catch.err (C:\repo\Postgres\plan\services\node_modules\sequelize\lib\dialects\postgres\query.js:86:18) at tryCatcher (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\util.js:16:23) at Promise._settlePromiseFromHandler (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\promise.js:512:31) at Promise._settlePromise (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\promise.js:569:18) at Promise._settlePromise0 (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\promise.js:614:10) at Promise._settlePromises (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\promise.js:689:18) at Async._drainQueue (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\async.js:133:16) at Async._drainQueues (C:\repo\Postgres\repo\services\node_modules\bluebird\js\release\async.js:143:10) at Immediate.Async.drainQueues (C:\repo\Postgres\plan\services\node_modules\bluebird\js\release\async.js:17:14) at runCallback (timers.js:672:20) at tryOnImmediate (timers.js:645:5)

Posted by jyo 7 months ago

0

IIS to JSON

Attempting to ship IIS logs to Logmatic using NXlog. I’ve gone through countless articles trying multiple methods. Seems the data is getting into Logmatic but its just the raw message. We are trying to add fields to it to make it readable (Prettify) as we have done with other integrations. Any tips or ticks would be appreciated at this point. ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #To change for your own system if necessary define ROOT C:\Program Files (x86)\nxlog #define ROOT_STRING C:\Program Files (x86)\nxlog #define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log ##Extension to JSON <Extension json> Module xm_json </Extension> <Extension w3c_parser> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, \ $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, \ $sc-version, $cs(User-Agent), $cs(Referer), $cs-host, $sc-status, \ $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken, $ClientSourceIP FieldTypes integer, integer, string, string, integer, string, \ string, string, integer, string, integer, \ string, string, string, string, integer, \ integer, integer, integer, integer, integer, integer Delimiter ' ' EscapeChar '"' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> ########## INPUTS ########### #Convert the IIS logs to JSON and use the original event time <Input iis_w3c> Module im_file File 'C:\ALB-WEB-05\W3SVC1\u_ex*.log' SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { w3c_parser->parse_csv(); $EventTime = parsedate($date + " " + $time); $SourceName= "IIS"; $Message = to_json(); } </Exec> </Input> ############ OUTPUTS ############## ##TCP output module <Output out> Module om_tcp Host api.logmatic.io Port 10514 Exec $raw_event="HIDDEN "+$raw_event; </Output> ############ ROUTES TO CHOOSE ##### <Route 1> Path iis_w3c => out </Route>

Posted by Joshua Jacobs 8 months ago

1
ANSWERED

[Syslog-ng] How to handle and manage multi-line events like Java stack traces?

You want to aggregate Java stack traces and multi-line logs as one event, and not splitting them into several events. For instance, your logs are formatted as follow: ``` 2014-11-23 23:25:22,119 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2014-11-23 23:25:22,222 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2014-11-23 23:25:22,481 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:970) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:673) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) 2014-11-23 23:25:22,485 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task ``` Syslog-ng has a Rsyslog similar functionality for multi-line management. it is called the regexp multi-line mode. So you can have the following configuration: ``` #For logmatic platform bis template LogmaticFormat_bis { template("YOUR_API_KEY <${PRI}>1 ${ISODATE} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} $(indent-multi-line ${MESSAGE})\n"); }; destination d_logmatic_bis { tcp("api.logmatic.io" port(10514) template(LogmaticFormat_bis));}; rewrite escape { subst("\n","\\n", value("MESSAGE"), flags("global")); }; source s_source1 { file("/var/log/toto.log",flags(no-parse),follow_freq(1),program_override("myprogram"), multi-line-mode(regexp), multi-line-prefix(""^(\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}""));}; log {source(s_source1 );rewrite(escape); destination(d_logmatic_bis); }; ``` It is important to notice the `rewrite`. It was added because any `\n` in the stack trace is interpreted by TCP as the end of a message. So in order to make sure that our stack strace is correctly included in the message we need to replace all `\n` by another sign that will then be used in the parser.

Posted by Nils Bunge about a year ago