The documentation is open-source and user-friendly. Feel free to contribute by clicking on "suggest edit" on any page. Let’s log!
First of all, we advise you to get later versions of Syslog-NG. By checking the changelog, the TLS TCP client seem to have several serious bug issues that have been fixed. This problem actually arises because Logmatic.io cuts any TCP connection after 2 minutes of unactivity. For some reason, there are Syslog-NG versions that are not able to properly reconnect properly when necessary. To mitigate this issue we propose to use time markers so the connection never stops. To do so, add time markers every 60 seconds in your options: ``` options {mark_freq(60)}; ``` And don't forget to restart: ``` sudo service syslog-ng restart ```
Posted by Renaud Boutet 3 years ago
Elastic Beanstalk allows you to place a configuration file in your app’s root directory to configure the instance it is launched on. In the root of your app’s directory, create the following hidden directory, `.ebextensions` and inside that `logmaticBeanstalk.config` Then copy the following into the config file seen below. ``` commands: 01edit-rsyslog-conf: command: echo '$template LogmaticFormat,"API_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n"' > /etc/rsyslog.d/logger.conf 02add-second-line: command: echo '*.* @@api.logmatic.io:10514;LogmaticFormat' >> /etc/rsyslog.d/logger.conf 03restart-syslog: command: service rsyslog restart ```
Posted by Pierre Guceski 3 years ago
I am working on building a custom endpoint to send Logmatic alerts for PagerDuty. I am difficulty getting the "link" data to transmit properly to PagerDuty. When I intercepted the webhook using Requestb.in, it received the following: ```json { "service_key":"*****", "event_type":"trigger", "description":"PROD Description", "client":"Logmatic", "client_url":"https://app.logmatic.io/logmatic#analyse/alerts/564aa7f23f54230a00992df8" } ``` Notice how the client_url field has `/` instead of `/`. The following is how I have the endpoint configured in Logmatic. ```json { "service_key": "*****", "event_type": "trigger", "description": "{{title}}", "client": "Logmatic", "client_url": "{{link}}" } ``` Is there something I can do in the configuration of the message to make it maintain the proper URL formatting?
Posted by gpolaert 3 years ago
In order to avoid having `127.0.0.1` as perfstats to syslog raw-event hostname with syslog-ng you need to add the following configuration in `/etc/syslog-ng/syslog-ng.conf` : In global option (top of the configuration file) : ``` options { chain_hostnames(off); (...) keep_hostname(yes); (...) }; ```
Posted by Pierre Guceski 3 years ago
In most compose files you have a command as the following: ``` hello_world: image: busybox ports: - "80:80" command: echo hi ``` This works fine with docker-compose. The only way to be able to represent it with ecs-cli compose is via the following construct: ``` hello_world: image: busybox ports: - "80:80" command: - echo - hi ``` So for the logmatic-docker you need to pass: ``` command: -"xxxxxxxx" -"-a" -"label=integration" -"--matchByImage" -"hub" ```` Fix inspired from this [issue](https://github.com/aws/amazon-ecs-cli/issues/2)
Posted by Pierre Guceski 3 years ago
In most compose files you have a command as the following ``` hello_world: image: busybox ports: - "80:80" command: echo hi This works fine with docker-compose. ``` The only way to be able to represent it with ecs-cli compose is via the following awkward construct: ``` hello_world: image: busybox ports: - "80:80" command: - echo - hi ``` So for the logmatic-docker you need to pass: ``` command: -"xxxxxxxx" -"-a" -"label=integration" -"--matchByImage" -"hub" ``` *Fix inspired from this [issue](https://github.com/aws/amazon-ecs-cli/issues/2)*
Posted by Pierre Guceski 3 years ago
Filtering is now possible in any module using the drop() procedure conditionally in the Exec directive. Example of dropping logs conditionally in the output tcp module: ``` ############ OUTPUTS ############## ##TCP output module <Output out> Module om_tcp Host api.logmatic.io Port 10514 ##Drop Debug logs Exec if $raw_event =~ /^Debug/ drop(); ##Add the API key before the event Exec $raw_event="<your_api_key> "+$raw_event; </Output> ```
Posted by Pierre Guceski 3 years ago
The syslog-ng OSE application uses a regular expression to detect credit card numbers, and provides two ways to accomplish this: - `credit-card-mask(value("<message-field-to-process>"))` Process the specified message field (by default, ${MESSAGE}), and replace the 7-12th character of any credit card numbers (Primary Account Number or PAN) with asterisks (*). For example, syslog-ng OSE replaces the number 5542043004559005 with 554204******9005. - `credit-card-hash(value("<message-field-to-process>"))` Process the specified message field (by default, ${MESSAGE}), and replace any credit card numbers (Primary Account Number or PAN) with its 16-character-long SHA-1 hash. **Usage:** ``` @include "scl/rewrite/cc-mask.conf" rewrite { credit-card-mask(value("<message-field-to-process>")); }; ``` By default, these rewrite rules process the MESSAGE part of the log message.
Posted by Pierre Guceski 3 years ago
There are many ways to modify log entries with Nxlog, - The simple method (which does not always work): To modify the $raw_event field (in case of syslog) without parsing the log. This can be done with [regular expressions](http://doc.logmatic.io/discuss/568cd99e13c5ad0d00b34ea8) using capturing, for example: ``` # Set the $EventTime field usually found in the logs by extracting it with a regexp. # If this is not set, the current system time will be used which might be a little off. Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1); ``` - The more complex method: To parse the log into fields, modify some fields and finally reconstruct the log from the fields: Here is an example changing the $severityValue to `error` of a log containing the word "error" in its `message` field: ``` (....) <Processor rewrite> Module pm_null Exec parse_syslog_bsd();\ if $Message =~ /error/ \ {\ $SeverityValue = syslog_severity_value("error");\ to_syslog_bsd(); \ } </Processor> (...) ############ ROUTES TO CHOOSE ##### <Route 1> Path syslog => rewrite => out </Route> ``` If you want to learn more on Nxlog regular expression and text replacement, please refere to [Nxlog documentation](https://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928)
Posted by Pierre Guceski 3 years ago
With Nxlog, the PCRE engine is used to to execute the regular expressions, so: - Regular expressions must be quoted with slashes as in Perl. - `=~` is the regular expression match operation as in Perl. Captured substrings are accessible through a numeric reference such as $1. **The full subject string is placed into $0**. - `!~` opposite of `=~` (the expression will evaluate to TRUE if the regular expresion does not match on the subject string). - the `s///` operator (Regexp based string substitution) is also supported as in Perl . The following regular expression modiers are supported: - The `/g` modifier can be used for global replacement. - The `/s` modifier can be used to have the `.` (The `.` normally matches any character except newline) match all characters including line terminator characters (LF and CRLF). - The `/m` modifier can be used to treat the string as multiple lines, i.e. `^` and `$` match newlines within data. - The `/i` modifier does case insensitive matching. If you want to learn more on Nxlog regular expression and text replacement, please refere to [Nxlog documentation](https://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928)
Posted by Pierre Guceski 3 years ago
If you want to centralize all your logs from several **originator servers** to a **collector server** before sending them to [logmatic.io](http://logmatic.io/), here is what you need to do: ##On the **Collector server** Add an input module into your collector server *nxlog.conf* file to listen to any tcp connections on a choosen port: ``` <Input input_collector> Module im_tcp Host 0.0.0.0 Port <your_choosen_port> </Input> ``` **The host field:** - This specifies the IP address or a DNS hostname which the module should listen on **to accept connections**. Because of security reasons the default listen address is localhost if this directive is not specified (the localhost loopback address is not accessible from the outside). You will most probably want to send logs from remote hosts, so make sure that the address specified here is accessible. **The any address 0.0.0.0 is commonly used here** Then edit your Route module of your collector server *nxlog.conf* file to forward all your log entries to logmatic.io: ``` ############ ROUTES TO CHOOSE ##### <Route 1> Path syslog,input_collector => out </Route> ``` ##On the **Originator server** Change the output module of your originator *nxlog.conf* file into: ``` <Output out> Module om_tcp Host <your_collector_server> Port <your_choosen_port> </Output> ``` **The host field:** This specifies the IP address or a DNS hostname to which the module should send the log entries
Posted by Pierre Guceski 3 years ago
To replace a part of the log message with Syslog-ng, you have to: - Define a string or regular expression to find the text to replace. - Define a string to replace the original text (macros work as well). - Select the field of the message that the rewrite rule should process. You can rewrite the structured-data fields of messages complying to the [RFC5424](https://tools.ietf.org/html/rfc5424) message format. **Substitution rules use the following syntax:** ``` rewrite <name_of_the_rule> { subst("<string or regular expression to find>", "<replacement string>", value(<field name>), flags() ); }; ``` The `type()` and `flags()` options are optional: - `type()` specifies the type of regular expression to use - `flags()` are the [flags](http://doc.logmatic.io/discuss/568cd56313c5ad0d00b34ea3) of the regular expressions. **The following example replace every occurence of IP in the text of the message with the string IP-Address:** ``` rewrite r_rewrite_subst{ subst("IP", "IP-Address", value("MESSAGE"), flags("global")); }; ``` A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect. **The following rules replace the first occurrence of the string IP with the string IP-Addresses:** ``` rewrite r_rewrite_subst{ subst("IP", "IP-Address", value("MESSAGE")); subst("Address", "Addresses", value("MESSAGE")); }; ```
Posted by Pierre Guceski 3 years ago
By default, Syslog-ng uses PCRE-style regular expressions To [use other expression types](https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-regexp-types.html), add the `type()` option after the regular expression PCRE regular expressions in Syslog-ng have the following **flag options**: - **global**: Usable only in rewrite rules; match for every occurrence of the expression, not only the first one. - **ignore-case**: Disable case-sensitivity. - **store-matches**: Store the matches of the regular expression into the $1, ... $255 variables. Named matches (also called named subpatterns), for example (?<name>...), are stored as well. Matches from the last filter expression can be referenced in regular expressions. - **unicode**: Use Unicode support for UTF-8 matches: UTF-8 character sequences are handled as single characters. - **utf8**: An alias for the unicode flag. If you want to learn more on Syslog-Ng regular expression please refere to [Syslog-ng documentation](https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/regular-expressions.html)
Posted by Pierre Guceski 3 years ago
To have some influence on the rate limiting we have basically two options: ``` $SystemLogRateLimitInterval [number] $SystemLogRateLimitBurst [number] ``` The SystemLogRateLimitInterval determines the amount of time that is being measured for rate limiting. By default this is set to 5 seconds. The SystemLogRateLimitBurst defines the amount of messages, that have to occur in the time limit of SystemLogRateLimitInterval, to trigger rate limiting. Here, the default is 200 messages. To change these settings, open the rsyslog configuration: ``` vi /etc/rsyslog.conf ``` Then search the right spot for the entries, find the following: ``` $ModLoad imuxsock.so ``` Now insert two new lines under the ModLoad command and fill them as follows: ``` $SystemLogRateLimitInterval 2 $SystemLogRateLimitBurst 50 ``` This means in plain words, that rate limiting will take effect if more than **50 messages** occur in **2 seconds**.
Posted by Pierre Guceski 3 years ago
Logmatic.io doesn’t provide standard log format for Tomcat or Java as they are all slightly different. But if you are running java code we advise you to go for JSON. For 3 reasons: Stack Traces are not multiline anymore and included in the event Add context to your log: customer ids, client ids or anything related to your application No formatting and then parsing process required. When you add new contextual attributes, no need to change the format and parsers.
Posted by Pierre Guceski 3 years ago