Parse incoming log events

Input parsers are applied on every log entry that is submitted to Logmatic.io.
They are really useful to extract attributes from semi-structured text to re-use them as fields or metrics.

How to access the input parsers?

If you need to enable or create parsers please click on the configure button on the side menu, and then on Enrichment & Parsing:

Enrichment & Parsing panel

Enrichment & Parsing panel

Available extensions

Available extensions are parsing rules that apply a specific usage. Logmatic.io provides a few of them and keeps on adding new ones continuously.
To enable an extension, you just need to click on the ON/OFF switch button and then press apply.

Enabling default extensions

Enabling default extensions

In less than a minute, Logmatic.io will create a dedicated parsing actor that will become part of your data enrichment chain.

Some extensions also create fields, metrics and groups that are interesting to have while the underlying log entries are properly parsed.
This is, for instance, the case for the "Apache" one, for which you get after activation fields like urls, repsonse codes, ip addresses, etc... and the "received bytes" metric.

Below we have an explanation of the available extensions.

Embedded JSON

JSON is an open standard format that uses human-readable text to transmit data objects consisting of attribute–value pairs.

{
  "firstName": "John",
  "lastName": "Smith",
  "isAlive": true,
  "age": 25,
  "height_cm": 167.6,
  "address": {
    "streetAddress": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postalCode": "10021-3100"
  },
  "children": [],
  "spouse": null
}

When the provided JSON is valid, this extension parses it and create all the associated fields.
Users are then able to get analytics over datasets in JSON by manipulating associated fields & metrics in the exploration view.

Key/Value

Some log entries define key-value pairs directly in the content of the message. If they follow the following syntax, Logmatic.io extracts them and creates associated fields and values in the final object.

To illustrate this, the following log line:

host=host1 app=app1 user=user1 otherfield= This is a message

Would be transformed into:

{
	"host":"host1",
	"app":"app1",
	"user":"user1",
	"message":"This is a message"
}

As you may have noticed, if a field has no value associated to it (eg otherfield in our example) it will be absent in the final object.

Commas are also accepted between pairs:

host=host1, app=app1, user=user1, otherfield= This is a message

Apache

This extension parses default Apache's server access log entries. Please refer to this documentation to get the details: Apache Log entries.

Recognized Apache common format - sample:

112.169.19.192 identity author [06/Mar/2013:01:36:30 +0900] \"GET / HTTP/1.1\" 200 44346

Recognized Apache combined format - sample:

112.169.19.192 identity author [06/Mar/2013:01:36:30 +0900] \"GET / HTTP/1.1\" 200 44346 \"some referrer\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22\"

Linux Secure log entries (or SSHD)

Linux secure log entries are, most of time, sent by syslog agents. They contain information about who got a success or a failure connecting to your machine through the SSH medium.
Find more details here: OpenSSH/Logging - WIKIBOOKS

There is not a unique format of SSHD log entries but many of them. This extension tries to extracte 4 kind of attributes out of these:

  • Action: the main action taken by the server implied by an SSH connection
  • Type: for some action, the type will give you an extra detail
  • User: we extract the username used during the action if defined
  • IP: we extract the IP address used during the action if defined

So for instance from a line of log like this:

Invalid user zxin10 from 114.215.110.81

We will extract:

{
  "message": "Invalid user zxin10 from 114.215.110.81",
  "sshd": {
    "action": "Invalid user",
    "user": "zxin10",
    "ip": "114.215.110.81"
  }
}

Create your own parser

Do you have your own log formats or you can't find our provided extensions?
Create you own parser by jumping to the next section.

Parse incoming log events

Input parsers are applied on every log entry that is submitted to Logmatic.io.
They are really useful to extract attributes from semi-structured text to re-use them as fields or metrics.