Basics to send data

There are various ways to send data to Logmatic.io. You can send events manually, through applications or through what are called log shippers.

Before you have fun with your data

You need to send it to Logmatic.io. No rush though... Let's go over some basics first...

Sending data to logmatic.io

The API key

The API key is your identifier when sending data to us. It is required if you want to send data manually or with any log shippers

In order to find it, just click on the Configure button on the Side Menu and on the API keys sub-menu:

API Key overview

API Key overview

When your platform is provisioned, a key will be generated automatically. But if need be, you can generate new ones.

This key will be referred to as <your_api_key> in all further steps.
Learn more here.

3 Available endpoints

As we mentioned, there is many ways to send data to Logmatic.io. We'll help you decide which one is best suited to your usage. If you want to understand what is behind this, you need to know that there are only 3 exposed input API endpoints:

Therefore, all further examples and use cases found in this documentation will always rely on one of these.

Each endpoint has advantages and drawbacks. They are all summarized in the following table:

Protocol
Secure
Garanteed delivery
Endpoint

NO

YES

api.logmatic.io PORT: 10514

YES

YES

api.logmatic.io PORT: 10515

NO

NO

api.logmatic.io PORT: 514

Best practices to send logs in realtime

Many open source projects can help you send machine data in real time to a remote machine. To make things more simple, we selected a few solutions that are good standards to ensure great performance, functionality, security, and reliability.

This table aims to present an overview of these options:

Name OS File fwd Enrichment features Endpoint used
Rsyslog Linux YES NO TCP/S, UDP
Nxlog Windows / Linux YES NO TCP/S
Logstash Windows / Linux YES YES HTTP/S, TCP/S, UDP
Syslog-NG Windows / Linux YES YES HTTP/S, TCP/S, UDP
Fluentd Windows/ Linux YES YES HTTP/S, TCP/S, UDP
Agent less Windows / Linux NO NO HTTP/S, TCP/S, UDP

Let's explain a few things here:

  • File fwd means that the log shippers can watch a file and push any appended content to a remote process
  • Enrichment features means that the log shippers can enrich incoming messages on-the-fly before sending them.

With Agentless we regroup all the solutions by directly calling one of the endpoints within your applications with:

or whatever other language you may use. You will find more information about this in the Log Integration Cases directory.

Reserved fields

the date field

By default Logmatic.io generates a timestamp that corresponds to the reception date. This field is used to display your data over the timeline.

However, you can decide to assign your own timestamps by assigning a set of reserved fields. These reserved fields are: timestamp, date, _timestamp, Timestamp, eventTime and published_date. Syslog's timestamps are also taken into account automatically.

The recognized date formats are: ISO8601, UNIX (the milliseconds EPOCH format) and RFC3164.

{
  "date":"2014-04-22T06:00:00Z",
}

the message field

By default, Logmatic.io considers the message field as the content to display in priority. This is also the only attribute indexed as text, so you can search over any tokenized word without mentioning the path to the attribute.

{
  //...
  "message":"<your_message>",
  //...
}

Recognized log content & parsers

When plain text is sent to Logmatic.io, the system tries to check if there are any regular syslogs.
If you need to extract more value from your logs (apache, key-value, JSON, etc...), you can also enable available parsing extensions or create new ones.
Check the chapter about input parsers here.

Recognized syslogs

Syslog is the standard for computer message logging. Syslog can be used for computer system management and security auditing, as well as generalized informational, analytical, and debugging messages. It is supported by a wide variety of devices (eg. printers and routers) and receivers across multiple platforms. Because of this, syslog can be used to integrate log data from many different types of systems into a central repository.

You can provide properly formatted syslog logs as long as they respect the RFC-5424 standard.

Once the message is extracted, Logmatic.io tries to extract other patterns like JSON or Key-Value pairs. This is very useful since syslog can become your main format to provide any type of log event.

Let's look at an example. The line below is a valid RFC-5424 syslog:

<85>0 2014-09-01T13:05:06.048413+00:00 client-logs sshd - - -  pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.34.120  user=root

It is automatically recognized as a syslog with key-value pairs in it and transformed into:

{
  "message": "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.34.120  user=root",
  "custom": {
    "uid": 0,
    "euid": 0,
    "tty": "ssh",
    "rhost": "222.186.34.120",
    "user": "root",
  },
  "syslog":  {
    "prival": 85,
    "severity": 5,
    "facility": 10,
    "version": 0,
    "appname": "sshd",
    "hostname": "client-logs",
    "timestamp": "2014-09-01T13:07:57.922742+00:00"
  }
}

Next steps?

  • If you are running under Unix/Linux you should visit here.
  • If you are running under Windows here.
  • If you are already using Logstash go here.
  • If you are already using Fluentd go here

Basics to send data

There are various ways to send data to Logmatic.io. You can send events manually, through applications or through what are called log shippers.