Log file parser plugin HLD

Requirement

Overview

The purpose of this plugin is to parse different log files using set of rules given by the user. Main work is done using regular expressions provided in configuration file. Plugin architecture is designed to be as much generic as possible.

Design abstract

Plugin makes use of src/utils_message_parser.c utility which takes care of reading new data from file, searching for regular expressions, storing output information.

Main structure containing data used by message parser utility:

Simplified data flow:

Message items produced by message_parser_read() is postprocessed and notification compliant with collectd is dispatched. All details needed are stored in meta-data structure inside.

Configuration

Main section

<Plugin log_parser>
  <Logfile "file01.log">
    FirstFullRead false
    <Message "msg_1">
       …
    </Message>
    <Message "msg_2">
       …
    </Message>
      …
    <Message "msg_NN">
       …
    </Message>
  </Logfile>
  <Logfile "file02.log">
     …
  </Logfile>
   …
  <Logfile "fileNN.log">
     …
  </Logfile>
</Plugin>

Configuration can consist of several <Logfile> sections allowing to parse multiple files at once.
FirstFullRead [true | false] – if set to true then reads whole content, otherwise looks only for newly arrived data.

Message section

<Message "PCI_Error">
  DefaultPluginInstance "plugin_instance"
  DefaultType "type"
  DefaultTypeInstance "type_instance"
  DefaultSeverity "ok"
  <Match "sample_error">
     …
  </Match>
  <Match "error time">
   …
  </Match>
     …
  <Match "match_name">
   …
  </Match>
</Message>

DefaultPluginInstance [string] – Sets plugin_instance
DefaultType [string] – Sets type
DefaultTypeInstance [string] – Sets type_instance
DefaultSeverity [string] – Sets severity

These options are used in case we want to have default values. They can be overridden by matches results values.

Match section

  <Match "sample_error">
    Regex "MSG:.*error occurred"
    SubmatchIdx -1
  </Match>
  <Match "error time">
    Regex "(... .. ..:..:..) .* pcieport.*AER"
    IsMandatory false
  </Match>
   …
  <Match "match_name">
     …
  </Match>

Match section used to get severity, type, type instance and plugin instance

  <Match "severity">
    Severity "warning"
    Regex "severity=([wW]arning|warn)"
  </Match>
  <Match "type">
    Type true
    Regex "…"
  </Match>
  <Match "type_instance">
    TypeInstance true
    Regex "…"
  </Match>
  <Match "plugin_instance">
    PluginInstance true
    Regex "…"
  </Match>

<Match [name]> – Definition of regular expression
               Regex [string] – Regular expression matching string. May contain subexpressions.
               SubmatchIdx [-1..n] – Index of subexpression to be used for notification. Default is 1. Setting it to -1 means that it is ignored.
               ExcludeRegex [string] – Regular expression for excluding lines containing specific matching strings.  This is processed before checking Regex pattern. It is optional and can be omitted.
               IsMandatory [true | false] – Indicates whether Match item is mandatory for message validation. If set to true, whole message is discarded if it's missing. For false its presence is optional. Default value is set to true.

               Severity [string] – Match result will be put into severity notification field.
               PluginInstance [true | string] – Match result will be put into plugin_instance notification field.
               Type [true | string] – Match result will be put into type notification field.
               TypeInstance [true | string] – Match result will be put into type_instance notification field.

Setting above four options to string will force plugin to ignore matching string and take the given one.
Note: Severity cannot be set to boolean and its value has to be “ok”, “warning” or “failure” as the only provided by collectd.

Key dependencies

Issues list

None