Log file parser plugin HLD
Requirement
Abstract | Notice | |
1.0 | Accept a number of log files | Each file should be specified in separate section in configuration |
2.0 | Find matches using regular expressions | |
3.0 | Count matches | Matches found using regular expressions should be counted |
4.0 | Dispatch collectd notification when hit occurs | Configuration should allow setting different severity depending on different keywords found in message |
5.0 | Attach corresponding log file part in notification meta data | Operation of extracting desired fields should be done during sending notifications |
Overview
The purpose of this plugin is to parse different log files using set of rules given by the user. Main work is done using regular expressions provided in configuration file. Plugin architecture is designed to be as much generic as possible.
Design abstract
Plugin makes use of src/utils_message_parser.c utility which takes care of reading new data from file, searching for regular expressions, storing output information.
...
Message items produced by message_parser_read() is postprocessed and notification compliant with collectd is dispatched. All details needed are stored in meta-data structure inside.
Configuration
Main section
<Plugin log_parser>
<Logfile "file01.log">
FirstFullRead false
<Message "msg_1">
…
</Message>
<Message "msg_2">
…
</Message>
…
<Message "msg_NN">
…
</Message>
</Logfile>
<Logfile "file02.log">
…
</Logfile>
…
<Logfile "fileNN.log">
…
</Logfile>
</Plugin>
Configuration can consist of several <Logfile> sections allowing to parse multiple files at once.
FirstFullRead [true | false] – if set to true then reads whole content, otherwise looks only for newly arrived data.
Message section
<Message "PCI_Error">
DefaultPluginInstance "plugin_instance"
DefaultType "type"
DefaultTypeInstance "type_instance"
DefaultSeverity "ok"
<Match "sample_error">
…
</Match>
<Match "error time">
…
</Match>
…
<Match "match_name">
…
</Match>
</Message>
...
These options are used in case we want to have default values. They can be overridden by matches results values.
Match section
<Match "sample_error">
Regex "MSG:.*error occurred"
SubmatchIdx -1
</Match>
<Match "error time">
Regex "(... .. ..:..:..) .* pcieport.*AER"
IsMandatory false
</Match>
…
<Match "match_name">
…
</Match>
Match section used to get severity, type, type instance and plugin instance
<Match "severity">
Severity "warning"
Regex "severity=([wW]arning|warn)"
</Match>
<Match "type">
Type true
Regex "…"
</Match>
<Match "type_instance">
TypeInstance true
Regex "…"
</Match>
<Match "plugin_instance">
PluginInstance true
Regex "…"
</Match>
...
Setting above four options to string will force plugin to ignore matching string and take the given one.
Note: Severity cannot be set to boolean and its value has to be “ok”, “warning” or “failure” as the only provided by collectd.
Key dependencies
Ref | Description | Status |
1 | Collectd message log parser utility | |
2 |
Issues list
None