...
The new VES application is simply a consumer of the collectd topic.
Collectd VES application with the new schema mapping:
Hardcoding of VES Schema fields has been removed from the implementation of the Collectd VES application. Instead you now provide a YAML file which describes how to map collectd fields/value lists to VES fields. The diagram above shows the usage example for the node.yaml configuration.
Producer throughput:
50 million small (100 byte) records as quickly as possible.
...
Test Case | Measurement |
Single Consumer | 940,521 records/sec (89.7 MB/sec) |
3x Consumers | 2,615,968 records/sec (249.5 MB/sec) |
End-to-end Latency | ~2 ms (median) |
Collectd kafka notification support
TODO
References:
0 https://www.cloudera.com/documentation/kafka/1-2-x/topics/kafka.html
...
2 Documentation (http://kafka.apache.org/documentation.html)
Benchmarking Apache Kafka (https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines)
Robust high performance C/C++ library with full protocol support (https://github.com/edenhill/librdkafka)