The problem we are trying to resolve is that today we don't have a transparent way of mapping from one telemetry framework data sets to other telemetry framework data sets. There is a multitude of write plugins that normalize data from one set to the other, resulting in clunky framework specific write plugins. We'd like to leverage a common description language to establish a mapping from one data set to another. Then use a write plugin (that doesn't do normalization) to submit the information to the relevant end point.
The description language of choice is JSON.
To visualize the schemas you can copy paste them to http://chris.photobooks.com/json/
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Collectd Mapping",
"description": "A mapping from collectd meters and events to *other* meters and events",
"type": "object",
"properties": {
"value list": {
"description": "Statistics in collectd consist of a value list",
"type": "object",
"properties": {
"values" : {
"type": "array",
"anyOf" : [
{ "type" : "object", "properties" : { "absolute" : { "type" : "number" } } },
{ "type" : "object", "properties" : { "counter" : { "type" : "number" } } },
{ "type" : "object", "properties" : { "derive" : { "type" : "number" } } },
{ "type" : "object", "properties" : { "gauge" : { "type" : "number" } } }
]
},
"value length" : {
"description": "The number of values in the data set",
"type": "number"
},
"time" : {
"description": "Time stamp at which the value was collected",
"type": "number"
},
"Interval" : {
"description": "interval at which to expect a new value",
"type": "number"
},
"host" : {
"description": "used to identify the host",
"type": "string"
},
"plugin" : {
"description": "used to identify the plugin",
"type": "string"
},
"plugin instance" : {
"description": "used to group a set of values together",
"type": "string"
},
"type" : {
"description": "unit used to measure a value",
"type": "string"
},
"type instance " : {
"description": "used to distinguish between values that have an identical type",
"type": "string"
},
"metadata" : {
"description": "an opaque data structure that enables the passing of additional information about a value list",
"type": "string"
}
}
},
"notifications": {
"description": "Notifications in collectd are generic messages",
"type": "object",
"properties": {
"severity" : {
"description": "can be one of OKAY, WARNING, and FAILURE",
"type": "string"
},
"time" : {
"description": "Time stamp at which the event was collected",
"type": "number"
},
"message" : {
"description": "The notification message",
"type": "string"
},
"host" : {
"description": "used to identify the host",
"type": "string"
},
"plugin" : {
"description": "used to identify the plugin",
"type": "string"
},
"plugin instance" : {
"description": "used to group a set of values together",
"type": "string"
},
"type" : {
"description": "unit used to measure a value",
"type": "string"
},
"type instance " : {
"description": "used to distinguish between values that have an identical type",
"type": "string"
},
"metadata" : {
"description": "an opaque data structure that enables the passing of additional information about a value list",
"type": "string"
}
}
}
}
}
|
collectd simple mapping schema:
...
...
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Collectd Mapping",
"description": "A mapping from collectd meters and events to *other* meters and events",
"type": "object",
"properties": {
"resource_id": {
"description": "Mappings from collectd resource_id to other framework resource_id",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_resource_id": "string",
"new_resource_id": "string"
}
}
},
"meters": {
"description": "Mappings from collectd meters to other framework meters",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_meter": "string",
"new_meter": "string",
"new_meter_unit": "string"
}
}
},
"events": {
"description": "Mappings from collectd events to other framework events",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_event": "string",
"new_event": "string",
"collectd_severity": "string",
"new_severity": "string"
}
}
}
}
}
collectd to ceilometer mapping example
...
| ceilometer meter field | collectd field |
|---|---|
counter_name | plugin, type |
counter_type | "gauge"-> "gauge", "derive"-> "delta "absolute"-> "cumulative" "counter"-> "cumulative" |
counter_unit | unit |
counter_volume | value |
message_id | - |
project_id | - |
recorded_at | - |
resource_id | hostname, plugin_instance, type_instance |
resource_metadata | meta |
source | collectd |
timestamp | time |
user_id | - |
using the new schema
collectd simple mapping schema (note this is too clunky so it can be ignored):
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Collectd Mapping",
"description": "A mapping from collectd meters and events to *other* meters and events",
"type": "object",
"properties": {
"resource_id": {
"description": "Mappings from collectd resource_id to other framework resource_id",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_resource_id": "string",
"new_resource_id": "string"
}
}
},
"meters": {
"description": "Mappings from collectd meters to other framework meters",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_meter": "string",
"new_meter": "string",
"new_meter_unit": "string"
}
}
},
"events": {
"description": "Mappings from collectd events to other framework events",
"type": "object",
"items": {
"type": "array",
"items": {
"collectd_event": "string",
"new_event": "string",
"collectd_severity": "string",
"new_severity": "string"
}
}
}
}
} |