To help work through the schema proposals, we will work through a use case to normalize data from the internal collectd format to the ceilometer format.

Ceilometer use case

Ceilometer provides the possibility to submit samples via the REST API to allow users to send custom samples into this service. The samples that can be sent to Telemetry are not limited to the actual existing meters. There is a possibility to provide data for any new, customer defined counter by filling out all the required fields of the POST request. If the sample corresponds to an existing meter, then the fields like meter-type and meter name should be matched accordingly.

...

and the meter code itself can be found here: https://github.com/openstack/collectd-ceilometer-plugin/blob/master/collectd_ceilometer/meters/base.py. Fields in the ceilometer write plugin at the moment are mapped as follows:

ceilometer meter field	collectd field
counter_name	plugin, type
counter_type	"gauge"-> "gauge", "derive"-> "delta "absolute"-> "cumulative" "counter"-> "cumulative"
counter_unit	unit
counter_volume	value
message_id	-
project_id	obtained from the auth token
recorded_at	-
resource_id	hostname, plugin_instance, type_instance
resource_metadata	meta
source	collectd
timestamp	time
user_id	obtained from credentials to obtain the auth token

...

As you can see, the meter sample aren't mapped to look like the meter samples submitted from an OpenStack service where there is overlap.

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

Looking at a combine Schema. This schema describes the collectd internal types, and the ceilometer fields that reference them

Code Block

language	bash
linenumbers	true

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "collectd Base Schema",
	"description": "A schema that defines the collectd meters and events",
	"definitions": {
	    "value list": {
		    "description": "Statistics in collectd consist of a value list",
			"type": "object",
			"properties": {
				"values" : {
					"type": "array",
					"anyOf" : [ 
						{ "type" : "object", "properties" : { "absolute" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "counter" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "derive" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "gauge" : { "type" : "number" } } }
					]
				},
				"value length" : {
					"description": "The number of values in the data set",
					"type": "number"
				},
				"time" : {
					"description": "Time stamp at which the value was collected",
					"type": "number"
				},
				"Interval" : {
					"description": "interval at which to expect a new value",
					"type": "number"
				},
				"host" : {
					"description": "used to identify the host",
					"type": "string"
				},
				"plugin" : {
					"description": "used to identify the plugin",
					"type": "string"
				},
				"plugin instance" : {
					"description": "used to group a set of values together",
					"type": "string"
				},
				"type" : {
					"description": "unit used to measure a value",
					"type": "string"
				},
				"type instance " : {
					"description": "used to distinguish between values that have an identical type",
					"type": "string"
				},
				"metadata" : {
					"description": "an opaque data structure that enables the passing of additional information about a value list",
					"type": "string"
				}
			}
		},
		"notifications": {
			"description": "Notifications in collectd are generic messages",
			"type": "object",
			"properties": {
				"severity" : {
						"description": "can be one of OKAY, WARNING, and FAILURE",
						"type": "string"
				},
				"time" : {
					"description": "Time stamp at which the event was collected",
					"type": "number"
				},
				"message" : {
					"description": "The notification message",
					"type": "string"
				},
				"host" : {
					"description": "used to identify the host",
					"type": "string"
				},
				"plugin" : {
					"description": "used to identify the plugin",
					"type": "string"
				},
				"plugin instance" : {
					"description": "used to group a set of values together",
					"type": "string"
				},
				"type" : {
					"description": "unit used to measure a value",
					"type": "string"
				},
				"type instance " : {
					"description": "used to distinguish between values that have an identical type",
					"type": "string"
				},
				"metadata" : {
					"description": "an opaque data structure that enables the passing of additional information about a value list",
					"type": "string"
				}
			}
		}
	},
	"counter_name": [ 
			{"$ref": "#/definitions/value list/plugin"}, 
            {"$ref": "#/definitions/value list/plugin instance"}
			
	],
	"counter_unit": [ 
	    {"$ref": "#/definitions/value list/type_instance"},
		{"$ref": "#/definitions/value list/type"}
	],
	"counter_volume": {
		"$ref": "#/definitions/value list/values"
	},
	"resource_id": [
		{"$ref": "#/definitions/value list/host"},
		{"$ref": "#/definitions/value list/plugin"}, 
		{"$ref": "#/definitions/value list/plugin instance"}
	],
	"timestamp": {
		"$ref": "#/definitions/value list/time"
	},
	"resource_metadata": {
        "$ref": "#/definitions/value list/metadata"
    },
	"source": {
	    "type": "string"
	},
	"user_id": {
	    "type": "string"
	}
}

Issues:

Proposal 1 is too high level and doesn't get into the internals of what meters get mapped to what and what units should be associated with the meters/events - in essence it doesn't improve on what's there today for overlapping meters. However is good for new meters.

Proposal 2

Involves 2 schema:

The final message format schema
The mapping schema

The final message format schema for the ceilometer sample is:

Code Block

language	bash

{
    "counter_name": "instance",
    "counter_type": "gauge",
    "counter_unit": "instance",
    "counter_volume": 1.0,
    "message_id": "5460acce-4fd6-480d-ab18-9735ec7b1996",
    "project_id": "35b17138-b364-4e6a-a131-8f3099c5be68",
    "recorded_at": "2015-01-01T12:00:00",
    "resource_id": "bd9431c1-8d69-4ad3-803a-8d4a6b89fd36",
    "resource_metadata": {
        "name1": "value1",
        "name2": "value2"
    },
    "source": "openstack",
    "timestamp": "2015-01-01T12:00:00",
    "user_id": "efd87807-12d2-4b38-9c70-5f5c2ac427ff"
}

The proposed mapping schema is:

Code Block

language	bash
linenumbers	true

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "collectd Base Schema",
    "description": "A schema that defines the collectd meters and events",
    "definitions": {
        "collectd_field": {
            "description": "collectd_fields",
            "type": "object",
            "properties": {
                "anyOf" : {
                    "Interval" : {
                        "description": "interval at which to expect a new value",
                        "type": "number"
                    },
                    "host" : {
                        "description": "used to identify the host",
                        "type": "string"
                    },
                    "message" : {
                        "description": "The notification message",
                        "type": "string"
                    },
                    "metadata" : {
                        "description": "an opaque data structure that enables the passing of additional information about a value list",
                        "type": "string"
                    },
                    "plugin" : {
                        "description": "used to identify the plugin",
                        "type": "string"
                    },
                    "plugin instance" : {
                        "description": "used to group a set of values together",
                        "type": "string"
                    },
                    "severity" : {
                        "description": "can be one of OKAY, WARNING, and FAILURE",
                        "type": "string"
                    },
                    "time" : {
                        "description": "Time stamp at which the value was collected",
                        "type": "number"
                    },
                    "type" : {
                        "description": "unit used to measure a value",
                        "type": "string"
                    },
                    "type instance " : {
                        "description": "used to distinguish between values that have an identical type",
                        "type": "string"
                    },
                    "values" : {
                        "type": "object",
                        "anyOf" : {
                            "absolute" : { "type" : "string" } ,
                            "counter" : { "type" : "string" },
                            "derive" : { "type" : "string" },
                            "gauge" : { "type" : "string" },
							"actual_values" : { "type" : "string" }
                        }
                    },
                    "value length" : {
                        "description": "The number of values in the data set",
                        "type": "number"
                    }
                }
            }
        },
        "mappings": {
            "description": "mapping pair",
            "type": "object",
            "properties": {
                "oneOf": {
                "$ref": "#/definitions/collectd_field",
                "collectd_string": { "type": "string" }
                },
                "new_string": { "type": "string" },
				"new_value": { "type": "string" }
            }
        },
        "mappings_with_conversion": {
            "description": "Mappings from collectd timestamps to other framework timestamps",
            "type": "object",
            "properties": {
                "transform": {
                    "description": "array of mappings from collectd to other frameworks",
                    "type": "array",
                    "items": {  
                        "$ref": "#/definitions/mapping",
                        "conversion": "string"
                    }
                }
            },
            "required": [ "mappings" ]
        }
    },
    "field_mappings": {
        "description": "array of field_mappings",
        "type": "array",
        "items": {
		    "set": {
			    "description": "set of associated of field_mappings",
				"type": "array",
				"anyOf":{
					"$ref": "#/definitions/mappings",
					"$ref1": "#/definitions/mappings_with_conversion"
				}
			}
        }
    }
}

Taking the following ceilometer meter as an example:

Code Block

  - name: 'compute.node.cpu.percent'

    event_type: 'compute.metrics.update'

    type: 'gauge'

    unit: 'percent'

    volume: $.payload.metrics[?(@.name='cpu.percent')].value * 100

    resource_id: $.payload.host + "_" + $.payload.nodename

    timestamp: $.payload.metrics[?(@.name='cpu.percent')].timestamp

    metadata:

      event_type: $.event_type

      host: $.publisher_id

      source: $.payload.metrics[?(@.name='cpu.percent')].source

Using the schema above:

Code Block

language	bash

{
    "field_mappings": [
	    "set": [
		    {
			    "new_string": "counter_name",
				"collectd_field": {
					"plugin": "cpu"
				},
				"conversion": "cpu"
			},
			{
				"new_string": "counter_type",
				"collectd_field

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

collectd json schema

This schema describes the collectd internal types. This schema will be referenced from other mapping schema.

Code Block

language	bash
linenumbers	true

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "collectd Base Schema",
	"description": "A mapping from collectd meters and events to *other* meters and events",
	"type": "object",
	"properties": {
		"value list": {
			"description": "Statistics in collectd consist of a value list",
			"type": "object",
			"properties": {
				"values" : {
					"type": "array",
					"anyOf" : [ 
						{ "type" : "object", "properties" : { "absolute" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "counter" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "derive" : { "type" : "number" } } },
						{ "type" : "object", "properties" : { "gauge" : { "type" : "number" } } }
					]
				},
				"value length" : {
					"description": "The number of values in the data set",
					"type": "number"
				},
				"time" : {
					"description": "Time stamp at which the value was collected",
					"type": "number"
				},
				"Interval" : {
					"description": "interval at which to expect a new value",
					"type": "number"
				},
				"host" : {
					"description": "used to identify the host",
					"type": "string"
				},
				"plugin" : {
					"description": "used to identify the plugin",
					"type": "string"
				},
				"plugin instance" : {
					"description": "used to group a set of values together",
					"type": "string"
				},
				"type" : {
					"description": "unit used to measure a value",
					"type": "string"
				},
				"type instance " : {
					"description": "used to distinguish between values that have an identical type",
					"type": "string"
				},
				"metadata" : {
					"description": "an opaque data structure that enables the passing of additional information about a value list",
					"type": "string"
				}
			}
		},
		"notifications": {
			"description": "Notifications in collectd are generic messages",
			"type": "object",
			"properties": {
				"severity" : {
						"description": "can be one of OKAY, WARNING, and FAILURE",
						"type": "string"
				},
				"time" : {
					"description": "Time stamp at which the event was collected",
					"type": "number"
				},
				"message" : {
					"description": "The notification message",
					"type": "string"
				},
				"host" : {
					"description": "used to identify the host",
					"type": "string"
				},
				"plugin" : {
					"description": "used to identify the plugin",
					"type": "string"
				},
				"plugin instance" : {
					"description": "used to group a set of values together",
					"type": "string"
				},
				"type" : {
					"description": "unit used to measure a value",
					"type": "string"
				},
				"type instance " : {
					"description": "used to distinguish between values that have an identical type",
					"type": "string"
				},
				"metadata" : {
					"description": "an opaque data structure that enables the passing of additional information about a value list",
					"type": "string"
				}
			}
		}
	}
}

collectd to ceilometer mapping example

Issues:

Proposal 1 is too highlevel and doesn't get into the internals of what meters get mapped to what and what units should be associated with the meters/events - in essence it doesn't improve on what's there today for overlapping meters. However is good for new meters.

Proposal 2

collectd simple mapping schema (note this is too clunky so it can be ignored):

Code Block

language	bash
linenumbers	true

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "Collectd Mapping",
	"description": "A mapping from collectd meters and events to *other* meters and events",
	"type": "object",
	"properties": {
		"resource_id": {
			"description": "Mappings from collectd resource_id to other framework resource_id",
			"type": "object",
			"items": {
				"type": "array",
				"items": {
					"collectd_resource_idvalues": "string",{
						"new_resource_idgauge": "stringcumulative"
					}
				}
			},
		"meters": 	{
				"descriptionnew_string": "Mappings from collectd meters to other framework meterscounter_volume",
				"typecollectd_field": "object",
{
					"itemsvalues": {
						"typeactual_values": "values_array",
				"items": {
		}
				}
				"collectd_meterconversion": "stringNone"
			},
			{
				"new_meterstring": "stringtimestamp",
					"newcollectd_meter_unitstring": "stringtimestamp"
				},
			}{
		},
		"eventsnew_string": {"source",
				"descriptionnew_value": "Mappings from collectd events to other framework events",
			"type": "object"},
			"items": {
				"typenew_string": "arrayresource_id",
				"itemscollectd_field": {
					"collectd_eventvalues": "string",{
						"new_eventhost": "stringuuid",
						"collectd_severityplugin instance": "stringplugin_instance",
					"new_severity": "string"}
				}
			}
		}]
	}]
}

Key_words: values_array.

References

http://docs.openstack.org/admin-guide/telemetry-data-collection.html

...

Space shortcuts

Page tree

Versions Compared

Old Version 15

New Version Current

Key

Ceilometer use case

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

Issues:

Proposal 2

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

collectd json schema

collectd to ceilometer mapping example

Issues:

Proposal 2

collectd simple mapping schema (note this is too clunky so it can be ignored):

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 15

New Version Current

Key

Ceilometer use case

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

Issues:

Proposal 2

So where there is overlap in meters in terms of the samples collected by collectd and published to ceilometer, we would like to describe the mapping in a generic way.

Proposal 1

collectd json schema

collectd to ceilometer mapping example

Issues:

Proposal 2

collectd simple mapping schema (note this is too clunky so it can be ignored):