The problem we are trying to resolve is that today we don't have a transparent way of mapping from one telemetry framework data sets to other telemetry framework data sets. There is a multitude of write plugins that normalize data from one set to the other, resulting in clunky framework specific write pluginshowever they are implementation specific (per framework/protocol they implement). We'd like to leverage a common description language to establish a methodology for normalizing data sets between frameworks, i.e. a way of describing mapping from one data set to another. Then use a generic write plugin (that doesn't do normalization) to plugins that can parse the appropriate schema, do the real mapping on the fly and submit the information to the relevant end point (unfortunately there still might need to be specific write plugins/utilities for authentication with the relevant end point).

The description language of choice is JSON. 

To visualize the schemas you can copy paste them to http://chris.photobooks.com/json/

...

the schema you can copy paste them to http://chris.photobooks.com/json/

To help work through the schema proposals, we will work through a use case to normalize data from the internal collectd format to the ceilometer format.

Ceilometer use case

Ceilometer provides the possibility to submit samples via the REST API to allow users to send custom samples into this service. If the sample corresponds to an existing meter, then the fields like meter-type and meter name should be matched accordingly.

A standard meter definition looks like:

metric:
  - name: 'meter name'
    event_type: 'event name'
    type: 'type of meter eg: gauge, cumulative or delta'
    unit: 'name of unit eg: MB'
    volume: 'path to a measurable value eg: $.payload.size'
    resource_id: 'path to resource id eg: $.payload.id'
    project_id: 'path to project id eg: $.payload.owner'

Ceilometer old samples schema

...

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "Collectd Mapping",
	"description": "A mapping from collectd meters and events to *other* meters and events",
	"type": "object",
	"properties": {
		"resource_id": {
			"description": "Mappings from collectd resource_id to other framework resource_id",
			"type": "object",
			"items": {
				"type": "array",
				"items": {
					"collectd_resource_id": "string",
					"new_resource_id": "string"
				}
			}
		},
		"meters": {
			"description": "Mappings from collectd meters to other framework meters",
			"type": "object",
			"items": {
				"type": "array",
				"items": {
					"collectd_meter": "string",
					"new_meter": "string",
					"new_meter_unit": "string"
				}
			}
		},
		"events": {
			"description": "Mappings from collectd events to other framework events",
			"type": "object",
			"items": {
				"type": "array",
				"items": {
					"collectd_event": "string",
					"new_event": "string",
					"collectd_severity": "string",
					"new_severity": "string"
				}
			}
		}
	}
}

References

http://docs.openstack.org/admin-guide/telemetry-data-collection.html