Meeting minutes:
Notes from last meeting:
Demo from Barometer:
- Possibly create a demo and record it?
- Demo around containers/VM with metrics
- Float around few ideas, get comments over next calls
- Find a new way to use Barometer, would be good
- Demo ideas:
Agenda:
virt plugin enhancements | Virt plugin should be enhanced to provide the following additional details: | Release G | X | X | BAROMETER-99 - Virt plugin enhancement -- VM names OPEN | |
ovs-dpdk-stats | The ovs-dpdk-stats plugin should include extended dpdk stats per interface | Release G | X | X | BAROMETER-96 - Enable OVS-DPDK PMD Statistics OPEN | |
| ||||||
DMA Functionality | The DMA Local Agent will provide the following features:
| Release G | X | X | BAROMETER-95 - The DMA project will provide additional functional for the Barometer project OPEN | |
DMA Container | Add DMA container build | Release G | X | X | BAROMETER-94 - The DMA project will exist as a container separate from the collectd container OPEN | |
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface | The ovs stats plugin collects the right set of data in case the dpdk interfaces are mapped directly under the ovs bridge. But for the use case where a bonding interface is needed underneath the ovs bridge, ovs stats cannot pull the dpdk interface statistics.
| Release G | X | X | BAROMETER-92 - Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface OPEN |
Updates from last week:
- Changed the Kolla container to include almost all Collectd plugins, taken from latest stable branch
- No new updates about community release of Collectd
- Collectd code reviews:
- Ansible scripts work:
- Milestone 8 – Oct 30th
- Review help:
Meeting minutes:
Demo from Barometer:
- Possibly create a demo and record it?
- Demo around containers/VM with metrics
- Float around few ideas, get comments over next calls
- Find a new way to use Barometer, would be good
- Demo ideas:
Updates:
- Changed the Kolla container to include almost all Collectd plugins, taken from latest stable branch
- No new updates about community release of Collectd
- Collectd code reviews:
- Ansible scripts work:
- Milestone 8 – Oct 30th
- Review help:
Good discussion last week around what service providers really want to use related to telemetry. Do they need to install additional agents for monitoring?
Node Exporter vs. Collectd vs. No additional agents
Not everything is nor will be cloud native -- Providing native poll interfaces may not be possible
Node Exporter is the Prometheus / cloud native agent for collecting data from the hardware.
Collectd is the historical metrics / events collection agent.
Multi-VIM environments
Need a solution across all environments
Need a solution for baremetal, switches, SDN, etc...
Filed exception for MS5
Missed Sept 11th meeting -- Will be covered in next meeting.
Gambia release behind
Milestones
MS6 9/07/18 (PAST)
Test case implementation completed.
Preliminary documentation completed.
Complete documentation compliance table for MS6.
First day that projects may request creation of the stable branch for their project.
MS7 9/21/18
Last day that projects may request creation of the stable branch for their project.
Any project not yet branched will be branched by LF.
Available in Gerrit
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers and feedback!
Add support for 'collectd-master' container
Need reviewers
Release Plan Items
Virt plugin enhancement -- VM names
Milestone 5.9
Work completed, needs to be merged into collectd
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
Good discussion around what service providers really want to use related to telemetry
Node Exporter vs. Collectd vs. No additional agents
Not everything is or will be cloud native
Multi-VIM environments
Filed exception for MS5
In approval process
Milestones
MS6 9/07/18
Test case implementation completed.
Preliminary documentation completed.
Complete documentation compliance table for MS6.
First day that projects may request creation of the stable branch for their project.
MS7 9/21/18
Last day that projects may request creation of the stable branch for their project.
Any project not yet branched will be branched by LF.
Thanks for the heads up on Doc change Emma
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers
Add support for 'collectd-master' container
Need reviewers
Release Plan Items
Virt plugin enhancement -- VM names
Milestone 5.9
Work completed, needs to be merged into collectd
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
DMA functionality
DMA container
Fix functest
Waiting internal review
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers
Additional virt plugin functionality (VM Naming)
PR ready for reviews
Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
Filed exception for MS5
Milestones
MS5 8/24/18
Scenario integration completed
All scenarios have deploy jobs in OPNFV CI
Scenario status page is locked. No new scenarios may be added. However, scenario owners may request the removal of a scenario.
MS6 9/07/18
Test case implementation completed.
Preliminary documentation completed.
Complete documentation compliance table for MS6.
First day that projects may request creation of the stable branch for their project.
MS7 9/21/18
Last day that projects may request creation of the stable branch for their project.
Any project not yet branched will be branched by LF.
Dev Updates
Release Plan Items
Virt plugin enhancement -- VM names
Milestone 5.9
Work completed, needs to be merged into collectd
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
DMA functionality
DMA container
Fix functest
Waiting internal review
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers
Additional virt plugin functionality (VM Naming)
PR ready for reviews
Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
Red Fish
Collectd module completed
Dev Updates
Waiting internal review
Week 33?
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers
PR ready for reviews
Week 33?
Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
Virt plugin enhancement -- VM names
Milestone 5.9
Red Fish
Collectd module completed
OPNFV Testing Harmonization
Self-healing Sig Update
Dev Updates
Waiting internal review
Week 33?
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
Tracked with https://github.com/collectd/collectd/issues/2858
PR submitted --https://github.com/collectd/collectd/pull/2865
This option seems to be the most popular
Looking for reviewers
PR ready for reviews
Week 33?
Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
Virt plugin enhancement -- VM names
Milestone 5.9
Red Fish
Collectd module completed
Upcoming milestones (Update with outcome)
MS6 -- August 24
Test cases implemented, preliminary documentation completed, and stable branch open
Major feature should at least be proposed and partly reviewed at this time.
Functest support should be mostly done (tests upstreamed for collectd-plugins), bug fixes are okay after this date but no new tests.
MS7 -- September 14
Stable branch window close
Need to offer help to collectd to get the next release out (5.9)
Contact maintainers and ask “what can I do to help?”
Also, join #collectd on Freenode if possible
Won’t make Gambia, given that we would need to provide config, docs, validation in Barometer once it is closed; However, we should continue to help.
New RFEs (NEEDINFO)
OPNFV Testing Harmonization
Self-healing Sig Update
Dev Updates
Waiting internal review?
Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface
PR submitted -- https://github.com/collectd/collectd/pull/2880.
Looking for reviewers
PR ready for reviews
Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
Enable OVS-DPDK PMD Statistics
Some related merges upstream in OVS
https://github.com/openvswitch/ovs/commit/971f4b394c6e8480300494787fe919869ff3886c
Virt plugin enhancement -- VM names
Upcoming milestones (Update with outcome)
MS6 -- August 24
Test cases implemented, preliminary documentation completed, and stable branch open
Major feature should at least be proposed and partly reviewed at this time.
Functest support should be mostly done (tests upstreamed for collectd-plugins), bug fixes are okay after this date but no new tests.
MS7 -- September 14
Stable branch window close
Need to offer help to collectd to get the next release out (5.9)
Contact maintainers and ask “what can I do to help?”
Also, join #collectd on Freenode if possible
Won’t make Gambia, given that we would need to provide config, docs, validation in Barometer once it is closed; However, we should continue to help.
Barometer-related paper submissions
KubeCon
Monitoring Tug of War: Matching Cloud Native and Infrastructure Monitoring (Sunku/Aaron)
OpenStack
DMA Team
Barometer on the Edge
OpenSource Summit (LinuxCon)
Barometer has two Prom dates
ONS Europe
DMA -- Tomo/Aaron
OPNFV Testing Harmonization
Self-healing Sig Update
Dev Updates
Write redis plugin enable
Need +1
Upcoming milestones
MS6 -- August 24
Test cases implemented, preliminary documentation completed, and stable branch open
Need to have development “mostly” complete by then
MS7 -- September 14
Stable branch window close
Need feedback on the following list of collect plugins
Feedback will help prioritize work
Need to offer help to collectd to get the next release out (5.9)
Contact maintainers and ask “what can I do to help?”
Also, join #collectd on Freenode if possible
Openstack voting has begun
Dev Update
Thanks for the update Radoslow
Do you think the fix will be a lot of work?
Tentative schedule
Write redis plugin enable
Late comment
Moving forward?
Upcoming milestones
MS6 -- August 24
Test cases implemented, preliminary documentation completed, and stable branch open
Need to have development “mostly” complete by then
MS7 -- September 14
Stable branch window close
Need feedback on the following list of collect plugins
Feedback will help prioritize work
Need to offer help to collectd to get the next release out (5.9)
Contact maintainers and ask “what can I do to help?”
Also, join #collectd on Freenode if possible
Openstack voting has begun
Barometer CI
Al
TST010 -- Using Barometer test code to invoke faults
Lucakz
Virt plugin enhancements by end of week
106 WIP
John, Prometheus WIP to be removed soon
Test lab /hardware for Nokia / RedFish
UNH
Parker Berberian <pberberian@iol.unh.edu>
Lincoln Lavoie <lylavoie@iol.unh.edu>
Sunku to send contacts for Intel
Pharos Lab
Non OpenStack installed hardware
Requestor to send e-mail
Prometheus / collectd native times stamp
Development
Apologies on the confusion on who would do the work.
Thanks for the review
Additional clarification added / needed
INTEL_PMU loading issue
Rename -- Need to check with APEX
Did new contributors get added?
Yes!
Plugfest (Al Morton)
ETSI NFVI Testing group might be interested in using Barometer
Might use Barometer to collect metrics during API compliance testing
Would also like to use fault triggering code from one of the Barometer demos
Bring an interface down / up, etc…
DMA team (Tomo) to present a demo of Local Agent collecting OpenStack metadata.
One of the features of the DMA project is to collect virtual infrastructure information from OpenStack and make it available to collectd so that metadata can be added to metrics / events.
Tomo gave a demo of Local Agent using OpenStack API to gather metadata information for use by Collectd.
Development
VM -- Names
Proposal
Variable defined
Installer Integration -- No_sdn - 6/15/18
Nomination of new contributors
Kamil: kamilx.wiatrowski@intel.com
Lukasz: lukaszx.cybura@intel.com
Sunku: sunku.ranganath@intel.com
Kevin: kevin.laatz@intel.com
Self Healing Sig
Use case templates about DMA project and local healing
What about 5 9’s
White paper on Barometer’s role in self-healing / availability
Originally discussed in Sydney
What should be included and what people care about
NEC, Verizon, etc…
Focus is HA.
Go over monitoring paper
Go with Confluence Wiki on Barometer home page
New dedicated resource for collectd.
Ryan McCabe (Red Hat)
Barometer OS Summit
Feedback was good
Questions
Starting to use collectd, Barometer
What different plugins available? Turns out most are already in Barometer
Prometheus was mentioned -- Mostly about host to get data into Prometheus from collectd. Not necessarily replacing collectd.
Virt Plugin
3 missing variables.
Added to JIRA
Ovs-stats
Extended DPDK stats
JC to check
PMU Plugin Wiki updates
Al Morton
List of VES metrics meeting this week
Going to agree with Al that IPMI are important. What will we do with those on ETSI side?
Give feedback to Al on VES.
NFVI Monitoring White Paper
Create google doc
Start outline
Should cover why to monitor specific metrics / conditions, not just these are the things that are available to monitor.
Sunku --
Had a chat Sando - Paul
Call at end of week
Self healing Sig
List of projects (OPNFV, etc…, )
Sunku to e-mail...
(MS3) Installer integration with OpenStack
3.0 Integration / nosdn deploy / run HC 6/15
3.1 deploy SDN / pass HC 6/29
PMU Plugin
Evaluating options
JIRA item to track virt plugin metric additions (gaps with ceilometer) -- https://jira.opnfv.org/browse/BAROMETER-98
JIRA item to track virt plugin VM name enhancements -- https://jira.opnfv.org/browse/BAROMETER-99
Ceilometer / Collectd parity
Can Ceilometer be replaced with collectd?
NFVI (OpenStack Services) monitoring
NFVI Monitoring Whitepaper -- https://jira.opnfv.org/browse/BAROMETER-97
How to collaborate? Google docs?
Gambia release planning
Gambia release will concentrate on maintenance and support of plugins delivered in Fraser release.
Need to check on status of ovs-dpdk-stats, ovs-dpdk-events plugins
One or two features might be added -- TBD
Need code review of container build changes! Gerrit # -- 53805
Standards
VES
Prometheus
Gambia Release -- traditional release
Need scenarios and deliverables defined by May 11
Gambia Additions
More information added to virt plugin?
Cpu pinning
NUMA node characteristics
What to do about (mostly) static information?
Additional plugins?
May need modification to RAS
MCElog might be incomplete
Looking at out of band platform hardware
Additional containers?
Ceilometer replacement?
How to get metadata?
Need code review of container build changes! Gerrit # -- 53805
Standards
VES
Prometheus
Interest in using Prometheus as polling engine / TSDB
Prometheus_collectd_exporter vs. Prometheus_node_exporter
Gambia Release
Traditional, XCI, both?
Not clear on XCI requirements.
When do the tests need to be in place for XCI?
What are tentative development goals for the Gambia release?
RPMs / Container build
New pull request - https://gerrit.opnfv.org/gerrit/#/c/53805/9
Standards
All the info we’re collecting will end up in analytic engines
Analytics processing tasks need info all the way down to the agent layer
Need to make sure there’s a measurement timestamp to provide context
Does test008 spec have the flexibility to talk about measurement of network interface using a particular VLAN tag?
Yes, scope parameter makes that possible
Scope must be conveyed with the measurements
VES
6.0 specs on the way (next week)
Multi-VM reqs
Compass support -- not for Gambia or Fraser?
Barometer will be supported in Gambia
HA scenario issues with build
A problem with OOO?
Fails every third build or so.
Will be looked at soon (next week)
Edge Cloud
Requirements
Categories
Use cases
Latency
Use cases where latency is critical
Example feedback loop?
What does lightweight mean in the context of edge telemetry?
Spoke with DMA project team. They are ok with working in a separate container. They will swing back in a few weeks with a demo.
Standards with Al Morton
No updates
VES with Bryan Sullivan
Most work done by Goku
Bryan will update VES for latest release to make sure it works with everything
Plan is place to move items forward in Barometer
RPMs with Leif
Making progress
Func tests
Will not support Compass this cycle
Collectd pull requests
Used to be tracked on wiki
We need to track open pull requests and put them on Barometer development update page: https://wiki.opnfv.org/display/fastpath/Barometer+Development+Updates
Release
Gambia path or something else?
Which track do we follow?
Continuous delivery
Flexibility with milestones, since you establish them yourself?
Bryan doesn’t think we should do traditional path -- not enough resources
FCI could be a good thing if it marries VES and Barometer
Edge Cloud
What are the requirements?
Need categories
Always about latency for faults and metrics
Framework and edge computing use cases would be good
Is VES a good fit?
Good for cloud-native
What is a cloud? What value does opensource ecosystem get from:
Common telemetry date model
Consistent set of mechanisms by which data delivered to collector
How lightweight?
Less than 1% CPU, and 5% memory
Two VES encodings: C and Java
Goku: tokenized VES data structure
Overhead of data and complexity of stack
Go from binary XML to tokenized JSON
Current implementation is not lightweight but is “easy”
Use gzip, as it is natively available to HTTP stack (works out of box)
DMA Project discussion
Positive feedback on DMA proposal. Want to move forward.
Work should be done in a separate container.
Standards
Al has been reviewing Alok’s VES proposals. Found some VMWare specific items. Vendor-specific provisions are discouraged. Al to talk with Alok ;).
VES is starting to incorporate some TOSCA information.
Collectd
Emma suggested that all collectd PRs be from the same fork, Barometer, in order to make collectd aware of who the reviews are for.
Also, should have more Red Hat people reviewing collectd. Need to talk with local Red Hatters.
VES
Bryan returned. Wants to keep the VES agent development moving forward. Did not know about DDS messaging stuff in ONAP. However, see’s that Kafka is not set in stone. What to do at the edge.
General
Next week I will be out. Andrew Bays will run meeting.
DMA Project
Presentation given by Tomofumi Hayashi. Slides available on the wiki.
Major takeaways…
Run time reconfiguration of collectd. There were some questions as to whether this should be a feature of something like an Ansible playbook.
Topology annotation. Questions about security and what information would be included. It was stated that only infrastructure information would be annotated.
Data reduction and analysis at the edge.
DMA project should be an add-on. Might need to be able to “unwind” project in the future.
OPNFV Bottlenecks project using Barometer container
Development
Thanks to Emma for updating the docs and helping track down Funtest issues
Still seeing an error with Functest… seems to be related to Functest, not Barometer. Emma tracked down the internal Barometer issues!
Barometer container -> multistage build from RPMs progressing slowly..
ONS recap
Made MS6...
Next upcoming milestone, MS7, April 6th -- last day to request a stable branch. Can request one anytime until then.
After than MS8, April 16, completion of testing
Current Fraser status
test_add_remove_volume
test_attach_volume_nowait
test_detach_volume_nowait
….
Perhaps it was a system / connectivity failure?
Emma good catch on Hugepages failing fix
Gnocchi failures
Results not appearing quickly enough due to Gnocchi archive policy
Change default policy to High
AODH failures
Need to modify the tests to perform specific failure action
REDFISH Plugin
Purpose -- node or rack?
Design review
Standards Update
VES Updates
Progress on ETSI?
VES Mobile edge platforms?
Is there a cadvisor plugin for collectd?
C, Java VES libraries
Microservice heatbeat
ONAP flow
Vendor VNF
Closed loop control
Services goes into VNF
Any missing heatheats
At the FM layer
Functest debug and then power loss ;(
Milestone 5 complete
Scenario deployment passed
Functest failing
MS6 03/16/18
Test case implementation completed.
Preliminary documentation completed.
Complete documentation compliance table for MS6.
First day that projects may request creation of the stable branch for their project.
APEX
Functests failing
Virtual deploy -- RDT, Hugepages… Should we skip during virtual deploy?
APEX deployment help -- Aaron has a test setup running and can answer questions.
Collectd
PRs
pmu plugin / pmu core -- Reviewer comments -- 03/06/2018
ovs link status -- Is this still relevant?
AMQP1 -- Needs review
Connectivity Event -- Reviews in progress
Process Event -- Reviews in progress
System Event -- Reviews in progress
DMA project would like to have another meeting (possibly at ONS)
Did you get your feedback to Alok? I know I didn’t yet :(.
Want to get comments by the 15th
Produce a spec update by end of March
Have a weekly update
VES Update
VES Project into ETSI
Parts of the project, metric / event definitions, transport
Pair things down to what’s important
10’s of collectors at present
VES mobile edge platforms
CAdvisor plugin for collectd?
Maybe try in the Barometer container?
Milestone 5 approaching
Thanks for the help on Apex and Barometer
Tracked down issue with deploy
Standards Update
VES Update
Barometer containers to be part of RDO release
Team to work on building Kolla containers
AMQP1.0 container being built
Collectd on Cumulus
Open Compute / Networking
Standards Updates?
VES Updates?
Development
Apex scenarios are currently failing
Ansible script reviews in progress
MS5 approaching
Barometer next?
Should Barometer Graduate? (https://www.opnfv.org/software/technical-project-governance/project-lifecycle)
Are we a part of enough scenarios (another installer)?
Would be a good recognition of the work done by Barometer
Have had good recognition at shows (OpenStack Summit, OPNFV, etc)
Bryan would be supportive of applying for graduation
No definitive answer, but positive reception
Standards Update (Al)
Meeting with IFA working group completed work on all measurement content. Al provided an edit / rewrite of the document and resubmitted.
Discussion of GAPs in Baromter
VES Update
Bryan working on Barometer Kubernetes scenario. Should be ready soon. Barometer as DaemonSet
Conversion of VES to gRPC in the testing phase.
Development Update
Ansible scripts submitted to Gerrit
Clarification that plugins not in collectd yet should still make it into Barometer / Fraser
(MS5) Scenario integration and feature freeze -- 03/02/18
Good discussion on “What’s next for Barometer”
Scope expansion -- Should Barometer move beyond metrics / events to policy
Scope continuation into container space --
OPNFV Barometer people going to ONS in March
Aaron
Al
Bryan
??
· Filed exception for MS5
o Sept 18th meeting for approval