October 16, 2018
Meeting Minutes:
October 09, 2018
Meeting Minutes:
- collectd documentation
- DPDK
- Is it considered open source?
- Is there anything specific to DPDK or are they just mirroring standard parameters?
- Should add DPDK specific metrics!
- DPDK telemetry
- DPDKSTATs replacement
- Maybe have a small presentation at a later date for input.
- REDFISH?
- MCELOG
- RAS – x86 specific
- RDT – x86 specific
Agenda:
- Release status
- Tour of Prometheus
October 02, 2018
Meeting minutes:
- Need a JIRA item for Ansible scripts
- Remaining documentation for Ansible scripts
- 1,2 issues – scaling, etc...
- One command install
- Moved to Centos 5.7 / Ubuntu 18.04
- Show internal – One click install
- Demo
- Video – 5 minutes to monitoring your NFVi
- Prometheus waiting on Ansible
- Participation in https://openmetrics.io/
Notes from last meeting:
Demo from Barometer:
- Possibly create a demo and record it?
- Demo around containers/VM with metrics
- Float around few ideas, get comments over next calls
- Find a new way to use Barometer, would be good
- Demo ideas:
- Barometer for system tests/debugging.
- Look at network interfaces, OVS stats, etc., to track whats happening in an openstack deployment
- Need to know if there are any openstack deployments already available in opnfv pods
- Have a real time demo to show graphana dashboards in opnfv pods
Agenda:
- Status of Release
- What do we want in the release?
| virt plugin enhancements | Virt plugin should be enhanced to provide the following additional details: | Release G | X | X |
|  BAROMETER-99 - Virt plugin enhancement -- VM names OPEN
|
| ovs-dpdk-stats | The ovs-dpdk-stats plugin should include extended dpdk stats per interface | Release G | X | X |
| BAROMETER-96 - Enable OVS-DPDK PMD Statistics OPEN |
ovs-dpdk-events link status | The ovs-dpdk-events plugin should indicate link status change events | Release G | X | X |
|  BAROMETER-109 - ovs-events link status OPEN
|
| DMA Functionality | The DMA Local Agent will provide the following features: - Local processing of metrics – Allows metrics to be sampled at a higher rate than storage rate for detection of short-lived conditions.
- Runtime configuration of collectd – Collectd configuration can be managed remotely with the Local Agent providing the writing of collectd configuration and collectd restart.
- Simple OpenStack metadata discovery – Provide runtime information related to the mapping between virtual and physical elements. The metadata can be attached to metric sent by collectd.
| Release G | X | X |
| BAROMETER-95 - The DMA project will provide additional functional for the Barometer project OPEN |
| DMA Container | Add DMA container build | Release G | X | X |
| BAROMETER-94 - The DMA project will exist as a container separate from the collectd container OPEN |
| Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface | The ovs stats plugin collects the right set of data in case the dpdk interfaces are mapped directly under the ovs bridge. But for the use case where a bonding interface is needed underneath the ovs bridge, ovs stats cannot pull the dpdk interface statistics. | Release G | X | X | | BAROMETER-92 - Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface OPEN |
- Original plan stalled by no release from Collectd.
- virt plugin enhancements – Complete waiting on release
- ovs-dpdk-stats –
- DMA Functionality - In progress
- DMA Container - In progress
- Collectd ovs_stats plugin does not collect dpdk interface statistics if under bond interface – completed waiting on release
- Participation in https://openmetrics.io/
Updates from last week:
- Changed the Kolla container to include almost all Collectd plugins, taken from latest stable branch
- Plugins are still taken from RPM build, relying on release version of Collectd
- Not included plugins would gaming plugins, etc.
- This is a great progress
- No new updates about community release of Collectd
- Intel team will prioritize backporting once Ansible script work is done
- Collectd code reviews:
- Lot of PRs, will take some time
- The DPDK patch got merged, need to backport to 5.8
- Sunku will ask the patch author to backport
- Ansible scripts work:
- Testing/validation on Ubuntu 16.04, 18.04 & Centos 7.5
- Centos 7.3 had docker issues, related to docker missing packages
- Latest docker Images is based on 7.5.3
- Centos 7.5 images (qcow2) have different naming scheme, very similar to Ubuntu 18.04
- There shouldn’t be any regressions, upgrading from 7.3 to 7.5.
- We can look in to if we need to on 7.3
- Require your help to review as patches are upstreamed
- Milestone 8 – Oct 30th
- Its not being pushed back
- Review help:
- Have 4 to 5 reviews pending in Barometer gerrit
- Require reviews
- Prometheus and Barometer
- Should Barometer participate in the standardization effort under way in Prometheus. (Yes!)
- Driven by Prometheus for data format
- Mentioned at Fosdem & Prometheus conference
- Standard is named as open metrics, submitted to IEEE
- Gone quite far, need 3 implementations of metrics format for this standard
- Google is doing one implementation, 2nd one by Prometheus community
- Trying to standardize data exchange format
- While Prometheus only supports metrics, standards also supports logs.
- Value of all this info is the correlation part
- Matthias would send more info.
- Need to make sure it wont impact our implementations of metrics in hardware/infrastructure level
- The community is not generally attuned to lower layers.
- Bring Barometer’s experience in monitoring the infrastructure.
- End result could be too PaaSy ;)
09/25/2018
Meeting minutes:
Demo from Barometer:
- Possibly create a demo and record it?
- Demo around containers/VM with metrics
- Float around few ideas, get comments over next calls
- Find a new way to use Barometer, would be good
- Demo ideas:
- Barometer for system tests/debugging.
- Look at network interfaces, OVS stats, etc., to track whats happening in an openstack deployment
- Need to know if there are any openstack deployments already available in opnfv pods
- Have a real time demo to show graphana dashboards in opnfv pods
Updates:
- Changed the Kolla container to include almost all Collectd plugins, taken from latest stable branch
- Plugins are still taken from RPM build, relying on release version of Collectd
- Not included plugins would gaming plugins, etc.
- This is a great progress
- No new updates about community release of Collectd
- Intel team will prioritize backporting once Ansible script work is done
- Collectd code reviews:
- Lot of PRs, will take some time
- The DPDK patch got merged, need to backport to 5.8
- Sunku will ask the patch author to backport
- Ansible scripts work:
- Testing/validation on Ubuntu 16.04, 18.04 & Centos 7.5
- Centos 7.3 had docker issues, related to docker missing packages
- Latest docker Images is based on 7.5.3
- Centos 7.5 images (qcow2) have different naming scheme, very similar to Ubuntu 18.04
- There shouldn’t be any regressions, upgrading from 7.3 to 7.5.
- We can look in to if we need to on 7.3
- Require your help to review as patches are upstreamed
- Milestone 8 – Oct 30th
- Its not being pushed back
- Review help:
- Have 4 to 5 reviews pending in Barometer gerrit
- Require reviews
- Gambia release behind
- Please have check-ins in good shape before submission
- Milestones
Release Plan Items - https://jira.opnfv.org/browse/BAROMETER-99
- https://jira.opnfv.org/browse/BAROMETER-96
- Ansible scripts update
- https://gerrit.opnfv.org/gerrit/#/c/61575/5
- Need to have the experimental Barometer container
- Need a place to send interested parties so they can try out newest features
- Early feedback is key
- Showing the host naming in virt plugin in experimental container would be good.
- If we have hardware, we can have virtualized RDO installed quickly in barometer opnfv pod
- A simple 1 controller/1 compute node running barometer container each, also on the pod host itself would be good to show metrics collection across all layers.
- Listing the steps in barometer wiki would be helpful
- Possibly have a packstack installation on a single machine.
- Prometheus and Barometer
- Should Barometer participate in the standardization effort under way in Prometheus. (Yes!)
- Driven by Prometheus for data format
- Mentioned at Fosdem & Prometheus conference
- Standard is named as open metrics, submitted to IEEE
- Gone quite far, need 3 implementations of metrics format for this standard
- Google is doing one implementation, 2nd one by Prometheus community
- Trying to standardize data exchange format
- While Prometheus only supports metrics, standards also supports logs.
- Value of all this info is the correlation part
- Matthias would send more info.
- Need to make sure it wont impact our implementations of metrics in hardware/infrastructure level
- The community is not generally attuned to lower layers.
- Bring Barometer’s experience in monitoring the infrastructure.
- End result could be too PaaSy ;)
09/18/2018
- Filed exception for MS5
- Sept 18th meeting for approval
- Gambia release behind
- Milestones
- MS6 9/07/18 (PAST)
- Test case implementation completed.
- Preliminary documentation completed.
- Complete documentation compliance table for MS6.
- First day that projects may request creation of the stable branch for their project.
- Exception request is approved
- MS7 9/21/18
- Last day that projects may request creation of the stable branch for their project.
- Any project not yet branched will be branched by LF.
- Exception request is approved
- We could cherrypick items and create a stable branch.
- Agree on "definition of done" →
- Have the Collectd PRs tracked through Barometer showcased thru experimental containers
- Experimental container are WIP
- Need to follow this for individual items in Jira too, update DoD value in every Jira item
- https://jira.opnfv.org/browse/BAROMETER-106
- https://jira.opnfv.org/browse/BAROMETER-92
- https://gerrit.opnfv.org/gerrit/#/c/61591/1
- Add support for 'collectd-master' container
- Need reviewers
- The CI integration part is WIP, Radek/Emma are following up
- Review: Ansible scripts
- Release Plan Items
- Introductions: Vinay Kumar
- Joined with RedHat, Metrics & Monitoring.
- Looking in to Gnocchi, AODH, Ceilometer, etc. Interested in integrating Collectd with OpenStack.
- Necessity of gRPC interface for collectd.
- Clover project and cloud native WG have proposal to integrate most of OPNFV projects with gRPC interface.
- GRPC is hot, good thing is binary format is very compact, while http is very verbose using up a lot of bandwidth
- ONAP talks a lot on using GRPC instead of kafka
- No downside on having GRPC
- Good to float the idea within Collectd community
- Florian & others added in write_prometheus, to enable cloud native
- It creates a scrape target
- Superior from Collectd exporter from Prometheus itself
- Real key is to provide correlation between app metrics and infrastructure metrics. Have them be in common data definitions & timestamps.
- Collectd timestamps are when metrics are generated
- Prometheus timestamp is when it gets scraped
- There is a disconnect
- We need to keep reminding CN folks about correlation on time/locality, between virtual/physical
- Need to check for direct integration between Jaeger and Collectd
- A list of metrics/capabilities through node_exporter in Prometheus might be helpful.
- Strengths, missing metrics?
- Get stuff in to node_exporter?
- CN folks might know as they are from different community than OpenStack, as there are different tools, etc.
- Aaron to help us out here to get started!
- Al to contribute here.
- Importance of having “hostname” as uniform reporting instead of IPv4 address:
- Memcached plugin is special -> provide both hostname & IPv4 address. Opened a PR in PuppetCollectd to fix proper reporting.
- It would be good to have additional text along with hostname, like rack name, tag datacenters
- Need to work with plugin authors
- Require following PR Reviews contributed by Intel:
September 11, 2018
September 04, 2018
08/28/2018
Filed exception for MS5
Milestones
MS5 8/24/18
Scenario integration completed
All scenarios have deploy jobs in OPNFV CI
Scenario status page is locked. No new scenarios may be added. However, scenario owners may request the removal of a scenario.
MS6 9/07/18
Test case implementation completed.
Preliminary documentation completed.
Complete documentation compliance table for MS6.
First day that projects may request creation of the stable branch for their project.
MS7 9/21/18
Dev Updates
08/21/2018
Dev Updates
Self-healing Sig Update
08/14/2018
08/07/2018
07/30/2018
· Dev Updates
[ELF] Patches created; waiting internal approval to upstream
§ Write redis plugin enable
§ Need +1
o Upcoming milestones
§ MS6 -- August 24
· Test cases implemented, preliminary documentation completed, and stable branch open
· Need to have development “mostly” complete by then
[ELF] Major feature should at least be proposed and partly reviewed at this time.
Functest support should be mostly done (tests upstreamed for collectd-plugins), bug fixes are okay after this date but no new tests.
§ MS7 -- September 14
· Stable branch window close
· Need feedback on the following list of collect plugins
o Feedback will help prioritize work
· Need to offer help to collectd to get the next release out (5.9)
o Contact maintainers and ask “what can I do to help?”
o Also, join #collectd on Freenode if possible
[ELF] Not likely to be available for Gambia, given that we would need to provide config, docs, validation in Barometer once it is closed; However, we should continue to help.
· Openstack voting has begun
[ELF] There was a very short voting period this time around, and it’s already closed
AOB:
- Collectd container split: Need to check whether the separate container build jobs (master and stable) are configured correctly; so we can start adding config and build support for newer features (Emma/Kamil)
07/30/2018
Dev Updates
Need feedback on the following list of collect plugins
Need to offer help to collectd to get the next release out (5.9)
Openstack voting has begun
07/24/2018
Dev Update
Need feedback on the following list of collect plugins
Need to offer help to collectd to get the next release out (5.9)
Openstack voting has begun
07/03/2018
06/12/2018
06/05/2018
DMA team (Tomo) to present a demo of Local Agent collecting OpenStack metadata.
One of the features of the DMA project is to collect virtual infrastructure information from OpenStack and make it available to collectd so that metadata can be added to metrics / events.
Tomo gave a demo of Local Agent using OpenStack API to gather metadata information for use by Collectd.
Development
Nomination of new contributors
Self Healing Sig
Use case templates about DMA project and local healing
What about 5 9’s
White paper on Barometer’s role in self-healing / availability
Originally discussed in Sydney
What should be included and what people care about
NEC, Verizon, etc…
Focus is HA.
Go over monitoring paper
New dedicated resource for collectd.
05/29/2018
05/22/2018
05/08/2018 --
05/01/2018 --
04/24/2018 --
04/17/2018 --
04/10/18 --
DMA Project discussion
Standards
Al has been reviewing Alok’s VES proposals. Found some VMWare specific items. Vendor-specific provisions are discouraged. Al to talk with Alok ;).
VES is starting to incorporate some TOSCA information.
Collectd
Emma suggested that all collectd PRs be from the same fork, Barometer, in order to make collectd aware of who the reviews are for.
Also, should have more Red Hat people reviewing collectd. Need to talk with local Red Hatters.
VES
General
04/03/18 --
DMA Project
Development
Thanks to Emma for updating the docs and helping track down Funtest issues
Still seeing an error with Functest… seems to be related to Functest, not Barometer. Emma tracked down the internal Barometer issues!
Barometer container -> multistage build from RPMs progressing slowly..
ONS recap
03/20/18 --
Made MS6...
Next upcoming milestone, MS7, April 6th -- last day to request a stable branch. Can request one anytime until then.
After than MS8, April 16, completion of testing
Current Fraser status
REDFISH Plugin
Purpose -- node or rack?
Design review
Standards Update
VES Updates
03/13/18 --
03/06/18 --
Milestone 5 complete
MS6 03/16/18
Test case implementation completed.
Preliminary documentation completed.
First day that projects may request creation of the stable branch for their project.
APEX
Functests failing
Virtual deploy -- RDT, Hugepages… Should we skip during virtual deploy?
APEX deployment help -- Aaron has a test setup running and can answer questions.
Collectd
DMA project would like to have another meeting (possibly at ONS)
Did you get your feedback to Alok? I know I didn’t yet :(.
VES Update
VES Project into ETSI
Parts of the project, metric / event definitions, transport
Pair things down to what’s important
10’s of collectors at present
VES mobile edge platforms
02/27/18 --
02/20/18 --
Collectd on Cumulus
Standards Updates?
VES Updates?
Development
Barometer next?
02/13/18 --
Should Barometer Graduate? (https://www.opnfv.org/software/technical-project-governance/project-lifecycle)
Are we a part of enough scenarios (another installer)?
Would be a good recognition of the work done by Barometer
Have had good recognition at shows (OpenStack Summit, OPNFV, etc)
Bryan would be supportive of applying for graduation
No definitive answer, but positive reception
Standards Update (Al)
VES Update
Development Update
(MS5) Scenario integration and feature freeze -- 03/02/18
Good discussion on “What’s next for Barometer”
OPNFV Barometer people going to ONS in March
· Filed exception for MS5
o Sept 18th meeting for approval