Anuket Project

Configuration/Setup instructions can be found here:  CPU Utilisation case study Configuration

The idea behind this wiki is to look at the CPU metrics from the baremetal host, the hypervisor and the VM itself to see the if the CPU utilisation metrics reported are consistent across all 3.

 

 

Two tests were conducted. One without isolating cpus from the OS and one with isolating CPUs from the OS on the baremetal host and pinning the vCPUs to those isolated cores. Results for the unisolated tests as expected have some unusual results... 

The CPU plugins for collectd were configured not to report the CPU meterics per state - so just a single metric for utilization

The interval used for the test was for 1 second (this interval was used for the host and guest collectd cpu plugins, the virt plugin wouldn't go down to a 1 second configuration).

1 second was selected in order to get the timing differences between starting the plugin on the host and starting the plugin on the guest down to a lower value... 

Tests were conducted with more/less an idle environment and will be repeated with a busy env.

Function used to convert nanoseconds virt metrics to percent:

 static double cpu_ns_to_percent(unsigned int node_cpus,
                                 unsigned long long cpu_time_old,
                                 unsigned long long cpu_time_new) {
   double percent = 0.0;
   unsigned long long cpu_time_diff = 0;
   double time_diff_sec = CDTIME_T_TO_DOUBLE(plugin_get_interval());
   if (node_cpus != 0 && time_diff_sec != 0 && cpu_time_old != 0) {
     cpu_time_diff = cpu_time_new - cpu_time_old;
     percent = ((double)(100 * cpu_time_diff)) /
               (time_diff_sec * node_cpus * NANOSEC_IN_SEC);
  }

With Isolation (stress test test, 1 second interval)

Host ViewVM ViewHypervisor View

silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

epochvalueepochvalueepochvaluepercent
    14924318551343320000000.00 
149243185678.35149243185662.1951214924318561344110000000.0079.72
1492431857100.00149243185782.0224714924318571345110000000.00100.00
1492431858100.00149243185856.2514924318581346110000000.00100.00
1492431859100.00149243185962.514924318591347110000000.00100.00
1492431860100.00149243186089.5833314924318601348110000000.00100.10
1492431861100.00149243186175.2577314924318611349110000000.0099.90
149243186223.001492431862014924318621349310000000.0020.00
14924318632.021492431863014924318631349330000000.002.00
14924318641.001492431864014924318641349350000000.002.00
14924318651.01149243186519.1919214924318651349390000000.004.00
149243186619.191492431866014924318661349550000000.0016.00
14924318670.99149243186714.1414114924318671349570000000.002.00
149243186882.83149243186810014924318681350420000000.0084.92
1492431869100.00149243186910014924318691351420000000.00100.00
1492431870100.00149243187010014924318701352420000000.00100.00
1492431871100.00149243187110014924318711353420000000.00100.00
1492431872100.00149243187210014924318721354420000000.00100.10
1492431873100.00149243187310014924318731355420000000.0099.90
1492431874100.00149243187410014924318741356420000000.00100.00
1492431875100.00  14924318751357420000000.00100.10
1492431876100.00  14924318761358420000000.0099.90
1492431877100.00  14924318771359420000000.00100.00
149243187833.33  14924318781359730000000.0031.00
14924318790.00  14924318791359730000000.000.00
14924318800.00  14924318801359750000000.002.00
14924318811.00  14924318811359760000000.001.00
14924318821.00  14924318821359770000000.001.00

With Isolation (stress test test, 10 second interval)

Note: stress was ran for 20 seconds in this case

Host View VM View    Hypervisor View
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07  
epochvalueepochvalueepochvaluepercent
    14924338151398840000000.00 
149243382577.56345149243383365.5771214924338251405960000000.0071.44
149243383585.4149243384340.2206614924338351414630000000.0086.70
149243384518.0722914924338531.00200414924338451416330000000.0017.00
14924338551.70511514924338631.20240514924338551416530000000.002.00
14924338651.606426149243387343.714924338651416730000000.002.00
149243387566.4149243388310014924338751423510000000.0067.79
1492433885100149243389359.3406614924338851433500000000.0099.91
149243389537.211641492433903014924338951437100000000.0036.00
14924339051.6064261492433903014924339051437300000000.002.00
14924339151.80360714924339130.90180414924339151437490000000.001.90

With Isolation (idle test)

 

Host View/cpu plugin for cpu 10VM ViewVirt view Virt Other metrics        
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07

 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07



6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

     the physical user/system cpu time consumed by the hypervisor, per-vm 
epochvalueepochvalueepochvaluepercent epochvaluepercentepochusersyst% user % systepochvaluepercent
1488880609.3401488880609.1401488880613.3716590000000.00  14888806133600000000 1488880613.37107400000009590000000.00  148888061331540000000 
1488880610.3401488880610.140               
1488880611.3401488880611.140               
1488880612.3401488880612.140               
1488880613.3401488880613.140.990099               
1488880614.340.9900991488880614.140               
1488880615.3401488880615.140               
1488880616.3401488880616.140               
1488880617.3401488880617.140               
1488880618.340.9900991488880618.140.990099               
1488880619.3401488880619.140               
1488880620.3401488880620.140               
1488880621.3401488880621.140               
1488880622.3401488880622.140               
1488880623.3401488880623.1401488880623.3616610000000.000.20 148888062336300000000.301488880623.36107500000009610000000.000.100.201488880623316000000000.60
1488880624.3401488880624.140               
1488880625.3401488880625.140               
1488880626.3401488880626.140               
1488880627.3411488880627.140               
1488880628.341.9801981488880628.140               
1488880629.3401488880629.140               
1488880630.3401488880630.140.990099               
1488880631.3401488880631.140               
1488880632.3401488880632.140               
1488880633.3401488880633.1401488880633.3616630000000.000.20 148888063336500000000.201488880633.36107600000009630000000.000.100.201488880633316600000000.60
1488880634.3401488880634.140               
1488880635.3401488880635.140               
1488880636.3401488880636.140               
1488880637.3401488880637.140               
1488880638.3411488880638.141               
1488880639.3411488880639.140               
1488880640.3401488880640.140               
1488880641.3401488880641.140               
1488880642.340.9900991488880642.140.990099               
1488880643.3401488880643.141.9801981488880643.3616660000000.000.30 148888064336700000000.201488880643.36107700000009650000000.000.100.201488880643317200000000.60
1488880644.3411488880644.141.980198               
1488880645.3401488880645.141.020408               
1488880646.342.9702971488880646.140               
1488880647.3401488880647.140               
1488880648.3411488880648.141.980198               
1488880649.3401488880649.140               
1488880650.3401488880650.140               
1488880651.340.9900991488880651.140               
1488880652.3401488880652.141               
1488880653.3411488880653.1401488880653.3616730000000.000.70 148888065337200000000.501488880653.36107800000009690000000.000.100.401488880653318700000001.50
1488880654.3411488880654.142.020202               
1488880655.342.0202021488880655.140               
1488880656.340.9900991488880656.140               
1488880657.3401488880657.140               
1488880658.3421488880658.141               
1488880659.3401488880659.140               
1488880660.3401488880660.140               
1488880661.3401488880661.140               
1488880662.3401488880662.140               
1488880663.3401488880663.1401488880663.3616780000000.000.50 148888066337700000000.501488880663.36108000000009720000000.000.200.301488880663319800000001.10
1488880664.3401488880664.140               
1488880665.3401488880665.140               
1488880666.3401488880666.141               
1488880667.341.9801981488880667.140.990099               
1488880668.341.0101011488880668.141.010101               
1488880669.3401488880669.140               
1488880670.3401488880670.140               
1488880671.3401488880671.140               
1488880672.343.9603961488880672.1410.89109               
1488880673.3471488880673.1401488880673.3716910000000.001.30 148888067339400000001.701488880673.37108200000009790000000.000.200.701488880673323200000003.40
1488880674.3401488880674.140               
1488880675.3401488880675.140               
1488880676.3401488880676.140.990099               
1488880677.3401488880677.140               
1488880678.3411488880678.142               
1488880679.3401488880679.140               
1488880680.3401488880680.140               
1488880681.341.9607841488880681.140               
1488880682.3401488880682.140               
1488880683.3401488880683.1401488880683.3616940000000.000.30 148888068339600000000.201488880683.36108300000009810000000.000.100.201488880683323900000000.70
1488880684.3401488880684.140               
1488880685.3401488880685.140               
1488880686.3401488880686.140               
1488880687.3401488880687.140               
1488880688.3411488880688.141               
1488880689.3401488880689.140               
1488880690.3401488880690.140               
1488880691.3401488880691.140.990099               
1488880692.340.9900991488880692.140               
1488880693.3401488880693.1411488880693.3616960000000.000.20 148888069339900000000.301488880693.36108400000009830000000.000.100.201488880693324500000000.60
1488880694.3401488880694.140               
1488880695.3401488880695.140               
1488880696.3401488880696.140               
1488880697.340.9900991488880697.140.990099               
1488880698.3401488880698.140               
1488880699.3401488880699.140               
1488880700.3401488880700.140               
1488880701.3401488880701.140               
1488880702.340.9900991488880702.140               
1488880703.3401488880703.1401488880703.3616970000000.000.10 148888070340300000000.401488880703.36108500000009850000000.000.100.201488880703325100000000.60

Without Isolation (idle test)

 

Host View VM View Virt view  
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07  
epochvalueepochvalueepochvaluepercent
1488877698.942.0202021488877698.641.0101011488877697.98628610000000.00 
1488877699.9414.141411488877699.640   
1488877700.9401488877700.640   
1488877701.9401488877701.640   
1488877702.9401488877702.640   
1488877703.942.9411761488877703.640   
1488877704.9401488877704.640   
1488877705.9401488877705.640   
1488877706.9401488877706.640   
1488877707.9401488877707.6401488877707.97628760000000.001.50
1488877708.9401488877708.640   
1488877709.941.9801981488877709.640   
1488877710.9401488877710.640   
1488877711.9401488877711.640   
1488877712.9401488877712.640   
1488877713.9401488877713.640   
1488877714.9401488877714.640   
1488877715.9401488877715.640   
1488877716.9401488877716.640   
1488877717.9401488877717.6401488877717.96628760000000.000.00
1488877718.9421488877718.641.010101   
1488877719.9411488877719.640   
1488877720.9401488877720.640   
1488877721.940.9900991488877721.640   
1488877722.9401488877722.640   
1488877723.9401488877723.640   
1488877724.941.9801981488877724.640   
1488877725.9401488877725.640   
1488877726.9401488877726.640   
1488877727.9411488877727.6401488877727.96628790000000.000.30
1488877728.9411488877728.641   
1488877729.941.9801981488877729.640   
1488877730.9401488877730.640   
1488877731.9401488877731.640   
1488877732.9401488877732.640   
1488877733.9401488877733.640   
1488877734.9401488877734.640   
1488877735.940.9900991488877735.640   
1488877736.9401488877736.640   
1488877737.940.9900991488877737.6501488877737.96628800000000.000.10
1488877738.948.0808081488877738.642.020202   
1488877739.9421488877739.640   
1488877740.9401488877740.640   
1488877741.9401488877741.640   
1488877742.9401488877742.640   
1488877743.9401488877743.640   
1488877744.9401488877744.640   
1488877745.940.9900991488877745.640   
1488877746.9401488877746.640   
1488877747.9401488877747.6401488877747.96628830000000.000.30
1488877748.9421488877748.640   
1488877749.9401488877749.640   
1488877750.9401488877750.640   
1488877751.9401488877751.640   
1488877752.9401488877752.640   
1488877753.9401488877753.641   
1488877754.9401488877754.640   
1488877755.9401488877755.640   
1488877756.9411488877756.640   
1488877757.940.9900991488877757.6401488877757.97628840000000.000.10
1488877758.941.0101011488877758.641.010101   
1488877759.9421488877759.640   
1488877760.9401488877760.640   
1488877761.9401488877761.640   
1488877762.940.9900991488877762.640   
1488877763.9401488877763.640   
1488877764.9401488877764.640   
1488877765.9401488877765.640   
1488877766.9411488877766.640   
1488877767.9401488877767.6401488877767.97628880000000.000.40
1488877768.9431488877768.641   
1488877769.941.0101011488877769.640   
1488877770.941.9607841488877770.640   
1488877771.9401488877771.640   
1488877772.9411488877772.640   
1488877773.9401488877773.640   
1488877774.940.9900991488877774.640   
1488877775.9411488877775.640   
1488877776.9401488877776.640   
1488877777.9401488877777.6501488877777.96628890000000.000.10
1488877778.943.9603961488877778.641   
1488877779.9411488877779.640   
1488877780.9401488877780.640   
1488877781.9401488877781.640   
1488877782.9401488877782.640   
1488877783.9401488877783.640   
1488877784.940.9900991488877784.640   
1488877785.940.9900991488877785.640   
1488877786.9401488877786.640   
1488877787.9401488877787.6401488877787.96628900000000.000.10
1488877788.9411488877788.640   
1488877789.941.9801981488877789.640   
1488877790.9401488877790.640   
1488877791.9401488877791.640   
1488877792.9401488877792.640   
1488877793.9411488877793.640   
1488877794.9401488877795.640   
1488877795.9411488877796.640   
1488877796.9411488877796.640   
1488877797.9401488877797.6501488877797.96628920000000.000.20

 

 

With core isloation from OS
Host ViewVM ViewVirt viewVirt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07
epochvalueepochvalueepochvaluepercentepochusersyst% user % syst
1488880609.3401488880609.1401488880613.3716590000000.00 1488880613.37107400000009590000000.00  
1488880610.3401488880610.140        
1488880611.3401488880611.140        
1488880612.3401488880612.140        
1488880613.3401488880613.140.990099        
1488880614.340.9900991488880614.140        
1488880615.3401488880615.140        
1488880616.3401488880616.140        
1488880617.3401488880617.140        
1488880618.340.9900991488880618.140.990099        
1488880619.3401488880619.140        
1488880620.3401488880620.140        
1488880621.3401488880621.140        
1488880622.3401488880622.140        
1488880623.3401488880623.1401488880623.3616610000000.000.201488880623.36107500000009610000000.000.100.20
1488880624.3401488880624.140        
1488880625.3401488880625.140        
1488880626.3401488880626.140        
1488880627.3411488880627.140        
1488880628.341.9801981488880628.140        
1488880629.3401488880629.140        
1488880630.3401488880630.140.990099        
1488880631.3401488880631.140        
1488880632.3401488880632.140        
1488880633.3401488880633.1401488880633.3616630000000.000.201488880633.36107600000009630000000.000.100.20
1488880634.3401488880634.140        
1488880635.3401488880635.140        
1488880636.3401488880636.140        
1488880637.3401488880637.140        
1488880638.3411488880638.141        
1488880639.3411488880639.140        
1488880640.3401488880640.140        
1488880641.3401488880641.140        
1488880642.340.9900991488880642.140.990099        
1488880643.3401488880643.141.9801981488880643.3616660000000.000.301488880643.36107700000009650000000.000.100.20
1488880644.3411488880644.141.980198        
1488880645.3401488880645.141.020408        
1488880646.342.9702971488880646.140        
1488880647.3401488880647.140        
1488880648.3411488880648.141.980198        
1488880649.3401488880649.140        
1488880650.3401488880650.140        
1488880651.340.9900991488880651.140        
1488880652.3401488880652.141        
1488880653.3411488880653.1401488880653.3616730000000.000.701488880653.36107800000009690000000.000.100.40
1488880654.3411488880654.142.020202        
1488880655.342.0202021488880655.140        
1488880656.340.9900991488880656.140        
1488880657.3401488880657.140        
1488880658.3421488880658.141        
1488880659.3401488880659.140        
1488880660.3401488880660.140        
1488880661.3401488880661.140        
1488880662.3401488880662.140        
1488880663.3401488880663.1401488880663.3616780000000.000.501488880663.36108000000009720000000.000.200.30
1488880664.3401488880664.140        
1488880665.3401488880665.140        
1488880666.3401488880666.141        
1488880667.341.9801981488880667.140.990099        
1488880668.341.0101011488880668.141.010101        
1488880669.3401488880669.140        
1488880670.3401488880670.140        
1488880671.3401488880671.140        
1488880672.343.9603961488880672.1410.89109        
1488880673.3471488880673.1401488880673.3716910000000.001.301488880673.37108200000009790000000.000.200.70
1488880674.3401488880674.140        
1488880675.3401488880675.140        
1488880676.3401488880676.140.990099        
1488880677.3401488880677.140        
1488880678.3411488880678.142        
1488880679.3401488880679.140        
1488880680.3401488880680.140        
1488880681.341.9607841488880681.140        
1488880682.3401488880682.140        
1488880683.3401488880683.1401488880683.3616940000000.000.301488880683.36108300000009810000000.000.100.20
1488880684.3401488880684.140        
1488880685.3401488880685.140        
1488880686.3401488880686.140        
1488880687.3401488880687.140        
1488880688.3411488880688.141        
1488880689.3401488880689.140        
1488880690.3401488880690.140        
1488880691.3401488880691.140.990099        
1488880692.340.9900991488880692.140        
1488880693.3401488880693.1411488880693.3616960000000.000.201488880693.36108400000009830000000.000.100.20
1488880694.3401488880694.140        
1488880695.3401488880695.140        
1488880696.3401488880696.140        
1488880697.340.9900991488880697.140.990099        
1488880698.3401488880698.140        
1488880699.3401488880699.140        
1488880700.3401488880700.140        
1488880701.3401488880701.140        
1488880702.340.9900991488880702.140        
1488880703.3401488880703.1401488880703.3616970000000.000.101488880703.36108500000009850000000.000.100.20

 

 

Host ViewVM ViewVirt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07
epochvalueepochvalueepochvaluepercentepochvaluepercentepochusersyst% user % systepochvaluepercent
               14924318551343320000000.00 
149243185678.35149243185662.19512           14924318561344110000000.0079.72
1492431857100.00149243185782.02247           14924318571345110000000.00100.00
1492431858100.00149243185856.25           14924318581346110000000.00100.00
1492431859100.00149243185962.5           14924318591347110000000.00100.00
1492431860100.00149243186089.58333           14924318601348110000000.00100.10
1492431861100.00149243186175.25773           14924318611349110000000.0099.90
149243186223.0014924318620           14924318621349310000000.0020.00
14924318632.0214924318630           14924318631349330000000.002.00
14924318641.0014924318640           14924318641349350000000.002.00
14924318651.01149243186519.19192           14924318651349390000000.004.00
149243186619.1914924318660           14924318661349550000000.0016.00
14924318670.99149243186714.14141           14924318671349570000000.002.00
149243186882.831492431868100           14924318681350420000000.0084.92
1492431869100.001492431869100           14924318691351420000000.00100.00
1492431870100.001492431870100           14924318701352420000000.00100.00
1492431871100.001492431871100           14924318711353420000000.00100.00
1492431872100.001492431872100           14924318721354420000000.00100.10
1492431873100.001492431873100           14924318731355420000000.0099.90
1492431874100.001492431874100           14924318741356420000000.00100.00
1492431875100.00             14924318751357420000000.00100.10
1492431876100.00             14924318761358420000000.0099.90
1492431877100.00             14924318771359420000000.00100.00
149243187833.33             14924318781359730000000.0031.00
14924318790.00             14924318791359730000000.000.00
14924318800.00             14924318801359750000000.002.00
14924318811.00             14924318811359760000000.001.00
14924318821.00             14924318821359770000000.001.00
  • No labels

1 Comment

  1. Thanks for running these tests!

    The Isolated case (Idle) and single utilization metric had favorable comparisons between baremetal, hypervisor (KVM?), and VM itself. There were some small deviations, but looking at the average of the ten 1 second measurements and comparing to VM, the results were usually +/- 0.1.

    This is definitely worth repeating at load. I've been digging around for ways to do this, and found https://linux.die.net/man/1/stress