Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Configuration/Setup instructions can be found here:  CPU Utilisation case study Configuration

The idea behind this wiki is to look at the CPU metrics from the baremetal host, the hypervisor and the VM itself to see the if the CPU utilisation metrics reported are consistent across all 3.

...

Two tests were conducted. One without isolating cpus from the OS and one with isolating CPUs from the OS on the baremetal host and pinning the vCPUs to those isolated cores. Results for the unisolated tests as expected have some unusual results.

With Isolation

Core 10 and 11 were isolated from the OS using the isolcpus argument to grub command line and using virsh to Pin the vCPUs to those cores:

$ virsh vcpupin test 0 10
$ virsh vcpupin test 1 11
 
$ virsh vcpupin test
VCPU: CPU Affinity
----------------------------------
   0: 10
   1: 11

 

.. 

The CPU plugins for collectd were configured not to report the CPU meterics per state - so just a single metric for utilization

The interval used for the test was for 1 second (this interval was used for the host and guest collectd cpu plugins, the virt plugin wouldn't go down to a 1 second configuration).

1 second was selected in order to get the timing differences between starting the plugin on the host and starting the plugin on the guest down to a lower value... 

Tests were conducted with more/less an idle environment and will be repeated with a busy env.

Function used to convert nanoseconds virt metrics to percent:

 static double cpu_ns_to_percent(unsigned int node_cpus,
                                 unsigned long long cpu_time_old,
                                 unsigned long long cpu_time_new) {
   double percent = 0.0;
   unsigned long long cpu_time_diff = 0;
   double time_diff_sec = CDTIME_T_TO_DOUBLE(plugin_get_interval());
   if (node_cpus != 0 && time_diff_sec != 0 && cpu_time_old != 0) {
     cpu_time_diff = cpu_time_new - cpu_time_old;
     percent = ((double)(100 * cpu_time_diff)) /
               (time_diff_sec * node_cpus * NANOSEC_IN_SEC);
  }

With Isolation (stress test test, 1 second interval)

Host ViewVM ViewHypervisor View

silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

epochvalueepochvalueepochvaluepercent
    14924318551343320000000.00 
149243185678.35149243185662.1951214924318561344110000000.0079.72
1492431857100.00149243185782.0224714924318571345110000000.00100.00
1492431858100.00149243185856.2514924318581346110000000.00100.00
1492431859100.00149243185962.514924318591347110000000.00100.00
1492431860100.00149243186089.5833314924318601348110000000.00100.10
1492431861100.00149243186175.2577314924318611349110000000.0099.90
149243186223.001492431862014924318621349310000000.0020.00
14924318632.021492431863014924318631349330000000.002.00
14924318641.001492431864014924318641349350000000.002.00
14924318651.01149243186519.1919214924318651349390000000.004.00
149243186619.191492431866014924318661349550000000.0016.00
14924318670.99149243186714.1414114924318671349570000000.002.00
149243186882.83149243186810014924318681350420000000.0084.92
1492431869100.00149243186910014924318691351420000000.00100.00
1492431870100.00149243187010014924318701352420000000.00100.00
1492431871100.00149243187110014924318711353420000000.00100.00
1492431872100.00149243187210014924318721354420000000.00100.10
1492431873100.00149243187310014924318731355420000000.0099.90
1492431874100.00149243187410014924318741356420000000.00100.00
1492431875100.00  14924318751357420000000.00100.10
1492431876100.00  14924318761358420000000.0099.90
1492431877100.00  14924318771359420000000.00100.00
149243187833.33  14924318781359730000000.0031.00
14924318790.00  14924318791359730000000.000.00
14924318800.00  14924318801359750000000.002.00
14924318811.00  14924318811359760000000.001.00
14924318821.00  14924318821359770000000.001.00

With Isolation (stress test test, 10 second interval)

Note: stress was ran for 20 seconds in this case

Host View VM View    Hypervisor View
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07  
epochvalueepochvalueepochvaluepercent
    14924338151398840000000.00 
149243382577.56345149243383365.5771214924338251405960000000.0071.44
149243383585.4149243384340.2206614924338351414630000000.0086.70
149243384518.0722914924338531.00200414924338451416330000000.0017.00
14924338551.70511514924338631.20240514924338551416530000000.002.00
14924338651.606426149243387343.714924338651416730000000.002.00
149243387566.4149243388310014924338751423510000000.0067.79
1492433885100149243389359.3406614924338851433500000000.0099.91
149243389537.211641492433903014924338951437100000000.0036.00
14924339051.6064261492433903014924339051437300000000.002.00
14924339151.80360714924339130.90180414924339151437490000000.001.90

With Isolation (idle test)

 

Host View/cpu plugin for cpu 10VM ViewVirt view Virt Other metrics        
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07

 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07



6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

     the physical user/system cpu time consumed by the hypervisor, per-vm 
epochvalueepochvalueepochvaluepercent epochvaluepercentepochusersyst% user % systepochvaluepercent
1488880609.3401488880609.1401488880613.3716590000000.00  14888806133600000000 1488880613.37107400000009590000000.00  148888061331540000000 
1488880610.3401488880610.140               
1488880611.3401488880611.140               
1488880612.3401488880612.140               
1488880613.3401488880613.140.990099               
1488880614.340.9900991488880614.140               
1488880615.3401488880615.140               
1488880616.3401488880616.140               
1488880617.3401488880617.140               
1488880618.340.9900991488880618.140.990099               
1488880619.3401488880619.140               
1488880620.3401488880620.140               
1488880621.3401488880621.140               
1488880622.3401488880622.140               
1488880623.3401488880623.1401488880623.3616610000000.000.20 148888062336300000000.301488880623.36107500000009610000000.000.100.201488880623316000000000.60
1488880624.3401488880624.140               
1488880625.3401488880625.140               
1488880626.3401488880626.140               
1488880627.3411488880627.140               
1488880628.341.9801981488880628.140               
1488880629.3401488880629.140               
1488880630.3401488880630.140.990099               
1488880631.3401488880631.140               
1488880632.3401488880632.140               
1488880633.3401488880633.1401488880633.3616630000000.000.20 148888063336500000000.201488880633.36107600000009630000000.000.100.201488880633316600000000.60
1488880634.3401488880634.140               
1488880635.3401488880635.140               
1488880636.3401488880636.140               
1488880637.3401488880637.140               
1488880638.3411488880638.141               
1488880639.3411488880639.140               
1488880640.3401488880640.140               
1488880641.3401488880641.140               
1488880642.340.9900991488880642.140.990099               
1488880643.3401488880643.141.9801981488880643.3616660000000.000.30 148888064336700000000.201488880643.36107700000009650000000.000.100.201488880643317200000000.60
1488880644.3411488880644.141.980198               
1488880645.3401488880645.141.020408               
1488880646.342.9702971488880646.140               
1488880647.3401488880647.140               
1488880648.3411488880648.141.980198               
1488880649.3401488880649.140               
1488880650.3401488880650.140               
1488880651.340.9900991488880651.140               
1488880652.3401488880652.141               
1488880653.3411488880653.1401488880653.3616730000000.000.70 148888065337200000000.501488880653.36107800000009690000000.000.100.401488880653318700000001.50
1488880654.3411488880654.142.020202               
1488880655.342.0202021488880655.140               
1488880656.340.9900991488880656.140               
1488880657.3401488880657.140               
1488880658.3421488880658.141               
1488880659.3401488880659.140               
1488880660.3401488880660.140               
1488880661.3401488880661.140               
1488880662.3401488880662.140               
1488880663.3401488880663.1401488880663.3616780000000.000.50 148888066337700000000.501488880663.36108000000009720000000.000.200.301488880663319800000001.10
1488880664.3401488880664.140               
1488880665.3401488880665.140               
1488880666.3401488880666.141               
1488880667.341.9801981488880667.140.990099               
1488880668.341.0101011488880668.141.010101               
1488880669.3401488880669.140               
1488880670.3401488880670.140               
1488880671.3401488880671.140               
1488880672.343.9603961488880672.1410.89109               
1488880673.3471488880673.1401488880673.3716910000000.001.30 148888067339400000001.701488880673.37108200000009790000000.000.200.701488880673323200000003.40
1488880674.3401488880674.140               
1488880675.3401488880675.140               
1488880676.3401488880676.140.990099               
1488880677.3401488880677.140               
1488880678.3411488880678.142               
1488880679.3401488880679.140               
1488880680.3401488880680.140               
1488880681.341.9607841488880681.140               
1488880682.3401488880682.140               
1488880683.3401488880683.1401488880683.3616940000000.000.30 148888068339600000000.201488880683.36108300000009810000000.000.100.201488880683323900000000.70
1488880684.3401488880684.140               
1488880685.3401488880685.140               
1488880686.3401488880686.140               
1488880687.3401488880687.140               
1488880688.3411488880688.141               
1488880689.3401488880689.140               
1488880690.3401488880690.140               
1488880691.3401488880691.140.990099               
1488880692.340.9900991488880692.140               
1488880693.3401488880693.1411488880693.3616960000000.000.20 148888069339900000000.301488880693.36108400000009830000000.000.100.201488880693324500000000.60
1488880694.3401488880694.140               
1488880695.3401488880695.140               
1488880696.3401488880696.140               
1488880697.340.9900991488880697.140.990099               
1488880698.3401488880698.140               
1488880699.3401488880699.140               
1488880700.3401488880700.140               
1488880701.3401488880701.140               
1488880702.340.9900991488880702.140               
1488880703.3401488880703.1401488880703.3616970000000.000.10 148888070340300000000.401488880703.36108500000009850000000.000.100.201488880703325100000000.60

Without Isolation (idle test)

 

Host View VM View Virt view  
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07  
epochvalueepochvalueepochvaluepercent
1488877698.942.0202021488877698.641.0101011488877697.98628610000000.00 
1488877699.9414.141411488877699.640   
1488877700.9401488877700.640   
1488877701.9401488877701.640   
1488877702.9401488877702.640   
1488877703.942.9411761488877703.640   
1488877704.9401488877704.640   
1488877705.9401488877705.640   
1488877706.9401488877706.640   
1488877707.9401488877707.6401488877707.97628760000000.001.50
1488877708.9401488877708.640   
1488877709.941.9801981488877709.640   
1488877710.9401488877710.640   
1488877711.9401488877711.640   
1488877712.9401488877712.640   
1488877713.9401488877713.640   
1488877714.9401488877714.640   
1488877715.9401488877715.640   
1488877716.9401488877716.640   
1488877717.9401488877717.6401488877717.96628760000000.000.00
1488877718.9421488877718.641.010101   
1488877719.9411488877719.640   
1488877720.9401488877720.640   
1488877721.940.9900991488877721.640   
1488877722.9401488877722.640   
1488877723.9401488877723.640   
1488877724.941.9801981488877724.640   
1488877725.9401488877725.640   
1488877726.9401488877726.640   
1488877727.9411488877727.6401488877727.96628790000000.000.30
1488877728.9411488877728.641   
1488877729.941.9801981488877729.640   
1488877730.9401488877730.640   
1488877731.9401488877731.640   
1488877732.9401488877732.640   
1488877733.9401488877733.640   
1488877734.9401488877734.640   
1488877735.940.9900991488877735.640   
1488877736.9401488877736.640   
1488877737.940.9900991488877737.6501488877737.96628800000000.000.10
1488877738.948.0808081488877738.642.020202   
1488877739.9421488877739.640   
1488877740.9401488877740.640   
1488877741.9401488877741.640   
1488877742.9401488877742.640   
1488877743.9401488877743.640   
1488877744.9401488877744.640   
1488877745.940.9900991488877745.640   
1488877746.9401488877746.640   
1488877747.9401488877747.6401488877747.96628830000000.000.30
1488877748.9421488877748.640   
1488877749.9401488877749.640   
1488877750.9401488877750.640   
1488877751.9401488877751.640   
1488877752.9401488877752.640   
1488877753.9401488877753.641   
1488877754.9401488877754.640   
1488877755.9401488877755.640   
1488877756.9411488877756.640   
1488877757.940.9900991488877757.6401488877757.97628840000000.000.10
1488877758.941.0101011488877758.641.010101   
1488877759.9421488877759.640   
1488877760.9401488877760.640   
1488877761.9401488877761.640   
1488877762.940.9900991488877762.640   
1488877763.9401488877763.640   
1488877764.9401488877764.640   
1488877765.9401488877765.640   
1488877766.9411488877766.640   
1488877767.9401488877767.6401488877767.97628880000000.000.40
1488877768.9431488877768.641   
1488877769.941.0101011488877769.640   
1488877770.941.9607841488877770.640   
1488877771.9401488877771.640   
1488877772.9411488877772.640   
1488877773.9401488877773.640   
1488877774.940.9900991488877774.640   
1488877775.9411488877775.640   
1488877776.9401488877776.640   
1488877777.9401488877777.6501488877777.96628890000000.000.10
1488877778.943.9603961488877778.641   
1488877779.9411488877779.640   
1488877780.9401488877780.640   
1488877781.9401488877781.640   
1488877782.9401488877782.640   
1488877783.9401488877783.640   
1488877784.940.9900991488877784.640   
1488877785.940.9900991488877785.640   
1488877786.9401488877786.640   
1488877787.9401488877787.6401488877787.96628900000000.000.10
1488877788.9411488877788.640   
1488877789.941.9801981488877789.640   
1488877790.9401488877790.640   
1488877791.9401488877791.640   
1488877792.9401488877792.640   
1488877793.9411488877793.640   
1488877794.9401488877795.640   
1488877795.9411488877796.640   
1488877796.9411488877796.640   
1488877797.9401488877797.6501488877797.96628920000000.000.20

 

 

With core isloation from OS
Host ViewVM ViewVirt viewVirt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07
epochvalueepochvalueepochvaluepercentepochusersyst% user % syst
1488880609.3401488880609.1401488880613.3716590000000.00 1488880613.37107400000009590000000.00  
1488880610.3401488880610.140        
1488880611.3401488880611.140        
1488880612.3401488880612.140        
1488880613.3401488880613.140.990099        
1488880614.340.9900991488880614.140        
1488880615.3401488880615.140        
1488880616.3401488880616.140        
1488880617.3401488880617.140        
1488880618.340.9900991488880618.140.990099        
1488880619.3401488880619.140        
1488880620.3401488880620.140        
1488880621.3401488880621.140        
1488880622.3401488880622.140        
1488880623.3401488880623.1401488880623.3616610000000.000.201488880623.36107500000009610000000.000.100.20
1488880624.3401488880624.140        
1488880625.3401488880625.140        
1488880626.3401488880626.140        
1488880627.3411488880627.140        
1488880628.341.9801981488880628.140        
1488880629.3401488880629.140        
1488880630.3401488880630.140.990099        
1488880631.3401488880631.140        
1488880632.3401488880632.140        
1488880633.3401488880633.1401488880633.3616630000000.000.201488880633.36107600000009630000000.000.100.20
1488880634.3401488880634.140        
1488880635.3401488880635.140        
1488880636.3401488880636.140        
1488880637.3401488880637.140        
1488880638.3411488880638.141        
1488880639.3411488880639.140        
1488880640.3401488880640.140        
1488880641.3401488880641.140        
1488880642.340.9900991488880642.140.990099        
1488880643.3401488880643.141.9801981488880643.3616660000000.000.301488880643.36107700000009650000000.000.100.20
1488880644.3411488880644.141.980198        
1488880645.3401488880645.141.020408        
1488880646.342.9702971488880646.140        
1488880647.3401488880647.140        
1488880648.3411488880648.141.980198        
1488880649.3401488880649.140        
1488880650.3401488880650.140        
1488880651.340.9900991488880651.140        
1488880652.3401488880652.141        
1488880653.3411488880653.1401488880653.3616730000000.000.701488880653.36107800000009690000000.000.100.40
1488880654.3411488880654.142.020202        
1488880655.342.0202021488880655.140        
1488880656.340.9900991488880656.140        
1488880657.3401488880657.140        
1488880658.3421488880658.141        
1488880659.3401488880659.140        
1488880660.3401488880660.140        
1488880661.3401488880661.140        
1488880662.3401488880662.140        
1488880663.3401488880663.1401488880663.3616780000000.000.501488880663.36108000000009720000000.000.200.30
1488880664.3401488880664.140        
1488880665.3401488880665.140        
1488880666.3401488880666.141        
1488880667.341.9801981488880667.140.990099        
1488880668.341.0101011488880668.141.010101        
1488880669.3401488880669.140        
1488880670.3401488880670.140        
1488880671.3401488880671.140        
1488880672.343.9603961488880672.1410.89109        
1488880673.3471488880673.1401488880673.3716910000000.001.301488880673.37108200000009790000000.000.200.70
1488880674.3401488880674.140        
1488880675.3401488880675.140        
1488880676.3401488880676.140.990099        
1488880677.3401488880677.140        
1488880678.3411488880678.142        
1488880679.3401488880679.140        
1488880680.3401488880680.140        
1488880681.341.9607841488880681.140        
1488880682.3401488880682.140        
1488880683.3401488880683.1401488880683.3616940000000.000.301488880683.36108300000009810000000.000.100.20
1488880684.3401488880684.140        
1488880685.3401488880685.140        
1488880686.3401488880686.140        
1488880687.3401488880687.140        
1488880688.3411488880688.141        
1488880689.3401488880689.140        
1488880690.3401488880690.140        
1488880691.3401488880691.140.990099        
1488880692.340.9900991488880692.140 
Host ViewVM ViewVirt viewsilpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07
       
1488880609
1488880693.340
1488880609
1488880693.1411488880693.3616960000000.000.20
1488880613
1488880693.
37
3610840000000
16590000000
9830000000.000.100.20
1488880694.3401488880694.140 
1488880610
       
1488880695.340
1488880610
1488880695.140   
1488880611.34
 
01488880611.140
    
1488880612
1488880696.340
1488880612
1488880696.140   
1488880613.34
 
01488880613.140.990099
    
1488880614
1488880697.340.990099
1488880614
1488880697.140.990099   
1488880615.34
 
01488880615.140
    
1488880616
1488880698.340
1488880616
1488880698.140   
1488880617.3401488880617.14
  
0
   
1488880618
1488880699.340
.990099
1488880618
1488880699.140
.990099
   
1488880619.3401488880619.14
  
0
   
1488880620
1488880700.340
1488880620
1488880700.140   
1488880621.3401488880621.140
     
1488880622
1488880701.340
1488880622
1488880701.140   
1488880623.34
 
0
 
1488880623.14
 
0
 
1488880623.36
 
16610000000
1488880702.
00
340.
201488880624.
990099
1488880624.340
1488880702.140   
1488880625.3401488880625.14
  
0
   
1488880626
1488880703.340
1488880626
1488880703.140
   1488880627.3411488880627.140   1488880628.341.9801981488880628.140   1488880629.3401488880629.140   1488880630.3401488880630.140.990099   1488880631.3401488880631.140   1488880632.3401488880632.140   1488880633.3401488880633.1401488880633.3616630000000.000.201488880634.3401488880634.14
1488880703.3616970000000.000.101488880703.36108500000009850000000.000.100.20

 

 

Host ViewVM ViewVirt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07
epochvalueepochvalueepochvaluepercentepochvaluepercentepochusersyst% user % systepochvaluepercent
    
0
   
1488880635.34
 
0
 
1488880635.14
 
0
    
1488880636.34
 
0
1492431855
1488880636
1343320000000.
140
00 
 1488880637
1492431856
 
78.
34
35
0
1492431856
1488880637
62.
14
19512
0
    
1488880638.34
 
1
 
1488880638.14
 
1
    
1488880639.3411488880639.14
14924318561344110000000.0079.72
1492431857100.00149243185782.02247 
0
   
1488880640.34
 
0
 
1488880640.14
 
0
    
1488880641.3401488880641.14
14924318571345110000000.00100.00
1492431858100.00149243185856.25
0
   
1488880642.340.9900991488880642.14
    
0.990099
   
1488880643.34
 
0
1492431858
1488880643
1346110000000.
14
00
1
100.
980198
00
1488880643.36
1492431859
16660000000
100.00
0.30
1492431859
1488880644
62.
34
5
11488880644.14
  
1.980198
   
1488880645.34
0
 
1488880645.14
  
1.020408
   
1488880646
14924318591347110000000.
34
00
2
100.
970297
00
1492431860
1488880646
100.
140
00149243186089.58333   
1488880647.34
 
0
 
1488880647.14
 
0
    
1488880648.34
 
1
1492431860
1488880648
1348110000000.
14
00
1
100.
980198
10
   
1492431861100.00149243186175.25773 
1488880649.3401488880649.140
   
1488880650.34
 
0
 
1488880650.14
 
0
    
1488880651
14924318611349110000000.
34
00
0
99.
990099
90
1492431862
1488880651
23.
14
0014924318620   
1488880652.34
 
0
 
1488880652.14
 
1
    
1488880653.34
 
1
1492431862
1488880653
1349310000000.
1416730000000.
00
01488880653.36
20.00
0.70
1492431863
1488880654
2.
34
02
1
1492431863
1488880654.14
0 
2.020202
   
1488880655.342.0202021488880655.14
    
0
   1492431863
1488880656
1349330000000.
34
00
0
2.
990099
00
1492431864
1488880656
1.
14
0014924318640   
1488880657.3401488880657.14
     
0
   
1488880658
14924318641349350000000.
34
002.00
1488880658.14
14924318651
   1488880659.3401488880659.14
.01149243186519.19192 
0
   
1488880660.34
 
0
 
1488880660.14
 
0
    
1488880661.340
14924318651349390000000.004.00
149243186619.191492431866
1488880661.14
0   
1488880662.34
 
0
 
1488880662.14
 
0
    
1488880663.34
 
0
1492431866
1488880663
1349550000000.
141488880663
00
0
16.
36
00
16780000000.00
14924318670.
50
99
1488880664.3401488880664.14
149243186714.14141 
0
   
1488880665.34
 
0
 
1488880665.14
 
0
    
1488880666.3401488880666.14
14924318671349570000000.002.00
149243186882.831492431868100
1
   
1488880667.341.9801981488880667.14
    
0.990099
   
1488880668.341.0101011488880668.141.010101   1488880669.3401488880669.14
 14924318681350420000000.0084.92
1492431869100.001492431869100 
0
   
1488880670.34
 
0
 
1488880670.14
 
0
    
1488880671.3401488880671.14
14924318691351420000000.00100.00
1492431870100.001492431870100
0
   
1488880672.343.9603961488880672.14
    
10.89109
   
1488880673.34
 
7
1492431870
1488880673
1352420000000.
1416910000000.
00
01488880673.37
100.00
1.30
1492431871
1488880674
100.
34
00
0
1492431871
1488880674.14
100
0
    
1488880675.34
 
0
 
1488880675.14
 
0
    
1488880676.3401488880676.14
14924318711353420000000.00100.00
1492431872100.001492431872100
0.990099
   
1488880677.34
 
0
 
1488880677.14
 
0
    
1488880678.34
 
1
1492431872
1488880678
1354420000000.
14
00
2
100.10
   1488880679.3401488880679.14
1492431873100.001492431873100 
0
   
1488880680.34
 
0
 
1488880680.14
 
0
    1492431873
1488880681
1355420000000.
34
00
1
99.
960784
90
1492431874
1488880681
100.
14
001492431874
0
100   
1488880682.3401488880682.14
  
0
   
1488880683.34
 
0
 
1488880683.14
 
0
1492431874
1488880683
1356420000000.
36
00
16940000000
100.00
0.30
1492431875
1488880684
100.
34
00
0
 
1488880684.14
 
0
    
1488880685.34
 
0
 
1488880685.14
 
0
    
1488880686.3401488880686.14
14924318751357420000000.00100.10
1492431876100.00   
0
   
1488880687.34
 
0
 
1488880687.14
 
0
    
1488880688.3411488880688.14
14924318761358420000000.0099.90
1492431877100.00 
1
   
1488880689.34
 
0
 
1488880689.14
 
0
    
1488880690.3401488880690.140
  
 
1492431877
1488880691
1359420000000.
34
00
01488880691.14
100.00
149243187833.33
0.990099
   
1488880692.340.9900991488880692.14
    
0
   
1488880693.34
 
0
 
1488880693.14
 
1
1492431878
1488880693
1359730000000.
36
00
16960000000
31.00
14924318790.
20
00
1488880694.34
 
0
 
1488880694.14
 
0
    
1488880695.34
 
0
 
1488880695.14
 
0
   
 
1492431879
1488880696
1359730000000.
34
000.00
1492431880
1488880696.14
0.00   
0
   
1488880697.340.9900991488880697.14
    
0.990099
   
1488880698.3401488880698.14
14924318801359750000000.002.00
14924318811.00 
0
   
1488880699.34
 
0
 
1488880699.14
 
0
    
1488880700.3401488880700.140
  
 
1492431881
 
1359760000000.00
1488880701
1.
34
00
0
1492431882
1488880701
1.
14
00
0
    
1488880702.340.9900991488880702.14
    
0
   
1488880703.34
 
0
 
1488880703.1401488880703.3616970000000.00
14924318821359770000000.001.00
0.10