Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Configuration/Setup instructions can be found here:  CPU Utilisation case study Configuration

The idea behind this wiki is to look at the CPU metrics from the baremetal host, the hypervisor and the VM itself to see the if the CPU utilisation metrics reported are consistent across all 3.

...

Two tests were conducted. One without isolating cpus from the OS and one with isolating CPUs from the OS on the baremetal host and pinning the vCPUs to those isolated cores. Results for the unisolated tests as expected have some unusual results... 

The CPU plugins for collectd were configured not to report the CPU meterics per state - so just a single metric for utilization

The interval used for the test was for 1 second .

Tests were conducted with more/less an idle environment and will be repeated with a busy env.

With Isolation

Core 10 and 11 were isolated from the OS using the isolcpus argument to grub command line and using virsh to Pin the vCPUs to those cores:

$ virsh vcpupin test 0 10
$ virsh vcpupin test 1 11
 
$ virsh vcpupin test
VCPU: CPU Affinity
----------------------------------
   0: 10
   1: 11

 

(this interval was used for the host and guest collectd cpu plugins, the virt plugin wouldn't go down to a 1 second configuration).

1 second was selected in order to get the timing differences between starting the plugin on the host and starting the plugin on the guest down to a lower value... 

Tests were conducted with more/less an idle environment and will be repeated with a busy env.

Function used to convert nanoseconds virt metrics to percent:

 static double cpu_ns_to_percent(unsigned int node_cpus,
                                 unsigned long long cpu_time_old,
                                 unsigned long long cpu_time_new) {
   double percent = 0.0;
   unsigned long long cpu_time_diff = 0;
   double time_diff_sec = CDTIME_T_TO_DOUBLE(plugin_get_interval());
   if (node_cpus != 0 && time_diff_sec != 0 && cpu_time_old != 0) {
     cpu_time_diff = cpu_time_new - cpu_time_old;
     percent = ((double)(100 * cpu_time_diff)) /
               (time_diff_sec * node_cpus * NANOSEC_IN_SEC);
  }

With Isolation (stress test test, 1 second interval)

Host ViewVM ViewHypervisor View

silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

epochvalueepochvalueepochvaluepercent
    14924318551343320000000.00 
149243185678.35149243185662.1951214924318561344110000000.0079.72
1492431857100.00149243185782.0224714924318571345110000000.00100.00
1492431858100.00149243185856.2514924318581346110000000.00100.00
1492431859100.00149243185962.514924318591347110000000.00100.00
1492431860100.00149243186089.5833314924318601348110000000.00100.10
1492431861100.00149243186175.2577314924318611349110000000.0099.90
149243186223.001492431862014924318621349310000000.0020.00
14924318632.021492431863014924318631349330000000.002.00
14924318641.001492431864014924318641349350000000.002.00
14924318651.01149243186519.1919214924318651349390000000.004.00
149243186619.191492431866014924318661349550000000.0016.00
14924318670.99149243186714.1414114924318671349570000000.002.00
149243186882.83149243186810014924318681350420000000.0084.92
1492431869100.00149243186910014924318691351420000000.00100.00
1492431870100.00149243187010014924318701352420000000.00100.00
1492431871100.00149243187110014924318711353420000000.00100.00
1492431872100.00149243187210014924318721354420000000.00100.10
1492431873100.00149243187310014924318731355420000000.0099.90
1492431874100.00149243187410014924318741356420000000.00100.00
1492431875100.00  14924318751357420000000.00100.10
1492431876100.00  14924318761358420000000.0099.90
1492431877100.00  14924318771359420000000.00100.00
149243187833.33  14924318781359730000000.0031.00
14924318790.00  14924318791359730000000.000.00
14924318800.00  14924318801359750000000.002.00
14924318811.00  14924318811359760000000.001.00
14924318821.00  14924318821359770000000.001.00

With Isolation (stress test test, 10 second interval)

Note: stress was ran for 20 seconds in this case

Host View VM View    Hypervisor View
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07  
epochvalueepochvalueepochvaluepercent
    14924338151398840000000.00 
149243382577.56345149243383365.5771214924338251405960000000.0071.44
149243383585.4149243384340.2206614924338351414630000000.0086.70
149243384518.0722914924338531.00200414924338451416330000000.0017.00
14924338551.70511514924338631.20240514924338551416530000000.002.00
14924338651.606426149243387343.714924338651416730000000.002.00
149243387566.4149243388310014924338751423510000000.0067.79
1492433885100149243389359.3406614924338851433500000000.0099.91
149243389537.211641492433903014924338951437100000000.0036.00
14924339051.6064261492433903014924339051437300000000.002.00
14924339151.80360714924339130.90180414924339151437490000000.001.90

With Isolation (idle test)

 

Host View/cpu plugin for cpu 10VM ViewVirt view Virt Other metrics        
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07

 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07



6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07

     the physical user/system cpu time consumed by the hypervisor, per-vm 
epochvalueepochvalueepochvaluepercent epochvaluepercentepochusersyst% user % systepochvaluepercent
1488880609.3401488880609.1401488880613.3716590000000.00  14888806133600000000 1488880613.37107400000009590000000.00  148888061331540000000 
1488880610.3401488880610.140               
1488880611.3401488880611.140               
1488880612.3401488880612.140               
1488880613.3401488880613.140.990099               
1488880614.340.9900991488880614.140               
1488880615.3401488880615.140               
1488880616.3401488880616.140               
1488880617.3401488880617.140               
1488880618.340.9900991488880618.140.990099               
1488880619.3401488880619.140               
1488880620.3401488880620.140               
1488880621.3401488880621.140               
1488880622.3401488880622.140               
1488880623.3401488880623.1401488880623.3616610000000.000.20 148888062336300000000.301488880623.36107500000009610000000.000.100.201488880623316000000000.60
1488880624.3401488880624.140               
1488880625.3401488880625.140               
1488880626.3401488880626.140               
1488880627.3411488880627.140               
1488880628.341.9801981488880628.140               
1488880629.3401488880629.140               
1488880630.3401488880630.140.990099               
1488880631.3401488880631.140               
1488880632.3401488880632.140               
1488880633.3401488880633.1401488880633.3616630000000.000.20 148888063336500000000.201488880633.36107600000009630000000.000.100.201488880633316600000000.60
1488880634.3401488880634.140               
1488880635.3401488880635.140               
1488880636.3401488880636.140               
1488880637.3401488880637.140               
1488880638.3411488880638.141               
1488880639.3411488880639.140               
1488880640.3401488880640.140               
1488880641.3401488880641.140               
1488880642.340.9900991488880642.140.990099               
1488880643.3401488880643.141.9801981488880643.3616660000000.000.30 148888064336700000000.201488880643.36107700000009650000000.000.100.201488880643317200000000.60
1488880644.3411488880644.141.980198               
1488880645.3401488880645.141.020408               
1488880646.342.9702971488880646.140               
1488880647.3401488880647.140               
1488880648.3411488880648.141.980198               
1488880649.3401488880649.140               
1488880650.3401488880650.140               
1488880651.340.9900991488880651.140               
1488880652.3401488880652.141               
1488880653.3411488880653.1401488880653.3616730000000.000.70 148888065337200000000.501488880653.36107800000009690000000.000.100.401488880653318700000001.50
1488880654.3411488880654.142.020202               
1488880655.342.0202021488880655.140               
1488880656.340.9900991488880656.140               
1488880657.3401488880657.140               
1488880658.3421488880658.141               
1488880659.3401488880659.140               
1488880660.3401488880660.140               
1488880661.3401488880661.140               
1488880662.3401488880662.140               
1488880663.3401488880663.1401488880663.3616780000000.000.50 148888066337700000000.501488880663.36108000000009720000000.000.200.301488880663319800000001.10
1488880664.3401488880664.140               
1488880665.3401488880665.140               
1488880666.3401488880666.141               
1488880667.341.9801981488880667.140.990099               
1488880668.341.0101011488880668.141.010101               
1488880669.3401488880669.140               
1488880670.3401488880670.140               
1488880671.3401488880671.140               
1488880672.343.9603961488880672.1410.89109               
1488880673.3471488880673.1401488880673.3716910000000.001.30 148888067339400000001.701488880673.37108200000009790000000.000.200.701488880673323200000003.40
1488880674.3401488880674.140               
1488880675.3401488880675.140               
1488880676.3401488880676.140.990099               
1488880677.3401488880677.140               
1488880678.3411488880678.142               
1488880679.3401488880679.140               
1488880680.3401488880680.140               
1488880681.341.9607841488880681.140               
1488880682.3401488880682.140               
1488880683.3401488880683.1401488880683.3616940000000.000.30 148888068339600000000.201488880683.36108300000009810000000.000.100.201488880683323900000000.70
1488880684.3401488880684.140               
1488880685.3401488880685.140               
1488880686.3401488880686.140               
1488880687.3401488880687.140               
1488880688.3411488880688.141               
1488880689.3401488880689.140               
1488880690.3401488880690.140               
1488880691.3401488880691.140.990099               
1488880692.340.9900991488880692.140               
1488880693.3401488880693.1411488880693.3616960000000.000.20 148888069339900000000.301488880693.36108400000009830000000.000.100.201488880693324500000000.60
1488880694.3401488880694.140               
1488880695.3401488880695.140               
1488880696.3401488880696.140               
1488880697.340.9900991488880697.140.990099               
1488880698.3401488880698.140               
1488880699.3401488880699.140               
1488880700.3401488880700.140               
1488880701.3401488880701.140               
1488880702.340.9900991488880702.140               
1488880703.3401488880703.1401488880703.3616970000000.000.10 148888070340300000000.401488880703.36108500000009850000000.000.100.201488880703325100000000.60

Without Isolation (idle test)

 

Host View VM View Virt view  
silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07  
epochvalueepochvalueepochvaluepercent
1488877698.942.0202021488877698.641.0101011488877697.98628610000000.00 
1488877699.9414.141411488877699.640   
1488877700.9401488877700.640   
1488877701.9401488877701.640   
1488877702.9401488877702.640   
1488877703.942.9411761488877703.640   
1488877704.9401488877704.640   
1488877705.9401488877705.640   
1488877706.9401488877706.640   
1488877707.9401488877707.6401488877707.97628760000000.001.50
1488877708.9401488877708.640   
1488877709.941.9801981488877709.640   
1488877710.9401488877710.640   
1488877711.9401488877711.640   
1488877712.9401488877712.640   
1488877713.9401488877713.640   
1488877714.9401488877714.640   
1488877715.9401488877715.640   
1488877716.9401488877716.640   
1488877717.9401488877717.6401488877717.96628760000000.000.00
1488877718.9421488877718.641.010101   
1488877719.9411488877719.640   
1488877720.9401488877720.640   
1488877721.940.9900991488877721.640   
1488877722.9401488877722.640   
1488877723.9401488877723.640   
1488877724.941.9801981488877724.640   
1488877725.9401488877725.640   
1488877726.9401488877726.640   
1488877727.9411488877727.6401488877727.96628790000000.000.30
1488877728.9411488877728.641   
1488877729.941.9801981488877729.640   
1488877730.9401488877730.640   
1488877731.9401488877731.640   
1488877732.9401488877732.640   
1488877733.9401488877733.640   
1488877734.9401488877734.640   
1488877735.940.9900991488877735.640   
1488877736.9401488877736.640   
1488877737.940.9900991488877737.6501488877737.96628800000000.000.10
1488877738.948.0808081488877738.642.020202   
1488877739.9421488877739.640   
1488877740.9401488877740.640   
1488877741.9401488877741.640   
1488877742.9401488877742.640   
1488877743.9401488877743.640   
1488877744.9401488877744.640   
1488877745.940.9900991488877745.640   
1488877746.9401488877746.640   
1488877747.9401488877747.6401488877747.96628830000000.000.30
1488877748.9421488877748.640   
1488877749.9401488877749.640   
1488877750.9401488877750.640   
1488877751.9401488877751.640   
1488877752.9401488877752.640   
1488877753.9401488877753.641   
1488877754.9401488877754.640   
1488877755.9401488877755.640   
1488877756.9411488877756.640   
1488877757.940.9900991488877757.6401488877757.97628840000000.000.10
1488877758.941.0101011488877758.641.010101   
1488877759.9421488877759.640   
1488877760.9401488877760.640   
1488877761.9401488877761.640   
1488877762.940.9900991488877762.640   
1488877763.9401488877763.640   
1488877764.9401488877764.640   
1488877765.9401488877765.640   
1488877766.9411488877766.640   
1488877767.9401488877767.6401488877767.97628880000000.000.40
1488877768.9431488877768.641   
1488877769.941.0101011488877769.640   
1488877770.941.9607841488877770.640   
1488877771.9401488877771.640   
1488877772.9411488877772.640   
1488877773.9401488877773.640   
1488877774.940.9900991488877774.640   
1488877775.9411488877775.640   
1488877776.9401488877776.640   
1488877777.9401488877777.6501488877777.96628890000000.000.10
1488877778.943.9603961488877778.641   
1488877779.9411488877779.640   
1488877780.9401488877780.640   
1488877781.9401488877781.640   
1488877782.9401488877782.640   
1488877783.9401488877783.640   
1488877784.940.9900991488877784.640   
1488877785.940.9900991488877785.640   
1488877786.9401488877786.640   
1488877787.9401488877787.6401488877787.96628900000000.000.10
1488877788.9411488877788.640   
1488877789.941.9801981488877789.640   
1488877790.9401488877790.640   
1488877791.9401488877791.640   
1488877792.9401488877792.640   
1488877793.9411488877793.640   
1488877794.9401488877795.640   
1488877795.9411488877796.640   
1488877796.9411488877796.640   
1488877797.9401488877797.6501488877797.96628920000000.000.20

 

 

With core isloation from OS
Host ViewVM ViewVirt view
Host ViewVM View
Virt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-07
epochvalueepochvalueepochvaluepercentepochusersyst% user % syst
1488880609.3401488880609.1401488880613.3716590000000.00 1488880613.37107400000009590000000.00  
1488880610.3401488880610.
140
140        
1488880611.3401488880611.140        
1488880611
1488880612.340
1488880611
1488880612.140   
1488880612.3401488880612.140
     
1488880613.3401488880613.140.990099        
1488880614.340.9900991488880614.140        
1488880615.3401488880615.140        
1488880616.3401488880616.140        
1488880617.3401488880617.140        
1488880618.340.9900991488880618.140.990099        
1488880619.3401488880619.14
0
0        
1488880620.3401488880620.140        
1488880621.3401488880621.140        
1488880622.3401488880622.140        
1488880623.3401488880623.1401488880623.3616610000000.000.201488880623.36107500000009610000000.000.100.20
1488880624.3401488880624.140        
1488880625.3401488880625.140        
1488880626.3401488880626.140        
1488880627.3411488880627.14
0
0        
1488880628.341.9801981488880628.140        
1488880629.3401488880629.140        
1488880630.3401488880630.140.990099        
1488880631.3401488880631.140        
1488880632.3401488880632.140        
1488880633.3401488880633.1401488880633.3616630000000.000.201488880633.36107600000009630000000.000.100.20
1488880634.3401488880634.140        
1488880635.3401488880635.
140
140        
1488880636.3401488880636.140        
1488880637.3401488880637.140        
1488880638.3411488880638.141        
1488880639.3411488880639.140        
1488880640.3401488880640.140        
1488880641.3401488880641.140        
1488880642.340.9900991488880642.140.990099        
1488880643.3401488880643.141.9801981488880643.3616660000000.000.301488880643.36
107700000009650000000.000.10
16660000000.00
0.
30
20
1488880644.3411488880644.141.980198        
1488880645.3401488880645.141.020408        
1488880646.342.9702971488880646.140        
1488880647.3401488880647.140        
1488880648.3411488880648.141.980198        
1488880649.3401488880649.140        
1488880650.3401488880650.140        
1488880651.340.9900991488880651.140
.9900991488880651.14
     
0
   
1488880652.3401488880652.141        
1488880653.3411488880653.1401488880653.3616730000000.000.701488880653.36107800000009690000000.000.100.40
1488880654.3411488880654.142.020202        
1488880655.342.0202021488880655.140        
1488880656.340.9900991488880656.140        
1488880657.3401488880657.140        
1488880658.34
21488880658.14
21488880658.141     
1
   
1488880659.3401488880659.140        
1488880660.3401488880660.140        
1488880661.3401488880661.140        
1488880662.3401488880662.140        
1488880663.3401488880663.1401488880663.3616780000000.000.501488880663.36108000000009720000000.000.200.30
1488880664.3401488880664.140   
1488880665.34
 
01488880665.140
    
1488880666
1488880665.340
1488880666
1488880665.14
1
0   
1488880667.341.9801981488880667.14
  
0.990099
   
1488880668
1488880666.34
1.010101
0
1488880668
1488880666.141
.010101
   
1488880669.3401488880669.140
     
1488880670
1488880667.34
0
1.980198
1488880670
1488880667.140.990099   
1488880671.3401488880671.14
  
0
   
1488880672
1488880668.34
3
1.
960396
010101
1488880672
1488880668.14
10
1.
89109
010101   
1488880673.34
 
7
 
1488880673.14
 
0
 
1488880673.3716910000000.001.30
 
1488880669
1488880674
.340
1488880674
1488880669.140       
1488880675
 
1488880670.340
1488880675
1488880670.140   
1488880676
     
1488880671.340
1488880676
1488880671.140
.990099
        
1488880677
1488880672.34
0
3.960396
1488880677
1488880672.14
0
10.89109        
1488880678
1488880673.3471488880673.1401488880673.3716910000000.001.30
1488880678
1488880673.
14
37
2
10820000000
   
9790000000.000.200.70
1488880674
1488880679
.340
1488880679
1488880674.140   
1488880680.34
 
01488880680.140
    
1488880681
1488880675.34
1.960784
0
1488880681
1488880675.140   
1488880682.3401488880682.14
  
0
   
1488880683
1488880676.340
1488880683
1488880676.140
1488880683
.
36
990099
16940000000.000.301488880684.3401488880684.140
        
1488880685
1488880677.340
1488880685
1488880677.140   
1488880686.3401488880686.14
  
0
   
1488880687
1488880678.34
0
1
1488880687
1488880678.14
0
2   
1488880688.34
 
11488880688.141
    
1488880689
1488880679.340
1488880689
1488880679.140   
1488880690.3401488880690.14
  
0
   
1488880691
1488880680.340
1488880691
1488880680.140
.990099
   
1488880692.340.9900991488880692.140
     
1488880693
1488880681.34
01488880693.14
1
1488880693
.
36
960784
16960000000
1488880681.
00
140
.201488880694.3401488880694.14
     
0
   
1488880695
1488880682.340
1488880695
1488880682.140   
1488880696.34
 
01488880696.140
    
1488880697
1488880683.340
.990099
1488880697
1488880683.140
.990099   
1488880683.3616940000000.000.301488880683.36108300000009810000000.000.100.20
1488880684
1488880698.3401488880698.140   1488880699
.340
1488880699
1488880684.140   
1488880700.34
 
01488880700.140
    
1488880701
1488880685.340
1488880701
1488880685.140   
1488880702.340.9900991488880702.14
  
0
   
1488880703
1488880686.340
1488880703
1488880686.140
1488880703.3616970000000.000.10
     

...

  
Host View VM View Virt view  silpixa00399126/cpu-10/percent-active-2017-03-07 ubuntu_vm/cpu-0/percent-active-2017-03-07 6c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-07  epochvalueepochvalueepochvaluepercent1488877698.942.0202021488877698.641.0101011488877697.98628610000000.00 1488877699.9414.141411488877699.64
 
1488880687.3401488880687.140        
1488880688.3411488880688.141     
0
   
1488877700
1488880689.
94
340
1488877700
1488880689.
64
140   
1488877701.94
 
01488877701.640
    
1488877702
1488880690.
94
340
1488877702
1488880690.
64
140   
1488877703.942.9411761488877703.64
  
0
   
1488877704
1488880691.
94
340
1488877704
1488880691.
64
140.990099   
1488877705.94
     
1488880692.340.990099
1488877705
1488880692.
64
140   
1488877706.94
 
01488877706.640
    
1488877707
1488880693.
94
340
1488877707
1488880693.
64
14
0
1
1488877707
1488880693.
97
36
628760000000
16960000000.00
1
0.
50
20
1488877708
1488880693.
94
36
0
10840000000
1488877708
9830000000.
64
000.10
   1488877709.941.9801981488877709.64
0.20
1488880694.3401488880694.140   
1488877710.94
 
01488877710.640
    
1488877711
1488880695.
94
340
1488877711
1488880695.
64
140   
1488877712.9401488877712.64
  
0
   
1488877713
1488880696.
94
340
1488877713
1488880696.
64
140   
1488877714.9401488877714.640
     
1488877715
1488880697.
94
340.990099
1488877715
1488880697.
64
140.990099   
1488877716.9401488877716.64
  
0
   
1488877717.9401488877717.64
1488880698.340
1488877717
1488880698.
96
14
628760000000.000.001488877718.9421488877718.64
0     
1.010101
   
1488877719
1488880699.
94
34
1
0
1488877719
1488880699.
64
140   
1488877720.9401488877720.64
  
0
   
1488877721
1488880700.
94
340
.990099
1488877721
1488880700.
64
140   
1488877722.94
 
01488877722.640
    
1488877723
1488880701.
94
340
1488877723
1488880701.
64
140   
1488877724.941.9801981488877724.64
  
0
   
1488877725
1488880702.
94
340.990099
1488877725
1488880702.
64
140   
1488877726.94
 
0
 
1488877726.64
 
0
  
 
1488880703.34
 
0
1488877727
1488880703.
94
1401488880703.3616970000000.000.101488880703.36108500000009850000000
11488877727.6401488877727.96628790000000
.000.
30
10
1488877728.9411488877728.641   1488877729.941.9801981488877729.640   1488877730.9401488877730.640   1488877731.9401488877731.640   1488877732.9401488877732.64
0.20

 

 

Host ViewVM ViewVirt view
silpixa00399126/cpu-10/percent-active-2017-03-07ubuntu_vm/cpu-0/percent-active-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-0-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_vcpu-1-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/ps_cputime-2017-03-076c33ff2f-40d0-7a45-873b-7f3ead32413b/virt/virt_cpu_total-2017-03-07
epochvalueepochvalueepochvaluepercentepochvaluepercentepochusersyst% user % systepochvaluepercent
0
   
1488877733.94
 
0
 
1488877733.64
 
0
    
1488877734.94
 
0
 
1488877734.64
 
0
  14924318551343320000000.00 
 
1492431856
1488877735
78.
94
35
0.990099
1492431856
1488877735
62.
64
19512
0
    
1488877736.94
 
0
 
1488877736.64
 
0
    
1488877737.94
1492431856
0
1344110000000.
990099
00
1488877737
79.
65
72
01488877737.96
1492431857100
628800000000
.00
0.10
149243185782.02247 
1488877738.948.0808081488877738.642.020202
   
1488877739.94
 
2
 
1488877739.64
 
0
    
1488877740.9401488877740.64
14924318571345110000000.00100.00
1492431858100.00149243185856.25
0
   
1488877741.94
 
0
 
1488877741.64
 
0
    
1488877742.94
 
0
1492431858
1488877742
1346110000000.
64
00
0
100.00
   1488877743.9401488877743.64
1492431859100.00149243185962.5 
0
   
1488877744.94
 
0
 
1488877744.64
 
0
    
1488877745
14924318591347110000000.
94
00
0
100.
990099
00
1492431860
1488877745
100.
64
00149243186089.58333
0
   
1488877746.94
 
0
 
1488877746.64
 
0
    
1488877747.94
 
0
1492431860
1488877747
1348110000000.
64
00
01488877747.96
100.10
1492431861100
628830000000
.00
0.30
149243186175.25773        
1488877748.9421488877748.640
   
1488877749.940
14924318611349110000000.0099.90
149243186223.001492431862
1488877749.64
0   
1488877750.9401488877750.64
     
0
   
1488877751.9401488877751.64
14924318621349310000000.0020.00
14924318632.0214924318630 
0
   
1488877752.94
 
0
 
1488877752.64
 
0
    
1488877753.9401488877753.64
14924318631349330000000.002.00
14924318641.0014924318640 
1
   
1488877754.94
 
0
 
1488877754.64
 
0
    
1488877755.9401488877755.64
14924318641349350000000.002.00
14924318651.01149243186519.19192
0
   
1488877756.94
 
1
 
1488877756.64
 
0
    
1488877757.940.9900991488877757.6401488877757.97628840000000.000.101488877758.941.0101011488877758.64
 14924318651349390000000.004.00
149243186619.1914924318660 
1.010101
   
1488877759.94
 
2
 
1488877759.64
 
0
    
1488877760.9401488877760.64
14924318661349550000000.0016.00
14924318670.99149243186714.14141 
0
   
1488877761.94
 
0
 
1488877761.64
 
0
    
1488877762
14924318671349570000000.
94
00
0
2.
990099
00
1492431868
1488877762.64
82.831492431868100 
0
   
1488877763.94
 
0
 
1488877763.64
 
0
    
1488877764.9401488877764.64
14924318681350420000000.0084.92
1492431869100.001492431869100
0
   
1488877765.94
 
0
 
1488877765.64
 
0
    
1488877766.94
 
1
1492431869
1488877766
1351420000000.
64
00
0
100.00
   1488877767.9401488877767.6401488877767.97628880000000.000.401488877768.9431488877768.64
1492431870100.001492431870100        
1
   1492431870
1488877769
1352420000000.
94
00
1
100.
010101
00
1492431871
1488877769.64
100.001492431871100 
0
   
1488877770.941.9607841488877770.64
    
0
   
1488877771.9401488877771.64
14924318711353420000000.00100.00
1492431872100.001492431872100 
0
   
1488877772.94
 
1
 
1488877772.64
 
0
    
1488877773.9401488877773.64
14924318721354420000000.00100.10
1492431873100.001492431873100  
0
   
1488877774.940.9900991488877774.64
    
0
  1492431873
 
1355420000000.00
1488877775
99.
94
90
1
1492431874
1488877775
100.
64
001492431874
0
100   
1488877776.94
 
0
 
1488877776.64
 
0
    
1488877777.94
 
0
1492431874
1488877777
1356420000000.
6501488877777.96
00100
628890000000
.00
0.10
1492431875
1488877778
100.
94
00
3.9603961488877778.64
   
1
   
1488877779.9411488877779.640
    
1488877780.9401488877780.640
   
1488877781.9401488877781.64
14924318751357420000000.00100.10
1492431876100.00   
0
   
1488877782.94
 
0
 
1488877782.64
 
0
    
1488877783.9401488877783.64
14924318761358420000000.0099.90
1492431877100.00   
0
   
1488877784.940.9900991488877784.64
    
0
   1492431877
1488877785
1359420000000.
94
00
0
100.
990099
00
1492431878
1488877785
33.
64
33
0
   
1488877786.94
 
0
 
1488877786.64
 
0
    
1488877787.94
 
0
 
1488877787.64
 
0
1492431878
1488877787
1359730000000.
96
00
628900000000
31.00
14924318790.
10
00
1488877788.94
 
1
 
1488877788.64
 
0
    
1488877789.941.9801981488877789.64
    
0
  
 
1492431879
1488877790
1359730000000.
94
000.00
1488877790.64
14924318800.00   
1488877791.9401488877791.64
  
0
   
1488877792.9401488877792.64
  
0
   
1488877793.9411488877793.64
14924318801359750000000.002.00
14924318811.00   
0
   
1488877794.94
 
0
 
1488877795.64
 
0
    
1488877795
14924318811359760000000.
94
001.00
1492431882
1488877796
1.
64
00
0
    
1488877796.94
 
1
 
1488877796.64
 
0
    
1488877797.94
 
0
 
1488877797.6501488877797.96628920000000.00
14924318821359770000000.001.00
0.20