Hiccups

In real time application and other time sensitive application it is crucial to have 100% of the CPU's cycles to the application.
In many cases there are interferences of different kinds, even if we isolate CPU cores for application.

The hiccups applican is an accurate measurement tool to detect those intereferences from different sources.

It runs on the requested cpus, and gets the hardware timestamp counter in tight loop, and check what is the time difference from the previous sample. For these differences it stores the minimum, maximum, number of samples, and histogram.

Program output:

Per thread the program will print:

         S1       S2            S3            S4          S5      S6           S7         S8              S9               S10                  S11
         |        |             |             |           |       |            |          |               |                |                    |   
thread#: 0 core#: 1 samples: 1654599  avg: 3.0218 min: 0.0010 (@769253) max: 6102.9484 (@1490) cycles: 10085426457 start: 21322780850491 end: 21332866276948
  [000.00-000.20): 774            1653825            0.04678%(  0.04678)   0.00087%(  0.00087)
  [000.20-000.40): 353            1653472            0.02133%(  0.06811)   0.00216%(  0.00303)
  [000.40-000.60): 336            1653136            0.02031%(  0.08842)   0.00337%(  0.00640)
         |          |               |                    |        |            |         |
         D1         D2              D3                   D4       D5           D6        D7

where:
Summary:
  S1 - the thread number,
  S2 - the core on which the thread was running.
  S3 - How many samples were collected by that thread
  S4 - Average time per sample [mico-seconds]
  S5 - the fastest sample
  S6 - sample# for fastest sample
  S7 - the longest time gap between samples, the goal of the system tuning is to get this number to the lowest possible number!
  S8 - sample# for longest sample, can detect issues that are common to multiple threads/cores
  S9 - how many cpu cycles the thread was active
  S10, S11 - Start ,end of by cpu cycles

Details:
  The measurements details are printed as historam, empty bins are skipped.
  D1 - Histogram range [micro-seconds]
  D2 - Number of samples in the bin.
  D3 - Number of samples in all bins above this bin, D2[0..N] + D3[N] is the total number of samples
  D4 - Percent of the samples in this bin
  D5 - Percent of samples longer then this bin time range
  D6 - Percent of the measurement time spent by the samples in this bin
  D7 - Percent of the measurement time spent by samples longer then this time range
 

How to use:

Compile:

Run:

Simplest form

hiccups
 - will run on the isolated cpus (isolcpus=.... in kernel command line /proc/cmdline)
 - will run three seconds
 - will report only the minimum,maximum,average per core.


hiccups -t 60 -c 3,4-31 -r 200
 - will run on cpu cores: 3,4-31
 - will run 60 seconds
 - will report histogram with resolution of 200 nano-seconds


Thanks:

 Many thanks to all of those who used earlier versions and provided their feedback

Feedback:

 The hiccups project was written by Erez Strauss <erez@erezstrauss.com>
 Please send feedback and/or requests to the above email with "[hiccups] ..." in the subject.
 It is hosted on github: hithub.com/erez-strauss/hiccups