Table of Contents
Cyclictest
Cyclictest accurately and repeatedly measures the difference between a thread's intended wake-up time and the time at which it actually wakes up from clock_nanosleep in order to provide statistics about the system's latencies. It can measure latencies in real-time systems caused by the hardware, the firmware, and the operating system.
The original test was written by Thomas Gleixner (tglx), but several people have subsequently contributed modifications. Cyclictest is currently maintained by Clark Williams and John Kacur and is part of the test suite rt-tests.
Use Cases
Cyclictest is most commonly used for benchmarking RT systems. It is one of the most frequently used tools for evaluating the relative performance of real-time systems. Some performance tests which use Cyclictest are listed below as well as some other uses for Cyclictest.
- System benchmarking: RTEval, Worst case latency test scenarios
OSADL (Open Source Automation Development Lab) uses Cyclictest to continuously monitor the latencies of several systems. More details about the project and about the specific Cyclictest options they use can be found on the OSADL website.
Technical Explanation
To measure latencies, Cyclictest runs a non real-time master thread (scheduling class SCHED_OTHER) which starts a defined number of measuring threads with a defined real-time priority (scheduling class SCHED_FIFO). The measuring threads are woken up periodically with a defined interval by an expiring timer (cyclic alarm). Subsequently, the difference between the programmed and the effective wake-up time is calculated and handed over to the master thread via shared memory. The master thread tracks the latency values and prints the minimum, maximum, and average latencies.
Installation
Cyclictest can be installed by following the instructions on the rt-tests page.
Execution
Cyclictest must be run as root, with sudo, or as a member of the realtime group. Here is a test case that is relevant for most SMP real-time systems:
# cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
The options above are not relevant for every situation on every system. The Cyclictest Test Design page goes into more detail about how to choose the right options for measuring a specific latency on a given system. Additionally, cyclictest -h provides a quick description of the different program options and more detailed explanations can be found by running man ./src/cyclictest/cyclictest.8 from the root of the rt-tests files.
Running Cyclictest without any options is not recommended as it will not necessarily measure anything useful. Without parameters, Cyclictest creates one thread with an intended wake-up period of 1 ms.
Results
Here is an example of what the Cyclictest results could look like:
T: 0 (821) P: 80 I: 200 C: 518063 Min: 1 Act: 1 Avg: 1 Max: 15 T: 1 (822) P: 80 I: 200 C: 518050 Min: 1 Act: 2 Avg: 1 Max: 23
These results were obtained by running the example command given above in the execution section for a short time on a 2 CPU machine running an RT patched version of the 4.9 kernel. The whitespace in the results above has been changed for formatting reasons.
The table below contains a description of the abbreviated labels in the results:
| abr. | Label | Description |
|---|---|---|
| T | Thread | Thread index and thread ID |
| P | Priority | RT thread priority |
| I | Interval | Intended wake up period for the latency measuring threads (in us) |
| C | Count | Number of times the latency was measured i.e. iteration count |
| Min | Minimum | Minimum latency that was measured (in us) |
| Act | Actual | Latency measured during the latest completed iteration (in us) |
| Max | Maximum | Maximum latency that was measured (in us) |
The default unit used to present the latency values is microseconds (us), but this can be changed to nanoseconds using the –nsecs option.
Histograms
The program options –histogram and –histofall can be used to output a histogram of the latencies to stdout at the end of the test. A more detailed description of these options can be found in the help menu (–help) or in the Cyclictest man page. Alternatively, an OSADL script can be used to run Cyclictest and produce a latency plot with the resulting histogram data. Before running the script, it should be modified so that the options passed to Cyclictest match the intended test case. More details about this can be found below in the Test Design section.
Analysis
The most important value in the results is the maximum detected latency as this value can give an idea of the worst case latency length in the evaluated situation. These results should always be evaluated carefully, as the maximum measured value does not necessarily represent the system's worst case. More details about why the results should be interpreted critically can be found via the Cyclictest Test Design page and in the Limitations section below.
It is not possible to specify a universally applicable recommended limit for latency length. The Cyclictest results should be analyzed based on the system's timing requirements for the latency of what was being measured. For example, if the latency of a specific real-time task was measured, then the results should be compared with the timing requirements for that specific task. The requirements will determine whether the latencies observed in a system are acceptable or too long.
Expected Results
An interesting way of seeing how the tested system compares to other real-time platforms is to run Cyclictest with the same options that are used by OSADL to benchmark systems in their Real-time QA Farm and to then compare the results with the results produced by their tested systems. The exact Cyclictest command used for a system is specified next to that system's latency plot. It is also possible to see a latency plot with the combined results of all the real-time devices that are currently being tested.
It can also be interesting to compare with results from other systems that were running loads, using various Cyclictest parameters, and running different real-time and non real-time kernels. These results give a general idea of the order of magnitude that the latencies should be. They also show the effect that different test conditions and parameters can have on the Cyclictest results.
Test Design
Cyclictest can be used to detect latencies that occur in many different systems and situations. However, a test must be designed carefully because it is relatively easy to not measure the latencies that were intended to be measured. Both the test setup and the parameters passed to Cyclictest need to be carefully chosen as a function of the specific situation in which the latencies are to be evaluated. If these elements are not considered thoughtfully, the Cyclictest results will not accurately represent the latencies that occur in that situation.
Limitations
The latencies measured by Cyclictest may be slightly optimistic depending on the final application that will execute on the system. This is because nanosleep is used by the Cyclictest measuring threads, so they essentially have the shortest possible wake-up time that an RT userspace task can have. When using nanosleep, the Cyclictest thread timer expiry path is executed directly in hard interrupt context with no additional indirection. So, if an application task's wake-up is not completed in hard interrupt context (e.g. threaded interrupt context), the application task will experience a longer wake-up latency than a Cyclictest measuring thread in an identical situation due to the additional level of indirection. In a case like this, the Cyclictest thread's latency measurement will be optimistic even if the latency starts at the exact time that the Cyclictest thread is intended to execute and the thread is affected by the full duration of the latency.
More Information
- get_cyclictest_snapshot – get a snapshot of running instances of cyclictest
