The rtla – Real-time Linux Analysis toolset analyzes real-time properties of the Linux kernel.

The tools has several modes. The default top mode displays periodic summary from a tracer. The hist mode displays histogram at the end of operation. Automatic mode (--auto) stops when the threshold is reached.

The tool offers several commands:

• rtla timerlat measures IRQ and kernel thread timer latency with timerlat tracer. Modes: top, hist.

• rtla osnoise provides information about operating system noise measured with osnoise tracer. It dispatches kernel threads per CPU to measure various sources of interference during execution, such as preemption, softirq, and IRQs. Modes: top, hist.

• rtla hwnoise detects and quantifies hardware-related noise with osnoise tracer. By disabling interrupts and scheduling of threads, only non-maskable interrupts and hardware-related noise are analyzed.

Each of these commands has various options for configuration, such as setting tracer periods, runtimes, thresholds, scheduling parameters, trace events and triggers, and more. These commands offer different ways to analyze and understand the real-time properties and noise sources in the Linux operating system.

Arguments to enable common events for automatic and trace options:

-e sched:sched_switch -e sched:sched_wakeup -e sched:sched_migrate_task -e irq -e irq_vectors -e timer -e workqueue

To see all enabled events, run:

grep 1 /sys/kernel/debug/tracing/instances/ /sys/kernel/debug/tracing/events/ -R --include enable

RT tools and utilities
Linux scheduling latency debug and analysis with RTLA
🎥 rtla timerlat: Debugging Real-time Linux Scheduling Latency, video
user mode source
trace_osnoise.c - kernel mode source