Differences

This shows you the differences between two versions of the page.

--- realtime:documentation:howto:tools:cpu-partitioning:start [2024/05/18 21:02]
alison [IRQ affinity]
+++ realtime:documentation:howto:tools:cpu-partitioning:start [2024/05/27 01:18]
alison [Realtime application best practices]
@@ Line 78: / Line 78: @@
   * **Tools**: [[realtime:documentation:howto:tools:cpu-partitioning:irqbalanced|irqbalanced]]
-  * *Tools**: [[realtime:documentation:howto:tools:cpu-partitioning:taskset|taskset]]
+  * **Tools**: [[realtime:documentation:howto:tools:cpu-partitioning:taskset|taskset]]
+==== Housekeeping cores ====
-===== Realtime application example =====
+A common paradigm with realtime systems is to pin latency-insensitive kernel and userspace tasks tasks on a designated "housekeeping" core.   For example, //taskset// can pin kernel threads like //kswapd// and //kauditd//.  Applications whose network traffic latency is not critical may wish to pin network IRQs there as well.   Userspace threads which are sometimes CPU-intensive like systemd and rsyslog may also be pinned on the housekeeping core.  Pinning userspace threads will not have the desired effect if much of their work is performed by [[https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cpu-partitioning/start#cpu_affinity_and_kworkers|unbound workqueues]], which may migrate to any core.
-FIXME and fill it with content.
+==== Softirqs and kthreads ====
+Softirqs are kernel threads which are often challenging to manage on realtime systems.  Softirqs may run in atomic context immediately following a hard IRQ which "raises" them, or they may be executed in process context by per-CPU kernel threads called //ksoftirqd/n//, where //n// is the core number.   There are 10 kinds of softirqs which perform diverse tasks for the networking, block, scheduling, timer and [[https://wiki.linuxfoundation.org/realtime/documentation/technical_details/rcu|RCU]] subsystems as well as executing callbacks for a large number of device drivers via the tasklet mechanism.   Only one softirq of any kind may be active at any given time on a core.   Thus if ksoftirqd is preempted by a hard IRQ, the associated soft interrupt is disabled from following it immediately, and must wait for ksoftirqd.   This unfortunate situation has been called "the new Big Kernel Lock" by realtime Linux maintainers.
+Kernel configuration allows system managers to move the NET_RX and RCU callbacks out of softirqs and into their own kthreads.  Since kernel 5.12, moving the NET_RX into its own kthread is possible by //echo//-ing '1' into the //threaded// sysfs attribute associated with a network device.  The process table will afterwards include a new kthread called //napi/xxx//, where xxx is the interface name. [Read more about the [[https://wiki.linuxfoundation.org/networking/napi?s[]=napi|NAPI]] mechanism in the networking wiki.]  Userspace may employ //taskset// to pin this kthread on any core.   Moving the softirq into its own kthread incurs a context-switch penalty, but even so may be worthwhile on systems where bursts of network traffic unacceptably delay applications.   [[https://wiki.linuxfoundation.org/realtime/documentation/technical_details/rcu?s[]=rcu#rcu_callback_offloading|RCU Callback Offloading]] produces a new set of kthreads, and can be accomplished via a combination of compile-time configuration with boot-time command-line parameters.
+===== Realtime application best practices =====
+Multithreaded applications which rely on glibc's libpthread are prone to unexpected latency delays since pthread condition variables do not honor priority inheritance ([[https://sourceware.org/bugzilla/show_bug.cgi?id=11588|bugzilla]]). [[https://github.com/dvhart/librtpi|librtpi]] is an alternative LGPL-licensed pthread implementation which supports priority inheritance, and whose API as close to glibc's as possible.  The alternative [[https://www.musl-libc.org/|MUSL libc's pthread implementation]] appears to be similar to glibc's.

Wiki

User Tools

Site Tools

Differences

Page Tools