====== Preventing CPU Idling ====== Preventing the CPUs from going into idle can be important when performing certain latency tests in order to have accurate results or to facilitate analysis. This page contains a couple simple ways that a CPU can be prevented from going into idle. ==== Idle task versus idle state ==== Before discussing the methods, it is important to note that there is a difference between a processor that is running the idle task and a processor that is idle. The idle task is the process that is scheduled when a CPU has nothing else to do. There is one idle task pinned to each CPU and it is always ready to run. The idle task usually performs a few operations in preparation for putting the CPU into the idle state and then puts the CPU into idle. A CPU is only considered to be idle once it enters this idle state. In other words, a CPU is not idle when the idle task is running and completing actions in preparation for entering the idle state. The actual implementation of this idle state depends on the system architecture and a number of other things, but in general it tends to be a low power state that the processor enters in order to reduce energy consumption when it has nothing else to do. ===== Userspace program ===== One easy way to make sure that the CPUs on a particular system never go idle is to write a simple program that never blocks, sleeps, or ends and to then run one instance of the program on each CPU. This method prevents the CPUs from going idle by making sure that they always have something to do and thereby preventing the idle task from executing. An advantage of this method is that it is not permanent and can be stopped or started without having to reboot the system. The short program could be an infinite while loop with a simple arithmetic operation such as: int main() { unsigned int tmp; while(1) tmp += 1; return 0; } After compilation, an instance of the program can be pinned to each CPU using the [[realtime:documentation:howto:tools:cpu-partitioning:taskset|taskset]] program. For example, below is a script that will make sure that a system with 4 CPUs will never go idle using a simple program called cpu-no-idle: taskset --cpu-list 0 ./cpu-no-idle & taskset --cpu-list 1 ./cpu-no-idle & taskset --cpu-list 2 ./cpu-no-idle & taskset --cpu-list 3 ./cpu-no-idle & An easy way to check that the CPUs are staying out of idle is to look at a full function trace which was taken after having followed the steps described above. A full function trace can be taken using Ftrace's function tracer. If the idle task never appears in the function trace then the method was used correctly. When the tests have been run, all the relevant data has been collected, and the CPUs can be allowed to idle again, the simple programs can be stopped with the pkill command: $ pkill cpu-no-idle ===== Kernel command-line parameter ===== Another way to make sure that the CPUs never go idle is to change the value of a [[https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html|kernel command-line parameter]]. A couple of options of parameters that can be used to prevent deeper CPU sleep states are presented below. These methods are permanent for the duration of the system's execution. So, to allow the system to resume its original behavior the parameter must be modified and the system must be restarted. Two methods for changing parameters are presented, a method where the new kernel parameter value is permanent until explicitly changed and a method where the kernel parameter applies for a single execution of the kernel. The idle parameter is used as an example to aid in the explanation of these methods, but the techniques below apply for any kernel command-line parameter. ==== idle parameter ==== A parameter that can be used to prevent a system's CPUs from idling is the boot parameter "idle". Changing this parameter does not prevent the idle task from executing, but it prevents the idle task from putting the CPUs in a low power idle state. To prevent the CPUs from going idle using the technique, the option idle=poll must be added to the command-line parameters passed to the kernel when it is executed on startup. This option forces the processor to poll when it has nothing to do instead of going into a low power idle state. More details about the "idle" option can be found in the Linux documentation about [[https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html|kernel parameters]]. ==== max_cstate parameters ==== One common motivation for wanting to prevent CPU idling is avoiding gaps in kernel function traces that can be mistaken for unusual firmware or hardware caused latencies. The max_cstate parameters //cannot be used to prevent CPU idling//, but they can be used to prevent gaps in function traces. A max_cstate kernel parameter can be used to limit the maximum C-State. Limiting the maximum C-State to 1 will prevent having noticeable gaps in full function traces. Briefly, C-States are different power saving idle states. C-States 1 and up are progressively deeper processor sleep states. C-State zero represents a processor that is not idle. C-State 1 is an idle state, but it is a very "shallow" idle state so it takes almost no time for the processor to enter and exit C-State 1 (around 1 us). This means that going in and out of C-State 1 will not create noticeable gaps in a trace. There are two max_cstate boot parameters that can be used for this, processor.max_cstate (which limits the maximum C-State for the acpi_idle driver) and intel_idle.max_cstate (which limits the maximum C-state for the intel_idle driver). The value of one of these parameters must be set to 1 (e.g. intel_idle.max_cstate=1). Only one of these boot parameters needs to be specified in the command-line depending on which idle driver is used by the kernel. The idle driver that is currently loaded can be checked via sysfs: # cat /sys/devices/system/cpu/cpuidle/current_driver However, there is also no harm in covering all possibilities and specifying a limit for both intel_idle and acpi_idle. Using a max_cstate parameter to prevent trace gaps is better than using the idle parameter because allowing the CPUs to enter C-State 1 will save power compared to forcing them to poll when they have nothing to do. ==== Change parameter permanently ==== To make the option changes persistent when the system is restarted, add the parameter and its desired value (e.g. idle=poll) to the contents of the GRUB_CMDLINE_LINUX field in the /etc/default/grub file. The user must be root or use sudo to modify this file. Here is an example of what the line in the file could look like after being modified: GRUB_CMDLINE_LINUX="idle=poll" After changing the file, run the command below to propagate the changes. Make sure to be root or use sudo before running the following: # update-grub ==== Change parameter for a single execution ==== To change the option for a single execution of the kernel, the GRUB options can be temporarily modified on startup. This method can be used by unprivileged users as long as the options in GRUB are not protected. To make these changes, boot the system and select the advanced options in the GRUB menu. Then, navigate to the kernel version that is to be executed and press 'e' to edit the commands used to start it. In the script that appears, there should be a line that looks similar to this: linux /boot/vmlinuz-4.15.0 root=UUID=cf89c5d8-e39f-a8c9-k45j2e85de2u ro quiet To make the temporary change, add the kernel parameter and its value (e.g. idle=poll) to the end of the line so that it looks like this: linux /boot/vmlinuz-4.15.0 root=UUID=cf89c5d8-e39f-a8c9-k45j2e85de2u ro quiet idle=poll and then boot the kernel. ==== Checking boot parameters ==== For either the persistent of the single execution method, to verify that the kernel booted with the modified "idle" option check the contents of /proc/cmdline with: $ cat /proc/cmdline and make sure that the kernel parameter and its value (e.g. idle=poll) is somewhere in the output.