Jonathan Corbet describes Anna-Maria Behnsen's work on better CPU selection for timer expiration https://lwn.net/Articles/913568/.

Jake Edge summarizes the discussion around printk() in a LWN.net article https://lwn.net/Articles/909980/. It is the last core piece needing changes before the RT_PREEMPT patches can be fully merged.

As there has not been a recent blog post in the last ten months, one might believe the real-time Linux project is not progressing any further, but quite the opposite is true.

So, let's briefly point out the recent news and developments:

• In April 2021, Thomas Gleixner has given an interesting interview on the history, current state and challenges of real-time Linux. The interview clearly shows that the technical challenges are well understood and can be resolved with the community. The actual challenge for the success of progressing real-time Linux is the mismatch of seeming relevance and reliance of various industry partners on the real-time functionality to the active contributions and support for the needed development and maintenance activities. Thomas Gleixner points out that the “Tragedy of the Commons” situation, with each individual company thinking their own minor contribution might not make a difference, can lead to a situation that makes the whole industry suffer the self-created lack of quality assurance and maintenance for real-time Linux in the future. He made very clear that the decisions made now will set the path to the future. Even though a lot of effort and success story surround the technical work on real-time Linux, the world can continue with a future without real-time Linux; it will be just devastating to those stakeholders that never concluded to ever support something they so centrally rely on in their business. Read more at https://www.linux.com/news/in-the-trenches-with-thomas-gleixner-real-time-linux-kernel-patch-set/.
• In May 2021, Thomas Gleixner presented “A Guided Tour Through the PREEMPT_RT castle.” at the ELISA (Enabling Linux In Safety Applications) Workshop. Read more and tune into the presentation at https://elisa.tech/blog/2021/08/25/a-guided-tour-through-the-preempt-rt-castle/.
• In August 2021, a large core part of the out-of-tree PREEMPT_RT patch set, the PREEMPT_RT locking core, has been finally merged. As always, Jonathan Corbet briefly mentions it in two LWN.net articles, https://lwn.net/Articles/866112/ and https://lwn.net/Articles/867821/. For the ones interested in digging into more details, the git merge commit may serve as a good entry point: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e5e726f7bb9f. With that part merged, mainlining the PREEMPT_RT is not yet finished, but did a big step forward.
• In September 2021, the annual Real-time Microconference has taken place as part of the Linux Plumbers Conference. The real-time Linux developers have been talking about current and future development. See more at https://linuxplumbersconf.org/event/11/sessions/108/#20210921.

As the real-time endgame, i.e., the stage of the game when few pieces are left on the board, continues advancing, Jonathan Corbet summarizes the development work on mainlining migration disabling in the scheduler. It nicely sketches the well-known tug of war between optimizing for minimal latencies and maximal throughput and how the kernel community iteratively works towards a solution that allows peaceful coexistence of both worlds.

And this is just one of many moves happening quickly now in this endgame, Jonathan Corbet has also reported on other activities related to the real-time work in the core kernel in further articles Atomic kmaps become local and Four short stories about preempt_count.

Our real-time application developers also get a nice introduction from John Ogness to make full proper use in their applications of the real-time kernel with the Embedded Linux Conference Europe (ELCE) 2020 presentation A Checklist for Writing Linux Real-Time Applications. Marta Rybczyńska has also summarized this great presentation in yet another LWN.net article A realtime developer's checklist.

Jake Edge summarizes the discussion around planning maintenance of the real-time work once the PREEMPT_RT branch is fully merged in mainline in the LWN.net article https://lwn.net/Articles/830660/. This already gives an interesting view of the work ahead when the first goal, the out-of-tree branch is fully merged, has been reached.

Marta Rybczyńska summarizes the Real-time Linux Project's work on making the locking APIs carry more semantic information, and hence allow the underlying kernel execution to be safe when being more flexible, e.g., allow more kinds of preemption, in the LWN.net article Local locks in the kernel. This article shows nicely the needs and challenges on reworking the kernel's core API “just to make Linux real-time, but done right”. Initially motivated by the needs for real-time systems, the locking API in the kernel has improved in clarity for the overall kernel community, i.e., all kernel developers and users.

Linus Torvalds, the Linux Kernel top level maintainer, merged a pull request adding the kernel configuration symbol CONFIG_PREEMPT_RT. The configuration symbol is currently hidden as it depends on a not yet enabled option which means it has absolutely no functional technical value at the moment.

The value is elsewhere. As stated in the pull request mail, the ongoing effort of merging the real-time preemption patch set into the mainline kernel ran into a hen and egg problem. While the preparatory work which was conducted in the past four years was mostly undisputed because it provided benefit to the mainline kernel independent of real-time support, the remaining bits and pieces are of different nature. They require rework of code which is from a mainline perspective perfectly fine. While upstream maintainers are generally very supportive towards the real-time efforts, they rightfully started to ask the question why these changes should be merged if there is no clear indication that the real-time preemption patch set will be fully integrated into the mainline kernel.

Adding CONFIG_PREEMPT_RT as a for now hidden configuration symbol addresses exactly this problem. It's a clear sign that real-time will be a first class citizen of the mainline kernel. That answers the question of maintainers and opens the flood gates for the outstanding (hard)core real-time bits. Of course this is not a free pass for forcing the missing bits and pieces into the mainline kernel. No, the real-time developer team will continue the careful and responsible integration as it has done for the past 15 years.

Quite some work ahead, but with a bit of luck most of the important changes could be queued in maintainer trees by mid of September and be ready to be merged into the mainline kernel during the Linux v5.4 merge window. Accidentally that will be almost exactly 15 years after the big real-time flame wars took place on the Linux Kernel Mailing List and 20 years after real-time project lead Thomas Gleixner started to investigate the options for Linux and real-time.

The real-time developer team wants to take this opportunity to express gratitude for the support and the trust we received from the members of the Real-time Linux collaborative project and the Linux Foundation. We are excited that the hard work of the past years brought us to the point where the most interesting chapter of the long and thrilling story of Real-time Linux will be written.

Jake Edge summarizes John Ogness work on redesigning printk() in the LWN.net article Reimplementing printk().

While at first, it might seem that printk() and real-time systems are unrelated, it turns out that to guarantee low latencies, all parts of the kernel need to become interruptible, and for printk(), the best way to do so is to make the design as lockless, interruptible and concurrent as possible. Once John Ogness has got his reimplementation upstream, the PREEMPT_RT patch set will decrease by a few earlier work-around patches on printk. Hence, we are getting another step closer down to the core real-time parts and the final patch providing the ultimate `real-time` config switch.

Jonathan Corbet reports on Frederic Weisbecker's softirq rework to reduce softirq latency in the LWN.net article Per-vector software-interrupt masking.

This work is yet another step to make the kernel more real-time capable, and doing it in a manner that it generally improves the softirq design in the kernel. Merging PREEMPT_RT mainline is all about refactoring various kernel functionalities to be properly and cleanly designed with respect to minimizing latencies. This means to minimize sections of non-interruptible executions, defer work to interruptible tasks and reduce dependencies between the executions of multiple kernel threads. The softirq rework is one step on the Linux Real-time Project's trail to mainline.

The goal of getting the remaining bits and pieces of Preempt-RT merged into the 2018 LTS kernel has always been ambitious, but we were working hard on making it. The assumption that 4.20 will be the next LTS kernel turned out to be wrong, which clearly made us miss the train.

Though blaming everything on the decision to select 4.19 as the next LTS kernel would be dishonest at best. There are a lot of other factors which led to this.

• The last year, since the speculation vulnerabilities started to surface, has been tough for a lot of key developers and maintainers in the kernel community and occupied many of them for a long period of time. The complexity of these vulnerabilities and the fact, that these issues forced us to destroy performance optimizations which we carefully built over time based on the assumption that the hardware holds up to the promises of protection domains, caused a lot of disruption not only for the RT development. Review and feedback cycles got significantly longer which caused delays in the range of months.
• Several tasks which were thought of as easy to solve turned into rather large work packages due to upstream changes and upstream maintainer feedback, refactoring the USB completion callback being only one of the examples.
• The time spent on tracking down subtle and hard to debug issues increased significantly in the last year. Most of these issue turned out to be upstream problems, which were exposed by RT. The recent analysis of a nasty cache line starvation problem accumulated to more than a month of work and it's still not fully resolved.

Having missed the LTS goal, the project members agreed thankfully that rushing everything into 4.20 just to be able to tick off a box in a spread sheet, does not make sense.

There has been a substantial amount of overtime accumulated in the RT team, so relaxing the time line a bit and giving the people their well deserved recreation time is surely better than rushing things for no real value.

There needs to be quite some effort made to switch over from the half stabilized 4.18-RT to 4.19-RT and it surely needs a lot of thought to make the LTS support for 4.19-RT as smooth as possible by aligning it as much as we can foresee to the final state of RT.

This also allows to take a step back and revisit some of the design decisions and we are looking forward to RT-Summit and the LPC RT microconference for fruitful technical discussions.

As always I want to take the opportunity to express my gratitude for the support of this effort by the RT project members and their understanding for unexpected roadblocks.

This blog post is written by Lukas Bulwahn from BMW Car IT GmbH, which is one of the Gold members in the Real-time Linux Project. The Real-time Linux Project encourages its members to report about its uses of the PREEMPT_RT patch and spread the word about the Real-time Linux Project:

Long ago in 2009, a small team of kernel developers had finished consolidating previous prototypic developments to make Linux real-time capable into a single out-of-tree patch set, called the PREEMPT_RT patch set. This patch set can be applied to turn a vanilla main-line Linux kernel without real-time capabilities into a real-time capable Linux kernel. Many companies use this patch set to build various industrial systems that required to implement hard real-time properties at comparatively relaxed time bounds of about one millisecond precision.

BMW Car IT also used to this patch set to build real-time capable prototypes for complex functions in the area of autonomous driving. However, from the beginning with the development of those prototypes, it was clear that any product with high quality demands requires to get the PREEMPT_RT patch set in the main-line development for increased compatibility of features, stronger quality assurance and reduced maintenance. Hence, BMW Car IT started driving efforts to make Linux real-time capable in 2014.

First, BMW Car IT joined OSADL, the Open-Source Automation Development Lab, as Gold member to support the real-time Linux development activities, which is collaboratively funded by the OSADL member at that time.

Second, our former colleague Daniel Wagner started to get acquainted with the existing PREEMPT_RT patch in 2014 and made a number of contributions to the Linux kernel related to real-time capabilities from 2015 until end of 2016. Due to his experience with the PREEMPT_RT patch, he is now the maintainer of the Linux 4.4 real-time stable branch, and one of the three maintainers for the real-time stable patch branches.

Since 2016, the Real-time Linux project is a collaborative project under the umbrella of the Linux Foundation. The project's goal is to make the main-line Linux real-time capable. The project ensures that the Linux kernel developers have the ability to continue development work, long-term support and future research for a real-time-capable Linux.

In the last two years, 2016 and 2017, the Real-time Linux development team rewrote the CPU hotplug infrastructure and refactored the timer wheel and high-resolution timers. This already reduced the out-of-tree PREEMPT_RT patch set significantly.

Due to a funding decrease that became apparent at the beginning of 2018, the development in the Real-time Linux project would have reduced its work force. Fortunately, Intel and BMW Car IT could close this funding gap. Intel increased their membership from Gold to Platinum and BMW Car IT joined Linux Foundation and the collaborative project as Gold member in the Real-time Linux Project. So now after those project adjustments, the Real-time Linux Project team is back on track and continues to prepare the remaining patches for inclusion into the main-line development with full speed.

In 2018, the Real-time Linux kernel team will be refactoring, rewriting and generally improving the printk und soft interrupt infrastructure and other smaller other parts. This work will prepare the Linux kernel source code so that all further real-time specific changes will smoothly be merged into the mainline kernel.

The real-time functionality touches the core kernel parts, i.e., it requires significant changes in timers, schedulers, locking mechanisms, interrupt handling and more, and it also is a cross-cutting concern for all drivers, i.e., every driver has to follow a certain discipline to make the overall kernel real-time capable. Hence, it is difficult to predict the exact date when the Real-time Linux Project will finally have all its patches merged into the main-line development. However, there is no doubt that the Linux kernel will eventually become real-time capable.

“The Linux kernel is a software development project of huge invest to us. Obviously, BMW Car IT has a high interest of making best possible use of this software asset. The automotive industry has particular requirements, such as higher real-time requirements and the need for longer maintenance periods, than the general IT and consumer electronics industry. With our investments in initiatives addressing these requirements, we can ensure that Linux fits to our needs”, says Kai-Uwe Balszuweit, the CEO of BMW Car IT.

Once the real-time capabilities have been integrated in the main-line development, the project work is of course not just finished and the Real-time Linux project cannot just be abandoned. After the final integration into the main-line development, the development activities will slowly shift its focus:

The core system will not require further changes for the real-time capability, but the Real-Time Linux development team will need to review, test and adjust new incoming features from other kernel development teams to keep the kernel real-time capable when these new features are included.

Furthermore, the already existing real-time stable trees must be further continued to be maintained until the end of life of the corresponding kernel LTS version, so commonly two years for most LTS versions, but possibly even longer. Slowly over the years, the real-time stable trees for older kernel versions will reach their end of life, while for younger LTS kernel versions, which have the real-time capabilities fully included, have no need to maintain a separate real-time stable branch. This will decrease the working effort on the current real-time stable maintainers and they can focus their work to assist in the quality assurance of the continuous main-line development.

Of course, all users and stakeholders of the real-time capability must continue to support all these activities over the next years.

This is well understood at BMW Car IT, and we expect that other companies that require the real-time capability in Linux will also follow and express this general common understanding. Beyond software development until start of production, operations and maintenance is an important software development activity that is not underestimated at BMW Car IT.

Christian Salzmann, the CEO of BMW Car IT, states it clearly: “Providing good software solutions to BMW for many years, BMW Car IT knows that continuous operations and maintenance is one of the major cornerstones for providing a great experience to our customers. The continuous activity of development and operations of software going hand-in-hand, in short DevOps, is part of BMW Car IT's company mindset. BMW Car IT's support for further development and operations in the Real-time Linux Project is no exception to this rule.”

The blog post has been acknowledged by the Real-time Linux Project for its technical soundness; the views and opinions in this article are those of the author and member company and do not necessarily reflect the official position of the Real-time Linux Project.

Linux became a research vehicle very early on. Real-Time researchers set out to transform Linux to a Real-Time Operating system. These efforts were following two different approaches:

• Dual kernel approach: Utilizes a Nano- or a Micro-Kernel as RTOS kernel which runs the Linux Kernel in its idle task. The most prominent variants were: RT/Linux, RTAI, DROPS and later on Xenomai.

• In kernel real-time: Modifies the kernel code itself to gain Real-Time OS properties. The well known variants are KURT and Linux/RK

None of these projects was ever seriously attempting to be merged into the mainline kernel. RT/Linux, RTAI, Xenomai and Linux/RK were also deployed commercially with varying success and sustainability.

In autumn 2004 various patch series were posted from different interested parties, which provided basic technology to transform the Linux kernel into a Real-Time Operating System. None of these patches was providing a full solution.

These postings resulted in one of the largest mail discussions on LKML accompanied by flamewars and more or less pointless competitions to gain single digit microseconds in certain benchmarks.

Quite surprising for most kernel developers, Ingo Molnar, at the time maintainer of the kernel scheduler, picked up some of the patches or rewrote them mostly from scratch and combined them to a consistent set of patches aiming for in full kernel Real-Time support. Within no time a small team of core developers formed. Thomas Gleixner, who was working with the late Douglas Niehaus from Kansas University on a 2.6 based variant of the KURT realtime extension, brought in patches and technology from KURT. Steven Rostedt who was familiar with the Linux/RK work, brought in expertise and ideas from there. Various other developers joined the effort temporarily and helped to create the first halfways usable RT version in a relatively short time frame. The developers were aiming to integrate the RT work in the upstream kernel from the very beginning.

At the Linux kernel summit 2005 the general approach of integrating RT into the upstream kernel was discussed and most developers agreed with it under the premise that it is not disrupting the existing work and does not create road blocks for future development.

In the first years the RT work was mostly conducted by developers employed by RedHat and IBM due to a Raytheon/US Navy contract which was based on Linux Real-Time. After the contract was fulfilled, the interest in RT from IBM and RedHat subsided and the involved developers moved on to other areas of work.

During the first five years substantial parts of the Preempt-RT work found their way into the Linux Kernel:

• Generic Timekeeping
• High resolution timers
• Mutex infrastructure
• Generic interrupt handling infrastructure
• Priority inheritance for user space mutexes
• Preemptible and hierarchical RCU
• Threaded interrupt handlers
• Tracing infrastructure
• Lock dependency validator
• …

With some part-time funding from RedHat Preempt-RT was maintained and kept alive mostly by Thomas Gleixner and his team at Linutronix. The available funding for RT was not even sufficient to sustain maintenance. Despite substantial effort the expertise around Preempt-RT did not result in sustainable commercial success.

During this time Thomas Gleixner started to warn about the non sustainability of Preempt-RT at conferences and in personal conversations. Aside of hilarious proposals and whining nothing useful happened.

After RedHat cut the RT funding completely, Gleixner and his company linutronix decided to stop their engagement in Preempt-RT and Gleixner declared at ELCE 2014 in Dusseldorf that Preempt-RT is now a hobbyist project.

Shortly after the conference OSADL stepped up and provided full time funding for two engineers to keep Preempt-RT at least maintained and forwarded to newer kernel versions in 2015.

Preempt-RT maintenance work was resumed, but Gleixner made it entirely clear that if there would be no way forward towards proper mainline integration by the end of the year he'd be not interested to maintain RT out of tree forever.

In 2015, the Linux Foundation started an effort, led by Kate Stewart, to create a Real-Time Linux Collaborative Project under the umbrella of the Linux Foundation. At ELCE 2015 in Dublin, the project was officially announced and the kick-off meeting took place. The funding allows five engineers to work full time on the project with the clear target of merging the Preempt-RT patches into the mainline kernel.

In the first two years hundreds of patches emerging from the Preempt-RT team around Thomas Gleixner at Linutronix were integrated into the Linux kernel. Most of this work was preparatory and refactoring existing upstream code in a way to make the integration of Preempt-RT easier. The main achievements were:

• Rewrite of the CPU hotplug infrastructure
• Refactoring of the timer wheel
• Refactoring of high resolution timers
• Various other bits and pieces

While this work makes RT integration and maintenance easier it is not solely RT specific and it is valuable for the quality and performance of the Linux kernel on its own.

This effort reduced the size of the RT patch set and led to the more interesting and hard to solve RT specific problems. While the scope of these pending changes is not as huge as the already merged work, it's more difficult to integrate it cleanly and justify the merge into the upstream kernel.

The Meltdown/Spectre security disaster slowed development down a bit for about two months at the end of 2017 and early 2018 as Preempt-RT team lead Thomas Gleixner was heavily involved in the development of the mitigations for the Linux kernel. The RT team is working hard to make up for the time lost and come back on the estimated project time line.

As of March 2018 the project is in an interesting situation. The funding for 2018 is not clear yet, which means that eventually work force has to be reduced and the expected time lines have to be adapted. There is lots of effort going on to remedy that situation, so it hopefully will get back on track soon.

The RT team is really grateful for the support we received so far from the project members and from the Linux Foundation. We are aware that the project members take responsibility for the RT using industry as a whole and it would be certainly a good chance now for bystanders to join the effort and help along to secure the technology on which their products depend on. RT is one of the most ambitious Linux technology projects and to some extent the most disruptive technology which aimed for mainline integration. So far we succeeded to keep the impact on mainline at a minimum and we hope that with funding secured for the time after mainline integration this can be kept up to everyone's satisfaction: RT users, users of regular Linux and the upstream Linux maintainers and developers.

As most of you know 2018 has started with a big bang in the computer industry, which will keep us occupied for quite some time. The upstream kernel mitigation of meltdown variant 3 via https://lwn.net/Articles/742693/PTI kept me beyond busy for the last two month.

Despite that disruption, which definitely required all brains collaborating, I've tried to keep up with the ongoing work of merging the Preempt-RT patch set into the Linux kernel. We've made great progress on various topics, especially the jinxed dcache problem. Patches will be posted as soon as folks are back from their well earned vacation.

In 2017 the RTL developer team was making enormous progress and we've upstreamed an impressive 1.5 patches a day on average. The deviation from the time line which we predicted end of 2015 is still on a relatively small margin of less than 20 percent.

We'd like to express our gratitude to the RTL project Members who provided the funding for the past two years, so we were able to work with five engineers full time on interesting, challenging and sometimes mind boggling problems.

As things stand for 2018 the funding will be reduced, so we expect to reduce the number of developers accordingly to something between three and four. That will have significant impact on the time line and postpone the final merge of RT by a not yet determined amount of time. I can't tell how much at the moment as this would be pure speculation (pun intended).

Nevertheless, we're starting into 2018 with vigor and the vision of making as much progress towards mainlining RT as possible.

Thomas & RTL developer team