This blog post is written by Lukas Bulwahn from BMW Car IT GmbH, which is one of the Gold members in the Real-time Linux Project. The Real-time Linux Project encourages its members to report about its uses of the PREEMPT_RT patch and spread the word about the Real-time Linux Project:

Long ago in 2009, a small team of kernel developers had finished consolidating previous prototypic developments to make Linux real-time capable into a single out-of-tree patch set, called the PREEMPT_RT patch set. This patch set can be applied to turn a vanilla main-line Linux kernel without real-time capabilities into a real-time capable Linux kernel. Many companies use this patch set to build various industrial systems that required to implement hard real-time properties at comparatively relaxed time bounds of about one millisecond precision.

BMW Car IT also used to this patch set to build real-time capable prototypes for complex functions in the area of autonomous driving. However, from the beginning with the development of those prototypes, it was clear that any product with high quality demands requires to get the PREEMPT_RT patch set in the main-line development for increased compatibility of features, stronger quality assurance and reduced maintenance. Hence, BMW Car IT started driving efforts to make Linux real-time capable in 2014.

First, BMW Car IT joined OSADL, the Open-Source Automation Development Lab, as Gold member to support the real-time Linux development activities, which is collaboratively funded by the OSADL member at that time.

Second, our former colleague Daniel Wagner started to get acquainted with the existing PREEMPT_RT patch in 2014 and made a number of contributions to the Linux kernel related to real-time capabilities from 2015 until end of 2016. Due to his experience with the PREEMPT_RT patch, he is now the maintainer of the Linux 4.4 real-time stable branch, and one of the three maintainers for the real-time stable patch branches.

Since 2016, the Real-time Linux project is a collaborative project under the umbrella of the Linux Foundation. The project's goal is to make the main-line Linux real-time capable. The project ensures that the Linux kernel developers have the ability to continue development work, long-term support and future research for a real-time-capable Linux.

In the last two years, 2016 and 2017, the Real-time Linux development team rewrote the CPU hotplug infrastructure and refactored the timer wheel and high-resolution timers. This already reduced the out-of-tree PREEMPT_RT patch set significantly.

Due to a funding decrease that became apparent at the beginning of 2018, the development in the Real-time Linux project would have reduced its work force. Fortunately, Intel and BMW Car IT could close this funding gap. Intel increased their membership from Gold to Platinum and BMW Car IT joined Linux Foundation and the collaborative project as Gold member in the Real-time Linux Project. So now after those project adjustments, the Real-time Linux Project team is back on track and continues to prepare the remaining patches for inclusion into the main-line development with full speed.

In 2018, the Real-time Linux kernel team will be refactoring, rewriting and generally improving the printk und soft interrupt infrastructure and other smaller other parts. This work will prepare the Linux kernel source code so that all further real-time specific changes will smoothly be merged into the mainline kernel.

The real-time functionality touches the core kernel parts, i.e., it requires significant changes in timers, schedulers, locking mechanisms, interrupt handling and more, and it also is a cross-cutting concern for all drivers, i.e., every driver has to follow a certain discipline to make the overall kernel real-time capable. Hence, it is difficult to predict the exact date when the Real-time Linux Project will finally have all its patches merged into the main-line development. However, there is no doubt that the Linux kernel will eventually become real-time capable.

“The Linux kernel is a software development project of huge invest to us. Obviously, BMW Car IT has a high interest of making best possible use of this software asset. The automotive industry has particular requirements, such as higher real-time requirements and the need for longer maintenance periods, than the general IT and consumer electronics industry. With our investments in initiatives addressing these requirements, we can ensure that Linux fits to our needs”, says Kai-Uwe Balszuweit, the CEO of BMW Car IT.

Once the real-time capabilities have been integrated in the main-line development, the project work is of course not just finished and the Real-time Linux project cannot just be abandoned. After the final integration into the main-line development, the development activities will slowly shift its focus:

The core system will not require further changes for the real-time capability, but the Real-Time Linux development team will need to review, test and adjust new incoming features from other kernel development teams to keep the kernel real-time capable when these new features are included.

Furthermore, the already existing real-time stable trees must be further continued to be maintained until the end of life of the corresponding kernel LTS version, so commonly two years for most LTS versions, but possibly even longer. Slowly over the years, the real-time stable trees for older kernel versions will reach their end of life, while for younger LTS kernel versions, which have the real-time capabilities fully included, have no need to maintain a separate real-time stable branch. This will decrease the working effort on the current real-time stable maintainers and they can focus their work to assist in the quality assurance of the continuous main-line development.

Of course, all users and stakeholders of the real-time capability must continue to support all these activities over the next years.

This is well understood at BMW Car IT, and we expect that other companies that require the real-time capability in Linux will also follow and express this general common understanding. Beyond software development until start of production, operations and maintenance is an important software development activity that is not underestimated at BMW Car IT.

Christian Salzmann, the CEO of BMW Car IT, states it clearly: “Providing good software solutions to BMW for many years, BMW Car IT knows that continuous operations and maintenance is one of the major cornerstones for providing a great experience to our customers. The continuous activity of development and operations of software going hand-in-hand, in short DevOps, is part of BMW Car IT's company mindset. BMW Car IT's support for further development and operations in the Real-time Linux Project is no exception to this rule.”

The blog post has been acknowledged by the Real-time Linux Project for its technical soundness; the views and opinions in this article are those of the author and member company and do not necessarily reflect the official position of the Real-time Linux Project.

Linux became a research vehicle very early on. Real-Time researchers set out to transform Linux to a Real-Time Operating system. These efforts were following two different approaches:

• Dual kernel approach: Utilizes a Nano- or a Micro-Kernel as RTOS kernel which runs the Linux Kernel in its idle task. The most prominent variants were: RT/Linux, RTAI, DROPS and later on Xenomai.

• In kernel real-time: Modifies the kernel code itself to gain Real-Time OS properties. The well known variants are KURT and Linux/RK

None of these projects was ever seriously attempting to be merged into the mainline kernel. RT/Linux, RTAI, Xenomai and Linux/RK were also deployed commercially with varying success and sustainability.

In autumn 2004 various patch series were posted from different interested parties, which provided basic technology to transform the Linux kernel into a Real-Time Operating System. None of these patches was providing a full solution.

These postings resulted in one of the largest mail discussions on LKML accompanied by flamewars and more or less pointless competitions to gain single digit microseconds in certain benchmarks.

Quite surprising for most kernel developers, Ingo Molnar, at the time maintainer of the kernel scheduler, picked up some of the patches or rewrote them mostly from scratch and combined them to a consistent set of patches aiming for in full kernel Real-Time support. Within no time a small team of core developers formed. Thomas Gleixner, who was working with the late Douglas Niehaus from Kansas University on a 2.6 based variant of the KURT realtime extension, brought in patches and technology from KURT. Steven Rostedt who was familiar with the Linux/RK work, brought in expertise and ideas from there. Various other developers joined the effort temporarily and helped to create the first halfways usable RT version in a relatively short time frame. The developers were aiming to integrate the RT work in the upstream kernel from the very beginning.

At the Linux kernel summit 2005 the general approach of integrating RT into the upstream kernel was discussed and most developers agreed with it under the premise that it is not disrupting the existing work and does not create road blocks for future development.

In the first years the RT work was mostly conducted by developers employed by RedHat and IBM due to a Raytheon/US Navy contract which was based on Linux Real-Time. After the contract was fulfilled, the interest in RT from IBM and RedHat subsided and the involved developers moved on to other areas of work.

During the first five years substantial parts of the Preempt-RT work found their way into the Linux Kernel:

• Generic Timekeeping
• High resolution timers
• Mutex infrastructure
• Generic interrupt handling infrastructure
• Priority inheritance for user space mutexes
• Preemptible and hierarchical RCU
• Threaded interrupt handlers
• Tracing infrastructure
• Lock dependency validator
• …

With some part-time funding from RedHat Preempt-RT was maintained and kept alive mostly by Thomas Gleixner and his team at Linutronix. The available funding for RT was not even sufficient to sustain maintenance. Despite substantial effort the expertise around Preempt-RT did not result in sustainable commercial success.

During this time Thomas Gleixner started to warn about the non sustainability of Preempt-RT at conferences and in personal conversations. Aside of hilarious proposals and whining nothing useful happened.

After RedHat cut the RT funding completely, Gleixner and his company linutronix decided to stop their engagement in Preempt-RT and Gleixner declared at ELCE 2014 in Dusseldorf that Preempt-RT is now a hobbyist project.

Shortly after the conference OSADL stepped up and provided full time funding for two engineers to keep Preempt-RT at least maintained and forwarded to newer kernel versions in 2015.

Preempt-RT maintenance work was resumed, but Gleixner made it entirely clear that if there would be no way forward towards proper mainline integration by the end of the year he'd be not interested to maintain RT out of tree forever.

In 2015, the Linux Foundation started an effort, led by Kate Stewart, to create a Real-Time Linux Collaborative Project under the umbrella of the Linux Foundation. At ELCE 2015 in Dublin, the project was officially announced and the kick-off meeting took place. The funding allows five engineers to work full time on the project with the clear target of merging the Preempt-RT patches into the mainline kernel.

In the first two years hundreds of patches emerging from the Preempt-RT team around Thomas Gleixner at Linutronix were integrated into the Linux kernel. Most of this work was preparatory and refactoring existing upstream code in a way to make the integration of Preempt-RT easier. The main achievements were:

• Rewrite of the CPU hotplug infrastructure
• Refactoring of the timer wheel
• Refactoring of high resolution timers
• Various other bits and pieces

While this work makes RT integration and maintenance easier it is not solely RT specific and it is valuable for the quality and performance of the Linux kernel on its own.

This effort reduced the size of the RT patch set and led to the more interesting and hard to solve RT specific problems. While the scope of these pending changes is not as huge as the already merged work, it's more difficult to integrate it cleanly and justify the merge into the upstream kernel.

The Meltdown/Spectre security disaster slowed development down a bit for about two months at the end of 2017 and early 2018 as Preempt-RT team lead Thomas Gleixner was heavily involved in the development of the mitigations for the Linux kernel. The RT team is working hard to make up for the time lost and come back on the estimated project time line.

As of March 2018 the project is in an interesting situation. The funding for 2018 is not clear yet, which means that eventually work force has to be reduced and the expected time lines have to be adapted. There is lots of effort going on to remedy that situation, so it hopefully will get back on track soon.

The RT team is really grateful for the support we received so far from the project members and from the Linux Foundation. We are aware that the project members take responsibility for the RT using industry as a whole and it would be certainly a good chance now for bystanders to join the effort and help along to secure the technology on which their products depend on. RT is one of the most ambitious Linux technology projects and to some extent the most disruptive technology which aimed for mainline integration. So far we succeeded to keep the impact on mainline at a minimum and we hope that with funding secured for the time after mainline integration this can be kept up to everyone's satisfaction: RT users, users of regular Linux and the upstream Linux maintainers and developers.

As most of you know 2018 has started with a big bang in the computer industry, which will keep us occupied for quite some time. The upstream kernel mitigation of meltdown variant 3 via https://lwn.net/Articles/742693/PTI kept me beyond busy for the last two month.

Despite that disruption, which definitely required all brains collaborating, I've tried to keep up with the ongoing work of merging the Preempt-RT patch set into the Linux kernel. We've made great progress on various topics, especially the jinxed dcache problem. Patches will be posted as soon as folks are back from their well earned vacation.

In 2017 the RTL developer team was making enormous progress and we've upstreamed an impressive 1.5 patches a day on average. The deviation from the time line which we predicted end of 2015 is still on a relatively small margin of less than 20 percent.

We'd like to express our gratitude to the RTL project Members who provided the funding for the past two years, so we were able to work with five engineers full time on interesting, challenging and sometimes mind boggling problems.

As things stand for 2018 the funding will be reduced, so we expect to reduce the number of developers accordingly to something between three and four. That will have significant impact on the time line and postpone the final merge of RT by a not yet determined amount of time. I can't tell how much at the moment as this would be pure speculation (pun intended).

Nevertheless, we're starting into 2018 with vigor and the vision of making as much progress towards mainlining RT as possible.

Thomas & RTL developer team