This wiki page describes the policies and recommendations that the CIP kernel maintainers are following in order to maintain the CIP kernel beyond the current periods established by the enterprise industry, in order to meet Industrial Grade requirements.

Nobuhiro Iwamatsu, from Toshiba Corporation, Pavel Machek, from DENX Software Engineering GmbH and Ulrich Hecht are the current CIP kernel maintainer. More developers will join in future. And when we release the CIP kernel, we will sign it with GnuPG by the maintainers. Each public key can be obtained from the following.

• Nobuhiro Iwamatsu (5E629EE5232197357B84CF4332247FBB40AD1FA6)
• Pavel Machek (4FA79EEFFCD4C44FC585B8C7C060224192DFCE96)
• Ulrich Hecht (719474A4F443B6E721C49E0B36A3BADB36B27332)

New major versions of commercial Linux distributions are released at 3-4 year intervals and maintained over 10-13 years, so that typically only 3-4 versions need to be supported at one time. Given that CIP's support period is meant to be even longer, it won’t be sustainable to extend every LTS branch, but only to take on a new branch every 2-4 years.

The longer the intervals between new SLTS branches, the greater need there will be for CIP or individual members to backport new hardware support (which carries its own risks). This trade-off is perhaps the most difficult issue to decide.

CIP will measure the effort involved in maintaining the kernel branch we already labelled as SLTS (4.4) and study the technical implications of this work during 2018 in order to define when the next SLTS branch should be started.

The use cases CIP project is targeting have a life cycle of between 25 and 50 years. In theory, this is the time in which products shipped with the CIP kernel will be under maintenance. However, identifying and backporting relevant fixes becomes increasingly difficult as upstream kernel development diverges further from a stable branch. Any given SLTS branch is unlikely to be economically maintainable for much more than 10 years.

The Linux kernel 4.4 was released on January 10th 2016. It was declared LTS (Long Term Support) by the stable team which means Greg Kroah-Hartman was supposed to maintain it for two years (Feb 2018) following the kernel LTS process. In September 2017 it was announced that the 4.4 LTS branch will be maintained during 6 years. Another stable maintainer could extend LTS maintenance for some time beyond that. After that period, CIP will maintain it.

The Linux kernel 4.19 was released on October 22nd, 2018.

The policy for which changes to accept is based on stable-kernel-rules.rst. For reference, these are the rules with numbering added:

1. It must be obviously correct and tested.
2. It cannot be bigger than 100 lines, with context.
3. It must fix only one thing.
4. It must fix a real bug that bothers people (not a, “This could be a problem…” type thing).
5. It must fix a problem that causes a build error (but not for things marked CONFIG_BROKEN), an oops, a hang, data corruption, a real security issue, or some “oh, that's not good” issue. In short, something critical.
6. Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact.
7. New device IDs and quirks are also accepted.
8. No “theoretical race condition” issues, unless an explanation of how the race can be exploited is also provided.
9. It cannot contain any “trivial” fixes in it (spelling changes, whitespace cleanups, etc).
10. It must follow the Documentation/SubmittingPatches rules.
11. It or an equivalent fix must already exist in Linus' tree (upstream).

In practice, exceptions are sometimes made to rules 2, 3, 8 and 9, and we will likewise treat those as should rather than must rules.

After the end of LTS at kernel.org, the fixes made to a SLTS branch will be more restricted:

• Architecture and hardware support will be limited to the needs of CIP members; fixes for other hardware will not be applied except where that makes it easier to apply the fixes we do need. For the 4.4 branch, the architectures we've committed to so far are arm (32-bit) and x86_64.
• Some features, particularly those labelled as experimental, may also be excluded from support. We have not done so yet.

Unlike an LTS branch, an SLTS branch may include larger changes to support new hardware, to bridge the gap between SLTS branches. However, this should only be done on the newest SLTS branch.

CIP will release sources, not binaries, except for the CIP platforms once we start to build a more complete platform. The SLTS kernel maintainer will push a signed tag to the kernel git repository for each release.

Once we have automated testing in place, a release candidate must be tested on all the platforms that are covered, with no regressions, before it is tagged as a release.

As we will be backporting fixes from mainline, which may change in-kernel APIs, we will not guarantee API or ABI stability within an SLTS branch. It is the responsibility of members to update any out-of-tree components they use in case they are affected by such a change. However, incompatible changes to user-space APIs or ABIs will not be acceptable.

CIP has released new CIP minor kernel version. This depends on the kernel version and flavor. linux-4.4.y is scheduled to be released once a month, linux-4.19.y is scheduled to be released twice a month, and the RT kernel is scheduled to be released once every two months. You can find the latest CIP kernel version in git.kernel.org. Each release will be announced in the cip-dev mailing list and occasionally on the new section of the CIP project website.

CIP maintainers will maintain and respond to bug reports from CIP members and their developers, but not system administrators or end users in general, about upstream patches. Bugs should be reported upstream.

The embedded systems that CIP will be used in will also often require out-of-tree drivers and will sometimes include other changes of unknown quality to their kernel. These modifications are in general unsupported. If a bug is found in such a modified kernel, Members will first demonstrate that it exists in the CIP source release in order for the CIP maintainers to act on it.

For specific CIP applied patches, the default communication channel will be the CIP-dev mailing list. Patches sent to the cip-dev mailing list are tracked on patchwork since October 12th, 2018.

Kernel configuration files provided by Members will be used as a reference to limit the maintenance scope. Here is a list of recommendations related to kernel configuration for long-term support:

• https://lists.cip-project.org/pipermail/cip-dev/2017-May/000263.html
• https://lists.cip-project.org/pipermail/cip-dev/2017-July/000387.html
• https://lists.cip-project.org/pipermail/cip-dev/2018-October/001578.html

CIP works towards reducing the window of vulnerability to zero. To achieve this goal, CIP will collaborate with the wider kernel community in initiatives like the Kernel Self Protection Project.

Although this goal may be achievable for CIP, its members may take much longer to release and deploy binary updates, maybe due to valid concerns about the risk of regression or limited opportunities to deploy updates. In the worst case, they may use CIP as an advertising/compliance point - please don't do that!

To track CVE status in each branch and provide this information either to members or publicly some scripts has been developed and published by Masami Ichikawa.

These recommendations has been written by Ben hutchings.

If you are doing this after a stable release, all the patches will be in the git history of the branch.

If you want to do this during the review period before release, you should be able to receive the patches through the stable@vger.kernel.org list. This is a high-traffic list, so you may want to filter by subject. However, at the time of writing, Greg's script that sends patches out for review has a bug that causes patches with non-ASCII characters to be dropped. Greg and most other stable maintainers also publish pending patches in a git repository, either as a quilt patch series or as a rebased git branch.

Pending patches for stable can be found at:

• Greg Kroah-Hartman queue.
• Alexander (Sasha) Levin queue.

Does the patch touch code that's maintained in the CIP branch? (This should be scripted. If not, ignore it and move on.)

Does the commit message explain why the change is important, i.e. does the explanation seem to fit the criteria for stable? If not, maybe this doesn't belong in the stable branch. But do consider:

• Sometimes an unimportant commit will be included because it's a dependency of an important fix. Look at the following patches to see if they're related.
• Sometimes a commit that didn't seem important at the time turns out to be important later, e.g. cleaning up a function turned out to fix a corner case that was a security flaw. You can check the stable mailing list archives to see if someone requested it and why.
• If the author is Linus, be aware that he often makes security fixes without describing the security implications.

Look for a Fixes: tag in the commit message, identifying which commit introduced (or only partly fixed) the bug that's being fixed:

• If it's not present, look to see if the free-form text identifies which commit introduced it.
• Is that commit actually present in the stable branch? Don't forget that it might have been backported as part of another stable update. If not, this probably doesn't belong in the stable branch.
• Find the first release containing a commit:

git describe --contains --match 'v*' hash

• Check whether a stable branch contains a backported commit:

git log --pretty=oneline --grep=hash vversion..stable/linux-version.y

The git-check-in-stable script from Debian kernel/kernel-team.git provides a shortcut (and slightly nicer output format) for the second command.

Patches only have 3 lines of context around the lines being changed, but in some cases you need to consider the whole of each function (or other definition) being changed. So you will often need to look at the files being patched, not just the patches.

• Does the patched code appear to do what the description says?
• Does the description cover all the code changes, or are there other changes which might not be appropriate for stable?
  • Ignore white-space changes - they shouldn't be combined with fixes, but at least they won't do any harm).
• If a file contains multiple similar blocks of code, is the right one being patched?
• If the patch changes a function to acquire and then release a lock or other resource, are there any additional paths through the function where the resource can leak? For example, there might be an extra return statement in the stable branch, compared to the upstream version.
• If the patch adds a return, goto, or break statement, again check for leaks.
• Sometimes the upstream commit will have been modified (backported) to create the stable patch, because it depended on new or changed APIs, or because a function that needed to be fixed has been merged or split up in some way. If it has been backported, do the modifications make sense? If someone other than the original author backported it, they should have added an explanation in the new commit message.
• If the patch changes the type or semantics of a function, all its callers probably need to be updated. In the stable branch, there might be additional callers compared to upstream. Have any of them been missed?

If you find a regression that is specific to the stable branch, it should be fixed by revising or rejecting the patch for the stable branch, or (if it has already been applied and released) by adding another stable-specific fix.

If you find a regression that's also present upstream, it should be fixed upstream first. In fact, that might already have happened:

• If there is already a fix in Linus's git repository, ask for it to be included in a stable update straight away.
• Otherwise, if the patch that introduces the regression hasn't yet been applied then you should ask for it to be deferred (or never applied, if it was reverted upstream).

• linux-4.4.y review
• new reviews are done in lts-commit-list repository
• linux-4.4.y failed patches

• CIP kernel release tutorial

Back to the parent page