====== Continuous Integration Testing Overview ======
The CIP CI infrastructure monitors the CIP Kernel branches and upstream stable release candidates for changes. Once a change is detected a build process is kicked off using [[https://docs.gitlab.com/ee/ci/|GitLab CI/CD]] that provides binaries for a number of different Kernel configurations. These binaries are then tested on real hardware (where appropriate) using [[https://validation.linaro.org/static/docs/v2/|LAVA]].


==== Tests ====
=== Boot Test (Healthcheck) ===
This checks that the provided Kernel will boot on the target device.

  * Example test job: https://lava.ciplatform.org/scheduler/job/796683

=== Spectre/Meltdown Checker ===
This test verifies whether the target device is susceptible to Spectre/Meltdown vulnerabilities.

  * Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/spectre-meltdown-checker-test/spectre-meltdown-checker-test.yaml
  * Example test job: https://lava.ciplatform.org/scheduler/job/796614
  * Example test results: https://lava.ciplatform.org/results/796614/0_spectre-meltdown-checker-test

=== Linux Test Project (LTP) ===
LTP consists of a collection of tools for testing the Linux Kernel. More information can be found on the [[http://linux-test-project.github.io/|LTP website]].

We currently support the following LTP tests:

  * ltp-cve-tests
  * ltp-dio-tests
  * ltp-fs-tests
  * ltp-ipc-tests
  * ltp-math-tests
  * ltp-sched-tests
  * ltp-syscalls-tests
  * ltp-timers-tests

The tests can take a long time to run, so currently they are only run during  [[civilinfrastructureplatform:ciptesting:centalisedtesting:releasecandidatetesting|CIP Kernel release candidate testing]].

  * Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/ltp
  * Example test job: https://lava.ciplatform.org/scheduler/job/793640
  * Example test results: https://lava.ciplatform.org/results/793640/2_ltp-timers-tests
==== Real-Time Tests ====
Some tests are designed specifically for Real-Time Linux enabled Kernels.

=== Cyclictest ===
This test measures event latency in the Linux Kernel.

  * Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/cyclictest/cyclictest.yaml
  * Example test job: https://lava.ciplatform.org/scheduler/job/779941
  * Example test results: https://lava.ciplatform.org/results/779941/0_cyclictest

=== Cyclicdeadline ===
Similar to cyclictest but uses SCHED_DEADLINE to measure jitter.

  * Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/cyclicdeadline/cyclicdeadline.yaml
  * Example test job: https://lava.ciplatform.org/scheduler/job/779824
  * Example test results: https://lava.ciplatform.org/results/779824/0_cyclicdeadline

=== Hackbench ===
Linux scheduler stress test.

  * Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/hackbench/hackbench.yaml
  * Example test job: https://lava.ciplatform.org/scheduler/job/779943
  * Example test results: https://lava.ciplatform.org/results/779943/0_hackbench

=== Cyclictest+Hackbench ===
This test measures event latency in the Linux Kernel, with hackbench running in the background to stress the system. This provides a more realistic use case for cyclictest.

  * Test definition: https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/blob/master/lava_templates/test_cyclictest+hackbench.yaml
  * Example test job: https://lava.ciplatform.org/scheduler/job/796250
  * Example test results: https://lava.ciplatform.org/results/796250/1_cyclictest

==== When Tests are Run ====
^  Kernel  ^  Normal Testing  ^  Release Candidate Testing  ^
| v5.10.y-cip |  boot\\ smc  |  boot\\ smc\\ ltp  |
| v5.10.y |  boot\\ smc  |  N/A  |
| v4.19.y-cip |  boot\\ smc  |  boot\\ smc\\ ltp  |
| v4.19.y-cip-rt |  boot\\ smc\\ cyclictest+hackbench  |  boot\\ smc\\ ltp\\ cyclictest+hackbench  |
| v4.19.y |  boot\\ smc  |  N/A  |
| v4.9.y |  boot\\ smc  |  N/A  |
| v4.4.y-cip |  boot\\ smc  |  boot\\ smc\\ ltp  |
| v4.4.y-cip-rt |  boot\\ smc\\ cyclictest+hackbench  |  boot\\ smc\\ ltp\\ cyclictest+hackbench  |
| v4.4.y-st |  boot\\ smc  |  N/A  |
| v4.4.y |  boot\\ smc  |  N/A  |

==== CIP Kernel Configurations ====
The Kernel configurations that CIP is using are stored in the [[https://gitlab.com/cip-project/cip-kernel/cip-kernel-config|cip-kernel-config]] repository.

The [[https://gitlab.com/cip-project/cip-testing/linux-cip-pipelines/|linux-cip-pipelines]] repository stores the GitLab CI configuration files that specify what Kernel configurations are built and tested, including the specific devices used and test cases run.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

^  Kernel  ^  Arch  ^  Configuration  ^  Built  ^  Tested  ^  Target Devices  ^  Device Tree  ^
| v5.10.y-cip | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_cyclonev_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_de0-nano-soc_defconfig |  Y  |  Y  |  de0-nano-soc  |  socfpga_cyclone5_de0_nano_soc.dtb  |
| ::: | ::: | siemens_imx6.config |  Y  |  N  |  -  |  -  |
| ::: | arm64 | ctj_zynqmp_defconfig |  Y  |  Y  |  zynqmp-zcu102  |  zynqmp-zcu102-rev1.0.dtb  |
| ::: | ::: | moxa_eds_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_defconfig |  Y  |  Y  |  r8a774a1-hihope-rzg2m-ex  |  r8a774a1-hihope-rzg2m-ex.dtb  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_ipc227e_defconfig |  Y  |  Y  |  x86-simatic-ipc227e  |  N/A  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_atom_baytrail_cip.config |  Y  |  N  |  -  |  -  |
| v5.10.y | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_cyclonev_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_de0-nano-soc_defconfig |  Y  |  Y  |  de0-nano-soc  |  socfpga_cyclone5_de0_sockit.dtb  |
| ::: | ::: | siemens_imx6.config |  Y  |  N  |  -  |  -  |
| ::: | arm64 | moxa_eds_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_ipc227e_defconfig |  Y  |  Y  |  x86  |  N/A  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_atom_baytrail_cip.config |  Y  |  N  |  -  |  -  |
| v4.19.y-cip | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_cyclonev_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_de0-nano-soc_defconfig |  Y  |  Y  |  de0-nano-soc  |  socfpga_cyclone5_de0_nano_soc.dtb  |
| ::: | ::: | siemens_imx6.config |  Y  |  N  |  -  |  -  |
| ::: | arm64 | ctj_zynqmp_defconfig |  Y  |  Y  |  zynqmp-zcu102  |  zynqmp-zcu102-rev1.0.dtb  |
| ::: | ::: | moxa_eds_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_defconfig |  Y  |  Y  |  r8a774a1-hihope-rzg2m-ex  |  r8a774a1-hihope-rzg2m-ex.dtb  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_ipc227e_defconfig |  Y  |  Y  |  x86-simatic-ipc227e  |  N/A  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_atom_baytrail_cip.config |  Y  |  N  |  -  |  -  |
| v4.19.y-cip-rt | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_cyclonev_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | renesas_shmobile-rt_defconfig |  Y  |  Y<sup>1</sup>  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_de0-nano-soc_defconfig |  Y  |  Y  |  de0-nano-soc  |  socfpga_cyclone5_de0_nano_soc.dtb  |
| ::: | ::: | siemens_imx6.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | socfpga_defconfig |  Y  |  N  |  -  |  -  |
| ::: | arm64 | ctj_zynqmp_defconfig |  Y  |  Y  |  zynqmp-zcu102  |  zynqmp-zcu102-rev1.0.dtb  |
| ::: | ::: | moxa_eds_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_defconfig |  Y  |  Y  |  r8a774a1-hihope-rzg2m-ex  |  r8a774a1-hihope-rzg2m-ex.dtb  |
| ::: | ::: | renesas-rt_defconfig |  Y  |  Y<sup>1</sup>  |  r8a774a1-hihope-rzg2m-ex  |  r8a774a1-hihope-rzg2m-ex.dtb  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_i386-rt.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_ipc227e_defconfig |  Y  |  Y  |  x86-simatic-ipc227e  |  N/A  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_atom_baytrail_cip.config |  Y  |  N  |  -  |  -  |
| v4.19.y | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_cyclonev_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_de0-nano-soc_defconfig |  Y  |  Y  |  de0-nano-soc  |  socfpga_cyclone5_de0_sockit.dtb  |
| ::: | ::: | siemens_imx6.config |  Y  |  N  |  -  |  -  |
| ::: | arm64 | ctj_zynqmp_defconfig |  Y  |  Y  |  zynqmp-zcu102  |  zynqmp-zcu102-rev1.0.dtb  |
| ::: | ::: | moxa_eds_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_ipc227e_defconfig |  Y  |  Y  |  x86-simatic-ipc227e  |  N/A  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_atom_baytrail_cip.config |  Y  |  N  |  -  |  -  |
| v4.4.y-cip | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_am335x-axm2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-draco_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-dxr2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-etamin_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am57xx-pxm3.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_dcu2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_imx6_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_tegra_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_zynq_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx1.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_defconfig |  Y  |  N  |  -  |  -  |
| v4.4.y-cip-rt | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | renesas_shmobile-rt_defconfig |  Y  |  Y<sup>1</sup>  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | ::: | siemens_am335x-axm2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-draco_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-dxr2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-etamin_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am57xx-pxm3.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_dcu2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_imx6_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_tegra_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_zynq_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx1.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_i386-rt.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_defconfig |  Y  |  N  |  -  |  -  |
| v4.4.y | arm | cip_bbb_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | hitachi_omap_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | moxa_mxc_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | renesas_shmobile_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-axm2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-draco_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-dxr2_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am335x-etamin_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_am57xx-pxm3.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_dcu2.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_imx6_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_tegra_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_zynq_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | cip_qemu_defconfig |  Y  |  Y  |  qemu  |  N/A  |
| ::: | ::: | plathome_obsvx1.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_iot2000.config |  Y  |  N  |  -  |  -  |
| ::: | ::: | siemens_server_defconfig |  Y  |  N  |  -  |  -  |
| ::: | ::: | toshiba_defconfig |  Y  |  N  |  -  |  -  |

<sup>1</sup> Tested with Real Time specific tests

==== Extra Kernel Configurations ====
CIP also builds and tests a few non-CIP related Kernel configurations that are present in the Kernel.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

^  Kernel  ^  Arch  ^  Configuration  ^  Built  ^  Tested  ^  Target Devices  ^  Device Tree  ^
| v5.10.y | arm | shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | arm | multi_v7_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | arm64 | defconfig |  Y  |  Y  |  r8a774a1-hihope-rzg2m-ex  |  r8a774a1-hihope-rzg2m-ex.dtb  |
| ::: | x86 | defconfig |  Y  |  N  |  -  |  -  |
| v4.19.y | arm | shmobile_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | arm | multi_v7_defconfig |  Y  |  Y  |  r8a7743-iwg20d-q7  |  r8a7743-iwg20d-q7-dbcm-ca.dtb  |
| ::: | arm64 | defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | defconfig |  Y  |  N  |  -  |  -  |
| v4.4.y | arm | shmobile_defconfig |  Y  |  N  |  -  |  -  |
| ::: | arm | multi_v7_defconfig |  Y  |  N  |  -  |  -  |
| ::: | x86 | defconfig |  Y  |  N  |  -  |  -  |