Continuous Integration Testing Overview

The CIP CI infrastructure monitors the CIP Kernel branches and upstream stable release candidates for changes. Once a change is detected a build process is kicked off using GitLab CI/CD that provides binaries for a number of different Kernel configurations. These binaries are then tested on real hardware (where appropriate) using LAVA.

Tests

Boot Test (Healthcheck)

This checks that the provided Kernel will boot on the target device.

Example test job: https://lava.ciplatform.org/scheduler/job/796683

Spectre/Meltdown Checker

This test verifies whether the target device is susceptible to Spectre/Meltdown vulnerabilities.

Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/spectre-meltdown-checker-test/spectre-meltdown-checker-test.yaml
Example test job: https://lava.ciplatform.org/scheduler/job/796614
Example test results: https://lava.ciplatform.org/results/796614/0_spectre-meltdown-checker-test

Linux Test Project (LTP)

LTP consists of a collection of tools for testing the Linux Kernel. More information can be found on the LTP website.

We currently support the following LTP tests:

ltp-cve-tests
ltp-dio-tests
ltp-fs-tests
ltp-ipc-tests
ltp-math-tests
ltp-sched-tests
ltp-syscalls-tests
ltp-timers-tests

The tests can take a long time to run, so currently they are only run during CIP Kernel release candidate testing.

Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/ltp
Example test job: https://lava.ciplatform.org/scheduler/job/793640
Example test results: https://lava.ciplatform.org/results/793640/2_ltp-timers-tests

Real-Time Tests

Some tests are designed specifically for Real-Time Linux enabled Kernels.

Cyclictest

This test measures event latency in the Linux Kernel.

Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/cyclictest/cyclictest.yaml
Example test job: https://lava.ciplatform.org/scheduler/job/779941
Example test results: https://lava.ciplatform.org/results/779941/0_cyclictest

Cyclicdeadline

Similar to cyclictest but uses SCHED_DEADLINE to measure jitter.

Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/cyclicdeadline/cyclicdeadline.yaml
Example test job: https://lava.ciplatform.org/scheduler/job/779824
Example test results: https://lava.ciplatform.org/results/779824/0_cyclicdeadline

Hackbench

Linux scheduler stress test.

Test definition: https://github.com/Linaro/test-definitions/blob/master/automated/linux/hackbench/hackbench.yaml
Example test job: https://lava.ciplatform.org/scheduler/job/779943
Example test results: https://lava.ciplatform.org/results/779943/0_hackbench

Cyclictest+Hackbench

This test measures event latency in the Linux Kernel, with hackbench running in the background to stress the system. This provides a more realistic use case for cyclictest.

Test definition: https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/blob/master/lava_templates/test_cyclictest+hackbench.yaml
Example test job: https://lava.ciplatform.org/scheduler/job/796250
Example test results: https://lava.ciplatform.org/results/796250/1_cyclictest

When Tests are Run

Kernel	Normal Testing	Release Candidate Testing
v5.10.y-cip	boot smc	boot smc ltp
v5.10.y	boot smc	N/A
v4.19.y-cip	boot smc	boot smc ltp
v4.19.y-cip-rt	boot smc cyclictest+hackbench	boot smc ltp cyclictest+hackbench
v4.19.y	boot smc	N/A
v4.9.y	boot smc	N/A
v4.4.y-cip	boot smc	boot smc ltp
v4.4.y-cip-rt	boot smc cyclictest+hackbench	boot smc ltp cyclictest+hackbench
v4.4.y-st	boot smc	N/A
v4.4.y	boot smc	N/A

CIP Kernel Configurations

The Kernel configurations that CIP is using are stored in the cip-kernel-config repository.

The linux-cip-pipelines repository stores the GitLab CI configuration files that specify what Kernel configurations are built and tested, including the specific devices used and test cases run.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

Kernel	Arch	Configuration	Built	Tested	Target Devices	Device Tree
v5.10.y-cip	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_cyclonev_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_de0-nano-soc_defconfig	Y	Y	de0-nano-soc	socfpga_cyclone5_de0_nano_soc.dtb
		siemens_imx6.config	Y	N	-	-
	arm64	ctj_zynqmp_defconfig	Y	Y	zynqmp-zcu102	zynqmp-zcu102-rev1.0.dtb
		moxa_eds_defconfig	Y	N	-	-
		renesas_defconfig	Y	Y	r8a774a1-hihope-rzg2m-ex	r8a774a1-hihope-rzg2m-ex.dtb
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx2.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_ipc227e_defconfig	Y	Y	x86-simatic-ipc227e	N/A
		siemens_server_defconfig	Y	N	-	-
		toshiba_atom_baytrail_cip.config	Y	N	-	-
v5.10.y	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_cyclonev_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_de0-nano-soc_defconfig	Y	Y	de0-nano-soc	socfpga_cyclone5_de0_sockit.dtb
		siemens_imx6.config	Y	N	-	-
	arm64	moxa_eds_defconfig	Y	N	-	-
	arm64	renesas_defconfig	Y	N	-	-
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx2.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_ipc227e_defconfig	Y	Y	x86	N/A
		siemens_server_defconfig	Y	N	-	-
		toshiba_atom_baytrail_cip.config	Y	N	-	-
v4.19.y-cip	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_cyclonev_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_de0-nano-soc_defconfig	Y	Y	de0-nano-soc	socfpga_cyclone5_de0_nano_soc.dtb
		siemens_imx6.config	Y	N	-	-
	arm64	ctj_zynqmp_defconfig	Y	Y	zynqmp-zcu102	zynqmp-zcu102-rev1.0.dtb
		moxa_eds_defconfig	Y	N	-	-
		renesas_defconfig	Y	Y	r8a774a1-hihope-rzg2m-ex	r8a774a1-hihope-rzg2m-ex.dtb
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx2.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_ipc227e_defconfig	Y	Y	x86-simatic-ipc227e	N/A
		siemens_server_defconfig	Y	N	-	-
		toshiba_atom_baytrail_cip.config	Y	N	-	-
v4.19.y-cip-rt	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_cyclonev_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		renesas_shmobile-rt_defconfig	Y	Y¹	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_de0-nano-soc_defconfig	Y	Y	de0-nano-soc	socfpga_cyclone5_de0_nano_soc.dtb
		siemens_imx6.config	Y	N	-	-
		socfpga_defconfig	Y	N	-	-
	arm64	ctj_zynqmp_defconfig	Y	Y	zynqmp-zcu102	zynqmp-zcu102-rev1.0.dtb
		moxa_eds_defconfig	Y	N	-	-
		renesas_defconfig	Y	Y	r8a774a1-hihope-rzg2m-ex	r8a774a1-hihope-rzg2m-ex.dtb
		renesas-rt_defconfig	Y	Y¹	r8a774a1-hihope-rzg2m-ex	r8a774a1-hihope-rzg2m-ex.dtb
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx2.config	Y	N	-	-
		siemens_i386-rt.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_ipc227e_defconfig	Y	Y	x86-simatic-ipc227e	N/A
		siemens_server_defconfig	Y	N	-	-
		toshiba_atom_baytrail_cip.config	Y	N	-	-
v4.19.y	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_cyclonev_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_de0-nano-soc_defconfig	Y	Y	de0-nano-soc	socfpga_cyclone5_de0_sockit.dtb
		siemens_imx6.config	Y	N	-	-
	arm64	ctj_zynqmp_defconfig	Y	Y	zynqmp-zcu102	zynqmp-zcu102-rev1.0.dtb
		moxa_eds_defconfig	Y	N	-	-
		renesas_defconfig	Y	N	-	-
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx2.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_ipc227e_defconfig	Y	Y	x86-simatic-ipc227e	N/A
		siemens_server_defconfig	Y	N	-	-
		toshiba_atom_baytrail_cip.config	Y	N	-	-
v4.4.y-cip	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_am335x-axm2_defconfig	Y	N	-	-
		siemens_am335x-draco_defconfig	Y	N	-	-
		siemens_am335x-dxr2_defconfig	Y	N	-	-
		siemens_am335x-etamin_defconfig	Y	N	-	-
		siemens_am57xx-pxm3.config	Y	N	-	-
		siemens_dcu2.config	Y	N	-	-
		siemens_imx6_defconfig	Y	N	-	-
		toshiba_tegra_defconfig	Y	N	-	-
		toshiba_zynq_defconfig	Y	N	-	-
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx1.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_server_defconfig	Y	N	-	-
		toshiba_defconfig	Y	N	-	-
v4.4.y-cip-rt	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		renesas_shmobile-rt_defconfig	Y	Y¹	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
		siemens_am335x-axm2_defconfig	Y	N	-	-
		siemens_am335x-draco_defconfig	Y	N	-	-
		siemens_am335x-dxr2_defconfig	Y	N	-	-
		siemens_am335x-etamin_defconfig	Y	N	-	-
		siemens_am57xx-pxm3.config	Y	N	-	-
		siemens_dcu2.config	Y	N	-	-
		siemens_imx6_defconfig	Y	N	-	-
		toshiba_tegra_defconfig	Y	N	-	-
		toshiba_zynq_defconfig	Y	N	-	-
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx1.config	Y	N	-	-
		siemens_i386-rt.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_server_defconfig	Y	N	-	-
		toshiba_defconfig	Y	N	-	-
v4.4.y	arm	cip_bbb_defconfig	Y	N	-	-
		hitachi_omap_defconfig	Y	N	-	-
		moxa_mxc_defconfig	Y	N	-	-
		renesas_shmobile_defconfig	Y	N	-	-
		siemens_am335x-axm2_defconfig	Y	N	-	-
		siemens_am335x-draco_defconfig	Y	N	-	-
		siemens_am335x-dxr2_defconfig	Y	N	-	-
		siemens_am335x-etamin_defconfig	Y	N	-	-
		siemens_am57xx-pxm3.config	Y	N	-	-
		siemens_dcu2.config	Y	N	-	-
		siemens_imx6_defconfig	Y	N	-	-
		toshiba_tegra_defconfig	Y	N	-	-
		toshiba_zynq_defconfig	Y	N	-	-
	x86	cip_qemu_defconfig	Y	Y	qemu	N/A
		plathome_obsvx1.config	Y	N	-	-
		siemens_iot2000.config	Y	N	-	-
		siemens_server_defconfig	Y	N	-	-
		toshiba_defconfig	Y	N	-	-

¹ Tested with Real Time specific tests

Extra Kernel Configurations

CIP also builds and tests a few non-CIP related Kernel configurations that are present in the Kernel.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

Kernel	Arch	Configuration	Built	Tested	Target Devices	Device Tree
v5.10.y	arm	shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
	arm	multi_v7_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
	arm64	defconfig	Y	Y	r8a774a1-hihope-rzg2m-ex	r8a774a1-hihope-rzg2m-ex.dtb
	x86	defconfig	Y	N	-	-
v4.19.y	arm	shmobile_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
	arm	multi_v7_defconfig	Y	Y	r8a7743-iwg20d-q7	r8a7743-iwg20d-q7-dbcm-ca.dtb
	arm64	defconfig	Y	N	-	-
	x86	defconfig	Y	N	-	-
v4.4.y	arm	shmobile_defconfig	Y	N	-	-
	arm	multi_v7_defconfig	Y	N	-	-
	x86	defconfig	Y	N	-	-

Wiki

Table of Contents

Continuous Integration Testing Overview

Tests

Boot Test (Healthcheck)

Spectre/Meltdown Checker

Linux Test Project (LTP)

Real-Time Tests

Cyclictest

Cyclicdeadline

Hackbench

Cyclictest+Hackbench

When Tests are Run

CIP Kernel Configurations

Extra Kernel Configurations

Wiki

User Tools

Site Tools

Table of Contents

Continuous Integration Testing Overview

Tests

Boot Test (Healthcheck)

Spectre/Meltdown Checker

Linux Test Project (LTP)

Real-Time Tests

Cyclictest

Cyclicdeadline

Hackbench

Cyclictest+Hackbench

When Tests are Run

CIP Kernel Configurations

Extra Kernel Configurations

Page Tools