Software engineering services

FuzzSync race exposition library

This leads on from the previous one.

• Problem: Reproducing some bugs requires reliably reproducing a data race. The usual methods of doing this are either resource intensive or require a particular kernel.
• Solution: Create a library which makes that easier
• Benefit: We can easily reproduce most bugs involving a data race without resorting to tricks that require a particular kernel config.

CGroup API

Control groups have emerged as a critical kernel interface. They are used by container, VM and system managers. As part of a larger effort to increase test coverage of them I increased LTP’s support.

• Problem: It’s difficult to write tests which interact with both Kernel CGroup APIs V1 and V2. Also to discover the existing CGroup setup created by, for e.g., systemd.
• Solution: Create a compatability layer which abstracts controller discovery, CGroup creation and interactions.
• Benefit: It is far easier now to write tests which interact with CGroups, for example cfs_bandwidth01 which I wrote. More importantly it encourages others to write tests interacting with CGroups.

Sparse static analysis

• Problem: We encounter repetitive mistakes during review especially around LTP library usage
• Solution: Implement our own C static analysis tool based on Sparse. So far only 3 checks were implemented
• Benefit: 3 less problems around improper usage of the API. A better experience for contributors and maintainers.

Improving the new user/contributor experience

• Problem: It wasn’t immediately clear how to run the tests or write a new one
• Solution: Add a quick start and other documentation
• Benefit: More new users and contributors assuming it wasn’t a coincidence

eBPF testing and more

I have introduced other areas of testing which follow the same pattern as above.

• Problem: It’s difficult to test some kernel feature consistently across different systems.
• Solution: Introduce some supporting code into the LTP test framework
• Benefit: Increased test coverage of the kernel and better feedback for kernel developers

CAN and SLIP

Found and fixed some issues in CAN and SLIP which are potentially exploitable:

• b9258a2cece4 slcan: Don’t transmit uninitialized stack data in padding
• 0ace17d56824 can, slip: Protect tty->disc_data in write_wakeup and close with RCU

vsock

• 4c1e34c0dbff vsock: Enable y2038 safe timeval for timeout
• 685c3f2fba29 vsock: Refactor vsock_*_getsockopt to resemble sock_getsockopt

memcg/slab

I found and suggested a fix for a bug in the memory CGroup. It was refused in favor of a more general fix.

mm: memcg/slab: Stop reparented obj_cgroups from charging root

New QEMU backend

The QEMU backend manages VMs used for testing. OpenQA makes use of snapshotting to revert a VM to a good state (anchor point) when something fails and continue testing.

• Problem: Snapshotting was slow and would fail under many circumstances. Additionally there were many smaller issues, such as exporting UEFI firmware variables. This made some testing impossible.
• Solution: Rewrite the QEMU backend
• Benefits: Our test matrix could be expanded by a considerable amount. Increasing coverage and finding more problems.

Real serial console

• Problem: Originally being purely a GUI testing framework, serial consoles were supported in a very roundabout way
• Solution: Directly support serial consoles (primarily virtio in QEMU)
• Benefits: Dramatic decrease in test runtime which means developers get test results faster and have more time to react to failures. It also decreased resource usage allowing more testing.

LTP test runner

• Problem: Only some LTP tests were being run, many were failing and determining whether it was due to a product bug or test bug was difficult
• Solution: Write a test runner for LTP within OpenQA that displayed test results well and provided debugging aids.
• Benefits: A big expansion in test coverage, kernel bugs were reported with useful info and LTP tests could be flagged for fixing.