Uncovering New Classes of Kernel Vulnerabilities

Modern operating systems evolved into massively complex pieces of software with tens of millions lines of code. It is inevitable to have bugs in such large code bases, many of them with serious security implications. For decades, the kernel of such operating systems has been an interesting target for attackers due to its elevated privileges. Initially, attacks primarily targeted traditional software vulnerabilities like memory corruption. However, recent academic research has increasingly highlighted side-channel and transient execution vulnerabilities as well. While kernels have mitigations deployed against the most common vulnerability classes, many are too expensive for production systems. Instead, they are often used during continuous fuzzing efforts to find bugs. In recent years the amount of bugs discovered increased steadily with the improvements in bug detection during fuzzing, indicating that we are still scratching the surface and far from bug-free kernels. Additionally, state-of-the-art kernel fuzzers only focus on well-known bug classes and still find too many bugs to fix, urging the need to improve the security of our kernels. In this thesis, we uncover new classes of kernel vulnerabilities. Within the category of side-channel vulnerabilities, we demonstrate a novel way to combine multiple side channels to overcome limitations when attacking the kernel. With our attack we demonstrate that the very same feature that makes mitigation of side channels efficient, opens up a new attack surface. For transient execution vulnerabilities, we demonstrated the first gadget scanner based on dynamic analysis for the kernel. Detecting such gadgets is often difficult without suffering from large amounts of false positives, we showed that we can yield more precise detection by facilitating dynamic taint tracking. We implemented our scanner as a sanitizer to expose transient execution to traditional fuzzing environments to rely on existing bug detection capabilities. For software vulnerabilities, we find previously undiscovered type confusion bugs which we call container confusion bugs. Such bugs can be found in many large C code bases, such as kernels, that use nested structures to implement object-orientated functionality. We designed a specialized sanitizer to detect such bug patterns with continuous fuzzing and designed static analyzers to expand our search to sections of the kernel that are difficult to reach during fuzzing. In conclusion, we demonstrate that it is not enough to focus on currently well-established bug types and need to continue looking for new classes of vulnerabilities. We explored such new classes and improved fuzzing in all the main categories: software vulnerabilities, side channels, and transient execution attacks. Only by exploring such new exploitation angles and including them in our bug detection capabilities, we can slowly turn our kernels into a safe foundation of modern computing.