Improving security at the system-call boundary in a type-safe operating system

Abstract

Historically, most approaches to operating sytems security aim to either protect the kernel (e.g., the MMU) or protect user applications (e.g., W [symbol] X). However, little study has been done into protecting the boundary between these layers. We describe a vulnerability in Tock, a type-safe operating system, at the system-call boundary. We then introduce a technique for providing memory safety at the boundary between userland and the kernel in Tock. We demonstrate that this technique works to prevent against the aforementioned vulnerability and a class of similar vulnerabilities, and we propose how it might be used to protect against simliar vulnerabilities in other operating systems.