CT-ISG: ADVANCED TECHNIQUES TO DETECT KERNEL-LEVEL ROOTKITS

The integrity of commodity operating system kernels is threatened by rootkitsthat modify key kernel data structures to achieve a variety of malicious goals.While rootkits have historically been known to affect control data in thekernel, recent work demonstrates rootkits that affect system security bymodifying non-control data, such as linked lists used to manage bookkeepinginformation and metadata used for memory management. Existing techniques failto detect such rootkits effectively.This project is developing techniques to provide real-time protection againstrootkits by detecting anomalies in both control and non-control kernel databehavior using automatically-generated integrity specifications. This goal isbeing achieved in two steps. First, a technique to mine specifications ofkernel data structure integrity is being developed. These specifications are bemined automatically as data structure invariants. Second, these techniques arebeing extended using operating system support to provide real-time detection.Impacts and Results: The techniques developed in this project will defendagainst the next generation of rootkits, and will enable real-time detection ofsuch rootkits. In addition, techniques to infer kernel invariants may also find applications in operating system reliability, fault tolerance andsoftware engineering. The PIs will disseminate the results by releasing the tools developed. The results of this project will equip the workforce with aninter-disciplinary toolkit, that combines operating systems, computer security,and software engineering, to address the challenges posed by the nextgeneration of stealth malware.