We are investigating how to use virtual machines to provide security in an operating-system-independent manner. Virtual-machine security services can work even if an attacker gains complete control over the guest operating system.
One hard part of designing virtual-machine security services is the semantic gap between the virtual machine and those services. Services in the virtual machine operate below the abstractions provided by the guest operating system and applications. This can make it difficult to provide services. For example, it is difficult to provide a service that checks file system integrity without knowledge of on-disk structures.
Another potential challenge of using virtual machines is that running all applications above the virtual machine hurts performance due to virtualization overhead. Commercial virtual machine monitors such as VMware achieve excellent performance by executing (mostly) directly on the bare hardware. However, we would like to use a virtual-machine monitor that runs as a user-mode application on top of a host operating system (a Type II VMM), and these tend to be an order of magnitude slower than a standalone system. We modified a host OS (Linux) to enable it to better support a virtual-machine monitor. The resulting virtual-machine monitor and modified guest OS (based on FAUmachine) runs even kernel-intensive applications at about 14-35% overhead. See our paper in USENIX 2003 for details.
We have designed and implemented a replay service for virtual machines called ReVirt. ReVirt logs enough information to replay a long-term execution of a virtual machine instruction-by-instruction. This enables it to provide arbitrarily detailed observations about what transpired on the system, even in the presence of non-deterministic attacks and executions. See our paper in OSDI 2002 for details.
While virtual-machine replay provides the ability to reconstruct an attack in arbitrary detail, it does not by itself analyze the attack. To help system administrators understand (and thereby recover from) an intrusion, we designed and implemented a system called BackTracker. BackTracker automatically identifies potential sequences of steps that occurred in an intrusion. Starting with a single detection point (e.g. a suspicious file), BackTracker identifies files and processes that could have affected that detection point and displays chains of events in a dependency graph. See our paper in SOSP 2003 for details. Similar techniques can be used to identify the effects of an attack on a local host and on other downstream hosts. See our paper in NDSS 2005 for details.
The ability to replay the execution of a virtual machine is useful in many ways besides intrusion analysis. For example, it enables one to replay and debug any portion of a prior execution. We have built an extension to gdb that uses virtual-machine replay to provide the illusion of time travel. In particular, we provide the ability to do reverse debugging, though commands such as reverse watchpoint and reverse breakpoint. graph. See our paper in USENIX 2005 for details. One can also use virtual-machine replay to synchronize two physical processors and thereby detect and recover from transient processor faults. See our work-in-progress paper for a preliminary description of this system.
Another technique that leverages the power of virtual machines is virtual-machine introspection, which is the ability of a service to examine and understand the internal state of a running virtual machine. We combine VM introspection with vulnerability-specific predicates to detect when a specific software vulnerability is being triggered in an executing virtual machine. When combined with virtual-machine replay, vulnerability-specific predicates allow us to answer a question that is all-to-rarely asked: "Did someone break into my system using a vulnerability that I (now) know about?". See our paper in SOSP 2005 for more details.
Virtual machine are a double-edged sword. While they can be used to improve security, they can also be used by attackers to gain more control over a system. We have implemented a virtual-machine based rootkit, called SubVirt, that can install itself below an existing operating system in a way that hides all the rootkit's state from the view of the operating system. See our paper in IEEE Security and Privacy 2006 for more details about this attack and how to defend against it.