Nytro Posted June 21, 2019 Report Posted June 21, 2019 LLDBagility: practical macOS kernel debugging Date Tue 18 June 2019 By Francesco Cagnin Category macOS. Tags macOS XNU kernel hypervisor debugging FDP LLDBagility This is the second of two blog posts about macOS kernel debugging. In the previous post, we defined most of the terminology used in both articles, described how kernel debugging is implemented for the macOS kernel and discussed the limitations of the available tools; here, we present LLDBagility, our solution for an easier and more functional macOS debugging experience. As the reader may have concluded after reading the previous blog post [1], in the current state of things debugging the macOS kernel (XNU) is not very practical, especially considering inconveniences like having to install debug kernel builds and the impossibility to set watchpoints or pause the execution of the kernel from the debugger. To improve the situation, we developed LLDBagility [2], a tool for debugging macOS virtual machines with the aid of the Fast Debugging Protocol (FDP, described later in the article). Above all else, LLDBagility implements a set of new LLDB commands that allow the debugger to: attach to running macOS VirtualBox virtual machines and debug their kernel, stealthily, without the need of changing the guest OS (e.g. no necessity of DEVELOPMENT or DEBUG kernels, boot-args modification or SIP disabling) and with minimal changes to the configuration of the VM [3]; interrupt (and later resume) the execution of the guest kernel at any moment; set hardware breakpoints anywhere in kernel code, even at the start of the boot process; set hardware watchpoints that trigger on read and/or write accesses of the specified memory locations; save and restore the state of the VM in a few seconds. These commands are intended to be used alongside the ones already available in LLDB, like memory/register read/write, breakpoint (software), step and so on. Moreover, in case the debugged kernel comes with its Kernel Debug Kit (and possibly even when it doesn't, as discussed later), the vast majority of lldbmacros work as expected. In the next sections, we first briefly describe what FDP is and how it enhances VirtualBox to enable virtual machine introspection; then, we examine how LLDBagility takes advantage of these new capabilities to connect transparently LLDB to macOS virtual machines, and demonstrate a simple kernel debugging session; finally, we present some ideas for using existing lldbmacros with kernel builds lacking a debug kit. Virtual machine introspection via the Fast Debugging Protocol The Fast Debugging Protocol [4] is a light, fast and efficient debugging API for stealthy introspection of virtual machines, written in C and currently only released as a patch to VirtualBox's source code [FDP]. This API makes it possible to: read and write VMs registers and memory; set and unset hardware/software breakpoints and watchpoints; pause, resume and single-step the execution of the virtual machines; save and restore the VMs state. FDP's stealthiness comes from the manipulation of the Extended Page Tables of the virtual machines (operation that can be very difficult to detect from the guests), while speed is the result of using shared memory between FDP and the VMs (reaching in some cases one million of memory reads per second). In addition to the low-level C interface, the API also provides Python bindings (PyFDP) that can be useful both for quick proof of concepts and for bigger projects like LLDBagility; as an example, here is a script that breaks on every system call: #!/usr/bin/env python2 from PyFDP.FDP import FDP # Open the debuggee VM by its name fdp = FDP("18E226") # Pause the VM to install a breakpoint fdp.Pause() # Install a hardware breakpoint (very fast and stealth) UnixSyscall64 = 0xffffff80115bae84 fdp.dr0 = UnixSyscall64 fdp.dr7 = 0x403 # Resume the execution after the breakpoint installation fdp.Resume() while True: # Wait for a breakpoint hit fdp.WaitForStateChanged() print "%x" % (fdp.rax) # Jump over the breakpoint fdp.SingleStep() # Resume the debuggee fdp.Resume() More details about FDP can be found in the original article on Winbagility [4] (in French) [WinbagilitySSTIC2016], LLDBagility's older counterpart for Windows and WinDbg. Attaching LLDB to virtual machines As presented at length in the previous post, during regular two-machine debugging, LLDB interacts with the macOS kernel of the debuggee by sending commands to its internal KDP stub, which, being itself part of the kernel, is then able to inspect and alter the state of the machine as requested and communicate back the results. The key idea behind LLDBagility is to replicate the functionalities provided by such agent using the analogous capabilities of (virtual) machine introspection and modification offered by FDP at the hypervisor level, so that the kernel's debugging stub can be replaced by an equivalent alternative external to the debugged machine. In addition, by maintaining compatibility with the KDP protocol, this new stub can take advantage of LLDB's existing support for the macOS kernel without modifying the debugger in any aspect, since there's no difference in communications with respect to the KDP server in the kernel. In this regard, Ian Beer used an analogous solution for his iOS kernel debugger [5]. This approach brings in two notable advantages: the elimination of the necessity of enabling KDP in the kernel for debugging, and the possibility of augmenting LLDBagility's stub with additional features: all the limitations of the classic debugging method described in the first blog post can now be overcome. First of all, since setting up XNU for debugging is not required anymore, it becomes unnecessary to modify the NVRAM and possibly disabling SIP; likewise, there is also no need to install DEBUG or DEVELOPMENT builds. Secondly, all system-wide side effects related to KDP code disappear, implying for example a more difficult debugger detection by rootkits. Thirdly, debugging can now start before the initialisation of the KDP stub during the kernel boot process. Finally, having complete control over the machine thanks to FDP makes it easy to implement hardware breakpoints, watchpoints and a mechanism for pausing the kernel at debugger's command. Overview of the tool As mentioned in the introduction, LLDBagility's front end is a set of new LLDB commands, implemented in lldbagility.py (from [LLDBagility100]😞 fdp-attach or simply fa, to connect the debugger to a running macOS VirtualBox virtual machine; fdp-hbreakpoint or fh, to set and unset read/write/execute hardware breakpoints; fdp-interrupt or fi, to pause the execution of the VM and return the control to the debugger (equivalent to the known sudo dtrace -w -n "BEGIN { breakpoint(); }"); fdp-save or fs, to save the current state of the VM; fdp-restore or fr, to restore the VM to the last saved state. On the other hand, the back end is composed of: kdpserver.py, the new KDP server (replacement of XNU's) to which LLDB is connected, whose job is to translate the KDP requests received by the debugger into commands for the VM and send back responses and exception notifications (e.g. breakpoint hits); stubvm.py, for abstracting common VM operations and actualising them with FDP. The connection between LLDB and the virtual machine to debug is established on fdp-attach. When the user executes this command, LLDBagility: instantiates a PyFDP client for interacting with the VM (lldbagility.py#L73); starts its own KDP server on a secondary thread (lldbagility.py#L93), waiting for requests (kdpserver.py#L182); executes the LLDB command kdp-remote to connect the debugger to the KDP server that has just started (lldbagility.py#L99, procedure explained in detail in the first blog post), kicking off the debugging process. As a last interesting note, the translation of KDP requests into corresponding FDP commands is not sufficient to have lldbmacros working correctly. In fact, such macros rely on the ability to use some fields of the kdp struct (osfmk/kdp/kdp_internal.h#L41/#L55, tools/lldbmacros/core/operating_system.py#L777 both from [XNU49032212]), populated by the kernel only when its KDP stub is enabled (e.g. osfmk/kdp/kdp_udp.c#L1349). LLDBagility obviates the problem by hooking memory reads from the debugger to the kdp struct and returning a patched memory chunk in which necessary fields (e.g. kdp_thread) are filled with proper values (stubvm.py#L242). Demo The following video demonstrates a quick debugging session with LLDBagility: In summary: LLDB is started and fdp-attach'ed to the virtual machine named "18E226" running in background; the lldbmacros (automatically loaded from .lldbinit) showcurrentthreads and showrights are executed; the execution of the VM is resumed with the continue command and then quickly interrupted with fdp-interrupt; the state of the VM is first saved with fdp-save, then modified by simply resuming the machine execution, and finally restored with fdp-restore; a read and write watchpoint is set with fdp-hbreakpoint on &(*(boot_args*)PE_state.bootArgs).flags, which triggers immediately thereafter inside csr_check(). Reusing debug information between kernels In this last section we explore some ideas about debugging kernels with lldbmacros coming from a Kernel Debug Kit released for a different kernel build. The methods proposed here have their obvious limitations, but they also have been proven to work sufficiently well and are worth a shot; see KDKutils/ for code and examples. As mentioned in the first post, the absence of KDKs for most kernel builds makes debugging more difficult, since in such cases types information and lldbmacros are not available. Luckily, the debug information required by these macros to work correctly comprises a relatively small number of global variables and structs, like version (config/version.c#L40) and kdp, whose definitions do not change much (or at all) from one kernel version to the next. In this light, it seems reasonable to try to reuse debug information from available KDKs so that most features of existing lldbmacros can be used with kernels lacking their own debug kit. As an example, let us assume we have debug information (i.e. the DWARF file generated during compilation) and lldbmacros for kernel build A, but not for build B. With the goal of loading and using A's lldbmacros for debugging B in LLDB, it is possible to provide the debugger with a "fake" DWARF file for kernel B, created by patching A's in such a way that: its Mach-O UUID matches B's instead of A's, and the load addresses of all symbols used by the macros match the virtual load addresses of the same symbols in B's memory at run time. These two small changes should be enough to have most macros working. The simple approach just discussed works well when the symbols used by lldbmacros have very similar definitions in the code of the two kernels, which is naturally the case when they are successive releases. When this does not happen (for example when a field is added or removed from the thread struct), then the workaround above is no longer sufficient, since it becomes necessary to also modify the DWARF file to alter types definitions; but this is not easy. For such cases, it is possible and better to create the fake DWARF from scratch, by first parsing a source DWARF file to extract types information about the structures used by lldbmacros, and then generating C source files containing the types definitions. These sources can easily be edited by hand to modify types as desired and then compiled so that a new DWARF file is created. Before loading it in LLDB, the DWARF file must be patched as above to change its UUID and load address to match the kernel being debugged. Wrapping up With this post, we presented LLDBagility, a convenient alternative to classic macOS kernel debugging based on virtual machine introspection. While not perfect (yet), our solution sets aside or greatly alleviates all current limitations of the standard approach, and at the same time offers powerful additional features that make debugging much more practical. LLDBagility has been actively used at Quarkslab in the past months and it is being released for others to experiment with; feedback via GitHub is welcome and encouraged. Happy debugging! References and footnotes [FDP] Source code of the Fast Debugging Protocol. Available at https://github.com/Winbagility/Winbagility/tree/fe1e38c2cbad6a1fb92dbad3440c6381768aa8cc/src/FDP/, https://github.com/Winbagility/Winbagility/tree/fe1e38c2cbad6a1fb92dbad3440c6381768aa8cc/src/VBoxPatch/ and https://github.com/Winbagility/Winbagility/tree/fe1e38c2cbad6a1fb92dbad3440c6381768aa8cc/bindings/python/ [WinbagilitySSTIC2016] Couffin, Nicolas. (2016). Débogage furtif et introspection de machine virtuelle. Available at https://www.sstic.org/media/SSTIC2016/SSTIC-actes/debogage_furtif_et_introspection_de_machines_virtu/SSTIC2016-Article-debogage_furtif_et_introspection_de_machines_virtuelles-couffin.pdf [LLDBagility100] Source code of LLDBagility 1.0.0. Available at https://github.com/quarkslab/LLDBagility/tree/v1.0.0/ [XNU49032212] Source code of XNU 4903.221.2 from macOS 10.14.1. Available at https://github.com/apple/darwin-xnu/tree/xnu-4903.221.2/ [1] https://blog.quarkslab.com/an-overview-of-macos-kernel-debugging.html [2] https://github.com/quarkslab/LLDBagility/ [3] As per current FDP limitations: the VM must use a single processor and no more than 2 GB of RAM. [4] (1, 2) https://winbagility.github.io/ [5] https://bugs.chromium.org/p/project-zero/issues/detail?id=1417#c16 Sursa: https://blog.quarkslab.com/lldbagility-practical-macos-kernel-debugging.html Quote
