Nytro Posted July 22, 2018 Report Posted July 22, 2018 Tutorial - emulate an iOS kernel in QEMU up to launchd and userspace Jul 21, 2018 I got launchd and recoveryd to start on an emulated iPhone running iOS 12 beta 4’s kernel using a modified QEMU. Here’s what I learned, and how you can try this yourself. Introduction This is Part 2 of a series on the iOS boot process. Part 1 is here. Sign up with your email to be the first to read new posts. skip to: tutorial, writeup First, let me repeat: this is completely useless unless you’re really interested in iOS internals. If you want to run iOS, you should ask @CorelliumHQ instead, or just buy an iPhone. I’ve been interested in how iOS starts, so I’ve been trying to boot the iOS kernel in QEMU. I was inspired by @cmwdotme’s Corellium, a service which can boot any iOS in a virtual machine. Since I don’t have 9 years to build a perfect simulation of an iPhone, I decided to go for a less lofty goal: getting enough of iOS emulated until launchd, the first program to run when iOS boots, is able to start. Since last week’s post, I got the iOS 12 beta 4 kernel to fully boot in QEMU, and even got it to run launchd and start recoveryd from the restore ramdisk. Here’s the output from the virtual serial port: iBoot version: corecrypto_kext_start called FIPSPOST_KEXT [64144875] fipspost_post:156: PASSED: (4 ms) - fipspost_post_integrity FIPSPOST_KEXT [64366750] fipspost_post:162: PASSED: (1 ms) - fipspost_post_hmac FIPSPOST_KEXT [64504187] fipspost_post:163: PASSED: (0 ms) - fipspost_post_aes_ecb FIPSPOST_KEXT [64659750] fipspost_post:164: PASSED: (0 ms) - fipspost_post_aes_cbc FIPSPOST_KEXT [72129500] fipspost_post:165: PASSED: (117 ms) - fipspost_post_rsa_sig FIPSPOST_KEXT [76481625] fipspost_post:166: PASSED: (67 ms) - fipspost_post_ecdsa FIPSPOST_KEXT [77264187] fipspost_post:167: PASSED: (11 ms) - fipspost_post_ecdh FIPSPOST_KEXT [77397875] fipspost_post:168: PASSED: (0 ms) - fipspost_post_drbg_ctr FIPSPOST_KEXT [77595812] fipspost_post:169: PASSED: (1 ms) - fipspost_post_aes_ccm FIPSPOST_KEXT [77765500] fipspost_post:171: PASSED: (1 ms) - fipspost_post_aes_gcm FIPSPOST_KEXT [77941875] fipspost_post:172: PASSED: (1 ms) - fipspost_post_aes_xts FIPSPOST_KEXT [78176875] fipspost_post:173: PASSED: (1 ms) - fipspost_post_tdes_cbc FIPSPOST_KEXT [78338625] fipspost_post:174: PASSED: (1 ms) - fipspost_post_drbg_hmac FIPSPOST_KEXT [78460125] fipspost_post:197: all tests PASSED (233 ms) AUC[<ptr>]::init(<ptr>) AUC[<ptr>]::probe(<ptr>, <ptr>) Darwin Image4 Validation Extension Version 1.0.0: Mon Jul 9 21:36:59 PDT 2018; root:AppleImage4-1.200.16~357/AppleImage4/RELEASE_ARM64 AppleCredentialManager: init: called, instance = <ptr>. ACMRM: init: called, ACMDRM_ENABLED=YES, ACMDRM_STATE_PUBLISHING_ENABLED=YES, ACMDRM_KEYBAG_OBSERVING_ENABLED=YES. ACMRM: _loadRestrictedModeForceEnable: restricted mode force-enabled = 0 . ACMRM-A: init: called, . ACMRM-A: _loadAnalyticsCollectionPeriod: analytics collection period = 86400 . ACMRM: _getDefaultStandardModeTimeout: default standard mode timeout = 604800 . ACMRM: _loadStandardModeTimeout: standard mode timeout = 604800 . ACMRM-A: notifyStandardModeTimeoutChanged: called, value = 604800 (modified = YES). ACMRM: _loadGracePeriodTimeout: device lock timeout = 3600 . ACMRM-A: notifyGracePeriodTimeoutChanged: called, value = 3600 (modified = YES). AppleCredentialManager: init: returning, result = true, instance = <ptr>. AUC[<ptr>]::start(<ptr>) AppleKeyStore starting (BUILT: Jul 9 2018 21:51:06) AppleSEPKeyStore::start: _sep_enabled = 1 AppleCredentialManager: start: called, instance = <ptr>. AppleCredentialManager: start: initializing power management, instance = <ptr>. AppleCredentialManager: start: started, instance = <ptr>. AppleCredentialManager: start: returning, result = true, instance = <ptr>. AppleARMPE::getGMTTimeOfDay can not provide time of day: RTC did not show up : apfs_module_start:1277: load: com.apple.filesystems.apfs, v748.200.53, 748.200.53.0.1, 2018/07/09 com.apple.AppleFSCompressionTypeZlib kmod start IOSurfaceRoot::installMemoryRegions() IOSurface disallowing global lookups apfs_sysctl_register:818: done registering sysctls. com.apple.AppleFSCompressionTypeZlib load succeeded L2TP domain init L2TP domain init complete PPTP domain init BSD root: md0, major 2, minor 0 apfs_vfsop_mountroot:1468: apfs: mountroot called! apfs_vfsop_mount:1231: unable to root from devvp <ptr> (root_device): 2 apfs_vfsop_mountroot:1472: apfs: mountroot failed, error: 2 hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0) : : Darwin Bootstrapper Version 6.0.0: Mon Jul 9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64 boot-args = debug=0x8 kextlog=0xfff cpus=1 rd=md0 Thu Jan 1 00:00:05 1970 localhost com.apple.xpc.launchd[1] <Notice>: Restore environment starting. If you would like to examine iOS’s boot process yourself, here’s how you can try it out. Building QEMU The emulation uses a patched copy of QEMU, which must be compiled from source. Install dependencies To compile QEMU, you first need to install some libraries. macOS: According to the QEMU wiki and the Homebrew recipe, you need to install Xcode and Homebrew, then run brew install pkg-config libtool jpeg glib pixman to install the required libraries to compile QEMU. Ubuntu 18.04: According to the QEMU wiki, run sudo apt install libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev libsdl1.2-dev to install the required libraries to compile QEMU. Windows: QEMU can be built on Windows, but their instructions doesn’t seem to work for this modified QEMU. Please build on macOS or Linux instead. You can set up a virtual machine running Ubuntu 18.04 with Virtualbox or VMWare Player. Download and build source Open a terminal, and run git clone https://github.com/zhuowei/qemu.git cd qemu git submodule init git submodule update mkdir build-aarch64 cd build-aarch64 ../configure --target-list=aarch64-softmmu make -j4 Preparing iOS files for QEMU Once QEMU is compiled, you need to obtain the required iOS kernelcache, device tree, and ramdisk. If you don’t want to extract these files yourself, I packaged all the files you need from iOS 12 beta 4. You can download this archive if you sign up for my mailing list. If you want to extract your own files directly from an iOS update, here’s how: 1. Download the required files: Download my XNUQEMUScripts repository: git clone https://github.com/zhuowei/XNUQEMUScripts.git cd XNUQEMUScripts Download the iOS 12 beta 4 for iPhone X. To decompress the kernel, download newosxbook’s Joker tool. 2. Extract the kernel using Joker: ./joker.universal -dec ~/path/to/iphonex12b4/kernelcache.release.iphone10b mv /tmp/kernel kcache_out.bin replace joker.universal with joker.ELF64 if you are using Linux. 3. extract the ramdisk: dd if=~/path/to/iphonex12b4/048-22007-059.dmg bs=27 skip=1 of=ramdisk.dmg 4. Modify the devicetree.dtb file: python3 modifydevicetree.py ~/Path/To/iphonex12b4/Firmware/all_flash/DeviceTree.d22ap.im4p devicetree.dtb Installing a debugger You will also need lldb or gdb for arm64 installed. macOS The version of lldb included in Xcode 9.3 should work. (Later versions should also work.) You don’t need to install anything in this step. Ubuntu 18.04 I can’t find an LLDB compatible with ARM64: neither the LLDB from the Ubuntu repository nor the version from LLVM’s own repos support ARM64. (Someone please build one!) Instead, you can use GDB on Linux. Two versions of GDB can be used: the version from devkitA64, or the Linaro GDB (recommended). Enter your xnuqemu directory (from the downloaded package or from the clone of the XNUQEMUScripts repo) Run ./linux_installgdb.sh to download the Linaro GDB. Running QEMU Place your qemu directory into the same directory as the scripts, kernel, devicetree, and ramdisk. You should have these files: ~/xnuqemu_dist$ ls README.md lldbit.sh devicetree.dtb lldbscript.lldb devicetreefromim4p.py modifydevicetree.py fixbootdelay_lldbscript_doc.txt qemu gdbit.sh ramdisk.dmg gdbscript.gdb readdevicetree.py kcache_out.bin runqemu.sh linux_installgdb.sh windows_installgdb.sh ./runqemu.sh to start QEMU. $ ./runqemu.sh QEMU 2.12.90 monitor - type 'help' for more information (qemu) xnu in a different terminal, ./lldbit.sh to start lldb, or if you’re using Linux, ./gdbit.sh to start gdb. $ ./lldbit.sh (lldb) target create "kcache_out.bin" Current executable set to 'kcache_out.bin' (arm64). (lldb) process connect --plugin gdb-remote connect://127.0.0.1:1234 (lldb) command source -s 0 'lldbscript.lldb' Executing commands in 'lldbscript.lldb'. (lldb) b *0xFFFFFFF007433BE8 Breakpoint 1: address = 0xfffffff007433be8 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xFFFFFFF005FA5D84 Breakpoint 2: address = 0xfffffff005fa5d84 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xfffffff00743e434 Breakpoint 3: address = 0xfffffff00743e434 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xfffffff00743e834 Breakpoint 4: address = 0xfffffff00743e834 (lldb) Type c into lldb or gdb to start execution. In the terminal running QEMU, you should see boot messages. Congratulations, you’ve just ran a tiny bit of iOS with a virtual iPhone! Or as UnthreadedJB would say, “#we r of #fakr!” What works Booting XNU all the way to running userspace programs Console output from virtual serial port What doesn’t work Wi-Fi Bluetooth USB Screen Internal storage Everything except the serial port You tell me Seriously, though, this only runs a tiny bit of iOS, and is nowhere close to iOS emulation. To borrow a simile from the creator of Corellium, if Corellium is a DeLorean time machine, then this is half a wheel at most. This experiment only finished the easy part of booting iOS, as it doesn’t emulate an iPhone at all, relying on only the parts common to all ARM devices. No drivers are loaded whatsoever, so there’s no emulation of the screen, the USB, the internal storage… You name it: it doesn’t work. For full iOS emulation, the next step would be reverse engineering the iPhone’s SoC to find out how its peripherals work. Unfortunately, that’s a 9-year project, as shown by the development history of Corellium. I can’t do that on my own - that’s why I wrote this tutorial! It’s my hope that this work inspires others to look into proper iOS emulation - from what I’ve seen, it’ll be a great learning experience. How I did this Last week, I started modifying QEMU to load an iOS kernel and device tree: the previous writeup is here. Here’s how I got from crashing when loading kernel modules to fully booting the kernel. Tweaking CPU emulation, part 3: Interrupting cow When we left off, the kernel crashed with a data abort when it tries to bzero a write only region of memory. Why? To confirm that it’s indeed writing to read-only memory, I implemented a command to dump out the kernel memory mappings, and enabled QEMU’s verbose MMU logging to detect changes to the memory map. I tracked down the crashing code to OSKext::updateLoadedKextSummaries. After every kext load, this code resets the kext summaries region to writable with vm_map_protect, writes information for the new kext, then sets the region back to read-only. The logs show that the call to protect the region modifies the memory mappings, but the call to reset it to read-write doesn’t do anything. Why isn’t it setting the page to writable? According to comments in vm_map_protect, it turns out that readonly->readwrite calls actually don’t change the protection immediately, but only sets it on-demand when a program tries - and fails - to write to the page. This is to implement copy on write. So, it seems the data abort exception is supposed to happen, but the panic is not. In the data abort exception, the page should be set to writable in arm_fast_fault. The code in open-source XNU can only return KERN_FAILURE or KERN_SUCCESS, but with a breakpoint, I saw it was returning KERN_PROTECTION_FAILURE. I checked the disassembly: yes, there’s extra code (0xFFFFFFF0071F953C in iOS 12 beta 4) returning KERN_PROTECTION_FAILURE if the page address doesn’t match one of the new KTRR registers added on the A11 processor . I had been ignoring all writes to KTRR registers, so this code can’t read the value from the register (which the kernel stored at startup), and believes that all addresses are invalid. Thus, instead of setting the page to writable, the kernel panics instead. I fixed this by adding these registers to QEMU’s virtual CPU, allowing the kernel to read and write them. After this change, a few more kexts started up, but the kernel then hangs… like it’s waiting for something. Connecting the timer interrupt My hunch for why the kernel hangs: one of the kexts tries to sleep for some time during initialization, but never wakes up because there are no timer interrupts, as shown by QEMU not logging any exceptions when it hangs. On ARM, there are two ways for hardware to signal the CPU: IRQ, shared by many devices, or FIQ, dedicated to just one device. QEMU’s virt machine hooks up the processor’s timer to IRQ, like most real ARM platforms. FIQ is usually reserved for debuggers. Apple, however, hooks up the timer directly to the FIQ. With virt’s timer hooked up to the wrong signal, the kernel would wait forever for an interrupt that would never come. All I had to do to get the timer working was to hook it up to FIQ. This gets me… a nice panic in the Image4 parser. Getting the Image4 parser module working panic(cpu 0 caller 0xfffffff006c1edb8): "could not instantiate ppl environment: 0x60"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleImage4/AppleImage4-1.200.12/include/abort.h:24 What does this mean? What’s error 0x60? I found the panic string, and looked for where the error message is generated. It turns out that the Image4 parser queries the device tree for various nodes in “/chosen” or “/default”; if the value doesn’t exist, it returns error 0x60. If the value is the wrong size, it returns 0x54. iOS’s device tree is missing two properties: chip-epoch and security-domain, which causes the module to panic with the 0x60 error. Oddly, the device tree doesn’t reserve extra space for these properties. I had to delete two existing properties to make space for them. With the modified device tree, the Image4 module initializes, but now I have a panic from a data abort in rorgn_lockdown. Failed attempt to get device drivers to not crash Of course the KTRR driver crashes when it tries to access the memory controller: there isn’t one! QEMU’s virt machine doesn’t have anything mapped at that address. Since I don’t have an emulation of the memory controller, I just added a block of empty memory to avoid the crash. This strategy didn’t work for the next crash, though, from the AppleInterruptController driver. That driver reads and validates values from the device, so just placing a blank block of memory causes the driver to panic. Something more drastic is needed if I don’t want to spend 9 years reverse engineering each driver. Driverless like we’re Waymo To boot XNU, I don’t really need all those drivers, do I? Who needs interrupts or the screen or power management or storage, anyways? All XNU needs to boot into userspace is a serial port and a timer. I disabled every other driver in the kernel. Drivers are loaded if their IONameMatch property corresponds to a device’s “compatible”, “name”, or “device_type” fields. To disable all the drivers, I erased every “compatible” property in the device tree, along with a few “name” and “device_type” properties. Now, with no drivers, XNU seems to hang, but after I patiently waited for a minute… Waiting on <dict ID="0"><key>IOProviderClass</key><string ID="1">IOMedia</string><key>Content</key><string ID="2">Apple_HFS</string></dict> Still waiting for root device It’s trying to mount the root filesystem! Loading a RAMDisk If it’s looking for a root filesystm, let’s give it one. I don’t have any drivers for storage, but I can mount an iOS Recovery RAMDisk, which requires no drivers. All I had to do was: Load the ramdisk at the end of the kernel, just before the device tree blob put its address and size in the device tree so XNU can find it set boot argument to rd=md0 to boot from ramdisk hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0) The kernel mounts the root filesystem! … but then hangs again. Using LLDB to patch out hanging functions By putting breakpoints all over bsd_init, I found that the kernel was hanging in IOBSDSecureRoot, when it tries to call the platform function. The platform function looks for a device, but since I removed all the device drivers, it waits forever, in vain. To fix this, I just skipped the problematic call. I used an LLDB breakpoint to jump over the call and simulate a true return instead. And, after three weeks, the virtual serial port finally printed out: : : Darwin Bootstrapper Version 6.0.0: Mon Jul 9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64 “Houston, the kernel has booted.” What I learned quirks of iOS memory management how iOS handles timer interrupts how iOS loads ramdisks building QEMU on different platforms modifying QEMU to add new CPU configuration registers differences between GDB and LLDB’s command syntax how to get people to subscribe to my mailing list. (muhahaha, one last signup link.) Thanks Thanks to everyone who shared or commented on my last article. To those who tried building and running it - sorry about taking so long to write up instructions! Thanks to @matteyeux, @h3adsh0tzz, @_th0ex, and @enzolovesbacon for testing the build instructions. Thanks to @winocm, whose darwin-on-arm project originally inspired me to learn about the XNU kernel. Sursa: https://worthdoingbadly.com/xnuqemu2/ 1 Quote
