Nytro Posted March 5, 2019 Report Posted March 5, 2019 Millions of Binaries Later: a Look Into Linux Hardening in the Wild February 28, 2019 By Theofilos Petsios TL;DR In this post, we explore the adoption of Linux hardening schemes across five popular distributions by examining their out-of-the-box properties. For each distribution, we analyzed its default kernel configuration, downloaded all its packages, and analyzed the hardening schemes of their enclosed binaries. Our dataset includes the OpenSUSE 12.4, Debian 9, CentOS and RHEL 6.10 & 7 distributions, as well as the Ubuntu 14.04, 12.04, and 18.04 LTS distributions. Our findings confirm that even basic hardening schemes, such as stack canaries and position independent code, are not fully adopted. The situation is even worse when it comes to other compiler protections like stack clash hardening, which recently came into the spotlight due to last month’s systemd vulnerabilities. However, not all is hopeless. A good portion of shipped binaries have basic mitigations in place, and the numbers have improved from version to version. Our experiments indicate that Ubuntu 18.04 shows the largest adoption of OS and application-level mitigations, followed by Debian 9. On the other hand, OpenSUSE 12.4, CentOS 7 and RHEL 7 also deploy common hardening schemes, and show wider adoption stack-clash mitigations while shipping a much more tight-knit set of packages by default. Introduction Shipping quality software is hard. Despite the vast numbers of sophisticated static and dynamic analysis1 toolchains, and the significant compiler and programming language developments of the past years, modern software is still plagued with vulnerabilities that are constantly exploited by attackers for profit. The situation is even worse when it comes to complex and rapidly changing software ecosystems that involve legacy code. In such cases, not only are we faced with the everlasting problem of finding possibly exploitable bugs, but we also are constrained by the hard limits of backwards-compatibility, which often dictate that we are stuck with a piece of code that is known to have limitations, or even worse, be vulnerable or buggy. This is where hardening techniques come into play. Since we cannot prevent certain types of bugs, we might as well make the attacker’s life harder, and work around the problem by preventing or hindering the exploitation of these bugs. Such software hardening defenses are currently deployed in all modern operating systems, however vary greatly in complexity, effectiveness an overhead: from stack canaries and ASLR to full-fledged CFI and ROP defenses, hardening schemes are not all designed, implemented, or used equally. In this blog post, we will take a deep look into the adoption of such defenses by the most popular Linux distributions, examining their default kernel configurations as well as the properties of the binaries distributed through the package management systems of each distribution. CVEs vs Security We all have seen articles bearing titles such as “most vulnerable applications of the year” or “most vulnerable operating systems”. Usually, these articles present statistics on Common Vulnerability and Exposures (CVE) entries, aggregated from sources like NIST’s National Vulnerability Database (NVD) and subsequently rank the compared applications or OSes based on the numbers of CVEs that have been published. Unfortunately, although very useful for keeping track of issues and informing vendors and users alike, CVEs alone do not tell us much about the underlying security properties of software. To make this clearer, let us examine the total number of CVEs published over the past four years for the Linux Kernel, as well as for five of the most popular server-oriented Linux distributions, namely Ubuntu, Debian, Red Hat Enterprise Linux, and OpenSUSE. Figure 1 What do we learn from the graph above? Is the number of CVEs per distribution indicative of the fact that one distribution might be more vulnerable than another? The answer is no. For instance, as you will see in this post, empirical evidence suggests that Debian has more hardening mechanisms in place compared to say, OpenSUSE or RedHat Linux, and yet it has the most CVEs. However, these CVEs do not necessarily denote weakened security: even in the presence of a CVE, it is not directly obvious if a vulnerability is exploitable. Severity score assignments do provide an indication of how likely a vulnerability is to be exploited but, at the end of the day, exploitability depends largely on the defenses present in the affected systems, as well as on attackers’ resources and capabilities. Moreover, the absence of CVE reports does not tell us anything about other unreported or unknown vulnerabilities that may be lurking, and differences in numbers may exist not because of differences in software quality but due to other factors, including the resources allocated to testing or the size of the user base. For our Debian example, the higher number of CVEs may simply indicate that Debian ships more software packages. That said, it goes without saying that the CVE enumeration system provides us with useful information that enables us to build appropriate defenses: the better we understand how software fails, the more likely we are to pinpoint the possible ways attackers might exploit it, and design the appropriate detection and response mechanisms to defend against them. To this end, let us examine, in Figure 2, the categories (in aggregate, across all distributions) for the CVEs of Figure 1 (sourced from here) over the last 4 years. It is immediately obvious that the majority of the CVEs reported over the past years fall into the following categories (as defined by the cvedetails categorization😞 Denial of Service (DoS), Code Execution, Overflow, Memory Corruption, Information Exfiltration, and Privilege Escalation. Although many of the CVEs are counted multiple times across various categories, we notice that overall, the same core problems persist year-to-year. In the following of this post, we will evaluate the adoption of different hardening schemes that are deployed by mainstream Linux distributions to prevent exploitation of the above vulnerabilities. Figure 2 Goals With this survey, we set out to answer the following questions: What are the security properties of different Linux distributions? What hardening defenses are in place when it comes to the kernel, and for user-space applications? How has the adoption of hardening mechanisms changed over time for different distributions? What are the average package and library dependencies for each distribution? What hardening defenses are in place for each binary? Distribution Selection It turns out that finding accurate statistics on adoption and deployment of different distributions is not trivial, since in most cases the number of downloads is not indicative of the number of real deployments. That said, UNIX variants represent the majority when it comes to server/infrastructure deployments, with Linux variants having a continuously growing market share2. Thus for our study, we focused on the different Linux distributions that are available out-of-the box in Google’s Cloud Platform (GCP). In particular, the Operating Systems we selected in our study were: Distribution / Release Kernel Build OpenSUSE 12.4 4.12.14-95.3-default #1 SMP Wed Dec 5 06:00:48 UTC 2018 (63a8d29) Debian 9 (stretch) 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) CentOS 6.10 2.6.32-754.10.1.el6.x86_64 #1 SMP Tue Jan 15 17:07:28 UTC 2019 CentOS 7 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 Red Hat Enterprise Linux Server 6.10 (Santiago) 2.6.32-754.9.1.el6.x86_64 #1 SMP Wed Nov 21 15:08:21 EST 2018 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 UTC 2018 Ubuntu 14.04 (Trusty Tahr) 4.4.0–140-generic #166~14.04.1-Ubuntu SMP Sat Nov 17 01:52:43 UTC 20… Ubuntu 16.04 (Xenial Xerus) 4.15.0–1026-gcp #27~16.04.1-Ubuntu SMP Fri Dec 7 09:59:47 UTC 2018 Ubuntu 18.04 (Bionic Beaver) 4.15.0–1026-gcp #27-Ubuntu SMP Thu Dec 6 18:27:01 UTC 2018 Table 1 Analysis In order for us to analyze the hardening defenses deployed by the different OSes outlined above, we examine their vanilla kernel configuration, as well as the properties of the packages that are available through each distribution’s package manager out-of-the-box. Thus, we only examined packages available from each distribution’s default mirrors, and omitted packages available from non-stable repositories (e.g., ‘testing’ mirrors in Debian) or third-party packages (e.g., NVIDIA packages available through non-default mirrors). Also, we did not consider custom kernel compilations or configurations that offer increased hardening. Kernel Configuration Analysis Using our analysis script based off an open-source kconfig checker, we examined the hardening config options that came out-of-the-box with the above distributions, and compared them against the list published by the Kernel Self Protection Project (KSPP). For each kernel hardening configuration option, Table 2 outlines the desired setting, and has a checkmark for all the distributions that conformed to the KSSP recommendation3. Kernel Configuration Setting Desired Value SLES Debian 9 CentOS 6.10 CentOS 7 RHEL 6.10 RHEL 7.6 Ubuntu 14.04 Ubuntu 16.04 Ubuntu 18.04 Significance X86_SMAP y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical STRICT_KERNEL_RWX y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical RANDOMIZE_BASE y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical RANDOMIZE_MEMORY y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical STACKPROTECTOR_STRONG y ✔️ ✔️ ✔️ ✔️ ✔️ Critical HARDENED_USERCOPY y ✔️ ✔️ ✔️ ✔️ ✔️ Critical LOCK_DOWN_KERNEL y ✔️ ✔️ ✔️ Critical STRICT_MODULE_RWX y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical SECURITY y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical SECCOMP y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical STRICT_DEVMEM y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical DEVKMEM not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Critical X86_INTEL_UMIP y ✔️ ✔️ ✔️ ✔️ High VMAP_STACK y ✔️ ✔️ ✔️ ✔️ High SLAB_FREELIST_HARDENED y ✔️ ✔️ High SLAB_FREELIST_RANDOM y ✔️ ✔️ ✔️ ✔️ High FORTIFY_SOURCE y ✔️ ✔️ High BUG_ON_DATA_CORRUPTION y High HARDENED_USERCOPY_FALLBACK not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High SECURITY_DMESG_RESTRICT y ✔️ ✔️ High SECURITY_YAMA y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High SECURITY_SELINUX_DISABLE not set ✔️ ✔️ ✔️ High SECCOMP_FILTER y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High ACPI_CUSTOM_METHOD not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High COMPAT_BRK not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High IO_STRICT_DEVMEM y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High LEGACY_VSYSCALL_NONE y High USERFAULTFD not set ✔️ ✔️ High LIVEPATCH not set ✔️ ✔️ High BPF_JIT not set ✔️ ✔️ High PAGE_TABLE_ISOLATION y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High RETPOLINE y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High X86_64 y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High DEBUG_WX y ✔️ ✔️ ✔️ High SCHED_STACK_END_CHECK y ✔️ ✔️ ✔️ ✔️ ✔️ High MODULE_SIG y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High REFCOUNT_FULL y High STATIC_USERMODEHELPER y High COMPAT_VDSO not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ High BINFMT_MISC not set High PROC_KCORE not set High MODIFY_LDT_SYSCALL not set ✔️ ✔️ ✔️ ✔️ High KPROBES not set High UPROBES not set ✔️ ✔️ High DEBUG_FS not set High BPF_SYSCALL not set ✔️ ✔️ High USER_NS not set High FTRACE not set High ARCH_MMAP_RND_BITS 32 High BUG y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium THREAD_INFO_IN_TASK y ✔️ ✔️ ✔️ ✔️ Medium MODULE_SIG_ALL y ✔️ ✔️ ✔️ ✔️ ✔️ Medium PAGE_POISONING y ✔️ Medium GCC_PLUGIN_RANDSTRUCT y Medium HIBERNATION not set Medium PROC_VMCORE not set Medium HWPOISON_INJECT not set Medium SLUB_DEBUG y ✔️ ✔️ ✔️ ✔️ ✔️ Medium SYN_COOKIES y ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium DEFAULT_MMAP_MIN_ADDR 65536 ✔️ ✔️ ✔️ ✔️ ✔️ Medium GCC_PLUGIN_LATENT_ENTROPY y Medium DEBUG_LIST y ✔️ ✔️ ✔️ ✔️ ✔️ Medium DEBUG_CREDENTIALS y Medium MODULE_SIG_FORCE y Medium GCC_PLUGIN_STACKLEAK y Medium SECURITY_LOADPIN y Medium PAGE_POISONING_NO_SANITY not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium PAGE_POISONING_ZERO not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium SLAB_MERGE_DEFAULT not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium X86_PTDUMP not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium DEBUG_KMEMLEAK not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Medium KEXEC not set Medium LEGACY_PTYS not set ✔️ ✔️ ✔️ ✔️ ✔️ Medium IA32_EMULATION not set Medium PROC_PAGE_MONITOR not set Medium GCC_PLUGIN_STRUCTLEAK y Medium GCC_PLUGIN_STRUCTLEAK_BYREF_ALL y Medium DEBUG_SG y Medium SLUB_DEBUG_ON y Medium INET_DIAG not set Medium X86_X32 not set ✔️ ✔️ ✔️ ✔️ ✔️ Medium USELIB not set ✔️ ✔️ ✔️ ✔️ Medium CHECKPOINT_RESTORE not set ✔️ ✔️ Medium MEM_SOFT_DIRTY not set ✔️ ✔️ Medium MMIOTRACE not set ✔️ ✔️ ✔️ ✔️ ✔️ Medium KEXEC_FILE not set ✔️ ✔️ Medium DEBUG_NOTIFIERS y Low ZSMALLOC_STAT not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Low PAGE_OWNER not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Low BINFMT_AOUT not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Low IP_DCCP not set ✔️ Low IP_SCTP not set Low DEVPORT not set Low NOTIFIER_ERROR_INJECTION not set ✔️ ✔️ ✔️ ✔️ ✔️ Low ACPI_TABLE_UPGRADE not set ✔️ ✔️ ✔️ ✔️ ✔️ Low ACPI_APEI_EINJ not set ✔️ Low PROFILING not set Low GCC_PLUGINS y Low MMIOTRACE_TEST not set ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ Low MODULE_SIG_SHA512 y ✔️ ✔️ ✔️ Low Total SLES Debian 9 CentOS 6.10 CentOS 7 RHEL 6.10 RHEL 7.6 Ubuntu 14.04 Ubuntu 16.04 Ubuntu 18.04 Critical (out of 12) 9 11 4 11 4 11 8 12 12 High (out of 37) 14 16 15 12 15 12 11 18 18 Medium (out of 37) 11 11 15 14 15 14 10 10 10 Low (out of 14) 5 5 6 6 6 6 6 5 5 Total 39 43 40 43 40 43 35 45 45 Table 2 Overall, we notice that newer kernels have stricter settings out-of-the-box. For instance, CentOS 6.10 and RHEL 6.10 running a kernel version 2.6.32 lack most of the critical features implemented in newer kernels such as SMAP, strict RWX permissions, base address randomization, or copy2usr protections. We should note at this point that, although many of the configuration options presented in Table 2 are not present for older kernel versions and are in reality not applicable, we count them as having the mitigation not present, and do not differentiate the two – likewise, if a configuration option is not present for a given kernel version and the desired setting is “not set”, this is still counted as a sane configuration. Another point we need to address when interpreting the above results is that some of the kernel configuration that increase the attack surface for attackers can also be utilized to implement security defenses. Such examples include uprobes and kprobes, kernel modules and BPF/eBPF. Our recommendation is to use the above mechanisms to actually provide defense-in-depth, as they are not trivial to exploit and their abuse already requires malicious actors to have a foothold on the system. However, if enabled, the system administrator must actively monitor for their abuse. Examining further the entries of Table 2, we observe that modern kernels provide several options that can hinder exploitability of vulnerabilities like information leaks and stack/heap overflows. However, we notice that even the latest commodity distributions do not yet deploy more sophisticated defenses (e.g., shipping with grsecurity patches), or state-of-the art defenses against code reuse attacks (e.g., combining randomization with R^X schemes for code). To make matters worse, even these more advanced defenses do not protect against the full spectrum of attacks. Thus, it is critical for system administrators to complement sane configurations with solutions that offer runtime detection and prevention of exploits. Application Analysis It comes as no surprise that different distributions have different package characteristics, compilation options, library dependencies, etc. Not only that, but differences exist even for distributions with close ancestry and packages with small amounts of dependencies (e.g., coreutils in Ubuntu vs. Debian). In order for us to evaluate the differences in the Linux distributions available in GCP, we downloaded all the available packages, extracted their contents, and analyzed their binaries and dependencies. For each package, we kept track of the other packages upon which it depends, and for each binary we kept track of its library dependencies. We outline our findings in this section. Distribution Packages In total, we downloaded 361,556 packages across all distributions, only fetching the packages available via the default mirrors. From the available packages of each distribution , we ignored packages that did not contain ELF executables, such as source packages, fonts, etc. After filtering out irrelevant packages, we analyzed a total of 129,569 packages, containing a total of 584,457 binaries. The total packages and binaries analyzed per distribution are presented in Figure 3. Figure 3 We notice that the more modern a distribution is, the more packages and binaries are added, which comes as no surprise. That being said, Ubuntu and Debian packages include many more binaries (both executables as well as dynamic modules and libraries) than those of CentOS, SUSE and RHEL, which is potentially impactful on Ubuntu and Debian’s attack surfaces4. This is particularly important given the fact often packages have dependencies to other packages, and thus, a vulnerability in a binary of a given package can affect many parts of the software ecosystem, similarly to how a vulnerable library might possibly affect all binaries importing it. As a point of reference, we present the distribution of the number of dependencies per package for the different OSes in Figure 4: Figure 4 We notice that for almost all distributions, 60% of the packages have at least 10 dependencies. Moreover, we notice that a few packages have large numbers of dependencies (> 100). The same applies to the reverse dependencies of a package: as expected, there are few packages which are used by many packages across the distribution, and therefore vulnerabilities in those select few are of high risk. As an example in the following table, we present the top 20 packages with the most reverse dependencies for SLES, Centos 7, Debian 9, and Ubuntu 18.04 (each cell denotes the package and the number of reverse dependencies). SLES 12.4 CentOS 7 Debian 9 Ubuntu 18.04 bc (3768) glibc-0:2.17-260.el7_6.3.x86_64 (7065) libc6 (23246) libc6 (21592) glibc (3667) glibc-0:2.17-260.el7.i686 (7036) libstdc++6 (7703) libstdc++6 (6575) dconf (1336) libgcc-0:4.8.5-36.el7.x86_64 (2433) libgcc1 (7184) libgcc1 (5683) at (1118) libstdc++-0:4.8.5-36.el7.x86_64 (2127) zlib1g (3416) libglib2.0-0 (3037) ed (1053) bash-0:4.2.46-31.el7.x86_64 (1664) libglib2.0-0 (3309) zlib1g (2485) sed (913) glib2-0:2.56.1-2.el7.x86_64 (1190) libgmp10 (2079) libgmp10 (1926) file (894) zlib-0:1.2.7-18.el7.x86_64 (1074) libx11-6 (1962) libx11-6 (1576) rpm (807) libX11-0:1.6.5-2.el7.x86_64 (638) libqt5core5a (1751) libqt5core5a (1486) pkg-config (746) cairo-0:1.15.12-3.el7.x86_64 (597) libcairo2 (1614) ruby (1315) coreutils (731) openssl-libs-1:1.0.2k-16.el7.x86_64 (597) libgdk-pixbuf2.0-0 (1547) nodejs (1298) perl (626) gdk-pixbuf2-0:2.36.12-3.el7.x86_64 (575) libpango-1.0-0 (1483) libqt5gui5 (1123) gcc (608) pango-0:1.42.4-1.el7.x86_64 (571) ruby (1452) dpkg (1045) python (542) atk-0:2.28.1-1.el7.x86_64 (495) nodejs (1434) libqt5widgets5 (984) xli (389) libxml2-0:2.9.1-6.el7_2.3.x86_64 (484) libqt5gui5 (1362) libghc-base-dev-4.9.1.0-d28d6 (898) tk (375) perl-4:5.16.3-294.el7_6.x86_64 (463) libpangocairo-1.0-0 (1295) libgdk-pixbuf2.0-0 (892) gd (309) python-0:2.7.5-76.el7.x86_64 (463) libatk1.0-0 (1202) python (874) bash (290) freetype-0:2.8-12.el7.x86_64 (417) libqt5widgets5 (1180) libgtk2.0-0 (869) systemd (236) systemd-0:219-62.el7.x86_64 (401) libatomic1 (1109) libgtk-3-0 (795) info (229) libgcc-0:4.8.5-36.el7.i686 (393) libxml2 (1101) libcairo2 (793) Table 3 Interestingly, although all the OSes analyzed were built for x86_64 architecture, and the majority of the packages also have a specified architecture of x86_64 and x86, we discovered that the packages often contained binaries for other architectures, as seen in Figure 5. Figure 5 In the following section we will dive deeper into the characteristics of the binaries analyzed. Binary Hardening Statistics At absolute minimum, we wanted to examine a basic set of hardening options for the binaries at hand. Several Linux distributions ship with scripts that perform such hardening checks. For instance, Debian/Ubuntu exposes a hardening-check script that provides such useful information. Here’s an example: $ hardening-check $(which docker) /usr/bin/docker: Position Independent Executable: yes Stack protected: yes Fortify Source functions: no, only unprotected functions found! Read-only relocations: yes Immediate binding: yes We notice that the above script checks for the following 5 hardening features: Position Independent Executable (PIE): Indicates if the text section of the program can be relocated in memory, so as to achieve randomization if ASLR is enabled in the kernel. Stack Protected: Whether stack canaries are in place to guard against stack smashing attacks. Fortify Source: Whether unsafe functions (e.g., strcpy) are replaced with their safer counterparts and calls that are verifiable at runtime replace their non-runtime-checked counterparts (e.g., __memcpy_chk vs memcpy). Read-only relocations (RELRO): Whether relocation table entries are marked as “read-only” if they were resolved before execution begun. Immediate binding: Whether the runtime linker resolves all relocations before starting program execution (this is equivalent to full RELRO). Are the above hardening mechanisms enough? Unfortunately, not at all. There are known techniques to bypass all the above defenses, but the more hardening defenses are in place, the higher the bar for the attacker. For instance, techniques to bypass RELRO mitigations are hindered if PIE and immediate binding are present. Likewise, full ASLR requires attackers to perform additional work to construct a working exploit. However, for sophisticated attackers, the above mitigations are something to expect — their absence in essence opens the door to your system much faster. Therefore, it is critical that at minimum they are put in place. For our analysis, we wanted to examine what percentage of the binaries in the distributions under examination had these mitigations in place, as well as the presence of three more mitigations: Non executable bit (NX), which prevents execution on any region that should not be executable such as the stack heap etc. RPATH/RUNPATH, which designates the run-time search path used by the dynamic loader to find the appropriate libraries. The former is a sine qua non for any modern system – its absence allows attackers to arbitrarily write a payload in memory and execute it as is. For the latter, non-sane runtime path configurations are responsible for introducing untrusted code that may lead to series of problems (e.g., privilege escalation as well as other issues). Stack Clash protection, which provides protection against attacks that cause the stack to overlap with other memory regions (e.g., heap). Given the recent exploits abusing stack clash vulnerabilities in systemd, we thought it relevant to include this mitigation in our dataset. So, without further ado, let us jump into the numbers: Tables 4 and 5 present the summary of our analysis for the executables and libraries of the different distributions, respectively.: We notice that with respect to NX enforcement, it is almost omnipresent, with few exceptions. Particularly, we notice slightly lower adoption in Ubuntu and Debian distributions compared to CentOS, RHEL and OpenSUSE. When it comes to stack canaries, we notice that they are missing from significant portions of the total executables, especially in distributions with older kernels, however there has been progress in latest Centos, RHEL, Debian and Ubuntu distributions. With the exception of Debian and Ubuntu 18.04, PIE support is poor in most distributions. Stack clash mitigations are scarce in OpenSUSE, Centos 7 and RHEL 7 and practically non-existent elsewhere. All distributions with modern kernels have some support for RELRO, with Ubuntu 18.04 leading the way, and Debian at second place. As mentioned earlier, the metrics presented in this table are averages across all different versions of a binary. Thus, we observe differences in say, PIE statistics compared to examining only the latest version of a binary (see for instance Debian’s progress with PIE packages). Moreover, whereas most distributions generally examine if at least some functions are fortified in a binary when counting the percentage of fortified binaries, for our analysis we measure a true percentage of the fortified functions. Therefore, if for a binary 5 out of 50 fortifiable functions are fortified, we will assign it a score of 0.1, as 10% of the functions are fortified. OS Canaries Stack Clash NX PIE RELRO (full) RELRO (partial) RUNPATH FORTIFIED SLES 12.4 55.34 1.15 99.29 6.33 7.30 89.84 85.62 41.04 CentOS 6.10 55.73 0.00 99.66 2.18 2.96 6.26 98.73 41.11 CentOS 7 84.55 0.15 99.83 5.83 11.80 88.00 98.44 41.51 RHEL 6.10 54.32 0.00 99.43 2.20 2.85 6.87 98.78 37.31 RHEL 7.6 82.26 0.13 99.56 5.57 10.93 88.89 98.45 36.66 Debian 9 73.03 0.01 97.78 38.04 35.34 62.15 88.48 31.10 Ubuntu 14.04 53.85 0.01 96.12 4.04 9.53 85.82 99.08 44.12 Ubuntu 16.04 75.58 0.01 95.84 4.76 12.01 85.09 97.54 43.54 Ubuntu 18.04 85.62 0.01 93.43 37.77 56.81 41.46 88.54 41.00 Table 4: Hardening Properties of Fig. 3 Executables (Percentage of Adoption) OS Canaries Stack Clash NX RELRO (full) RELRO (partial) RUNPATH FORTIFIED SLES 12.4 49.46 0.37 99.49 6.41 90.41 85.05 35.29 CentOS 6.10 53.15 0.00 99.91 2.72 5.37 98.66 38.07 CentOS 7 83.12 0.06 99.84 10.49 89.39 99.06 38.10 RHEL 6.10 52.28 0.00 99.91 2.51 5.96 98.71 38.06 RHEL 7.6 82.83 0.07 99.89 9.45 90.47 98.88 38.31 Debian 9 74.46 0.01 96.94 33.17 63.88 89.02 33.00 Ubuntu 14.04 43.37 0.01 94.60 9.40 85.00 98.79 38.89 Ubuntu 16.04 67.37 0.01 93.29 12.23 83.76 96.73 35.95 Ubuntu 18.04 81.96 0.01 89.57 32.01 65.97 88.35 33.81 Table 5:Hardening Properties of Fig. 3 Libraries (Percentage of Adoption) So are things progressing? They definitely are: this can be seen from statistics on individual distributions (e.g., Debian), as well as from the above tables. As a use case, we present a summary of the hardening mechanism adoption for three consecutive Ubuntu LTS distributions in Figure 5 (we omit stack-clash statistics due to the low adoption). We notice that consistently more binaries have stack canaries from version to version, whereas significantly more binaries in 18.04 are shipped with full RELRO compared to previous versions. Ubuntu Executables Ubuntu Libraries Figure 6 Unfortunately, multiple executables are present still in the different distributions, having none of the above mitigations. For instance, taking a quick look at Ubuntu 18.04, we notice the ngetty binary (which is a getty replacement) ships without canaries, NX, PIE and fortified source, and so do the mksh and lksh shells, the picolisp interpreter as well as the packages nvidia-cuda-toolkit (which is a popular package for creating GPU-accelerated applications like machine learning frameworks) and klibc-utils. Similarly, the mandos-client binary (which is an administrative tool allowing unattended reboots with encrypted root filesystems), as well as the rsh-redone-client (a re-implementation of rsh and rlogin), ship without NX and have SUID set :(. Likewise, several suid-binaries lack basic protections like stack canaries (e.g., the Xorg.wrap binary of the Xorg package). Summary & Closing Remarks In this post, we highlighted several characteristics of modern Linux distributions when it comes to hardening. Our analysis demonstrated that the latest Ubuntu LTS distribution (18.04) has on average the strongest OS-level and application mitigations amongst the distributions with newer kernels that we examined such as Ubuntu 14.04, 12.04 and Debian 9. The examined CentOS, RHEL and OpenSUSE distributions in our dataset, however, ship a tighter set of packages by default, and in their latest versions (for CentOS and RHEL) have higher numbers of stack clash adoption compared to their debian-based rivals (Debian and Ubuntu). Comparing CentOS and RedHat versions, we notice large improvements in deployment of stack canaries and RELRO from versions 6 to 7, however on average more functions are fortified in CentOS than in RHEL. Overall, we notice that the most progress to be made across all distributions is with regards to adoption of PIE, which, with the exception of Debian 9 and Ubuntu 18.04 is below 10% for the binaries of our dataset. Finally we should note that while custom analysis such the above is possible, there is a plethora of security tools (e.g., Lynis, Tiger, Hubble) that may be used for analysis of a system and to prevent insecure OS and application configurations. Unfortunately, even with hardening mitigations and sane configurations, vulnerabilities eventually will lead to exploitation. This is why we strongly believe that it is vital to have solid real-time monitoring and prevention of attacks, by focusing and preventing exploitation patterns. We will explore such exploitation patterns in future posts, as well as how one defend against them. Stay tuned! Notes 1. Static analysis tools examine code properties without executing the software, by only examining the source code or generated binary. Dynamic techniques require execution of a piece of software and perform runtime analysis.↩2. W3techs reports that some UNIX variant is used by 69.2% of all websites, and the percentage is even larger when it comes to supercomputers. A multitude of other sources (e.g., 1, 2, 3) verify this trend, which is especially true for container deployments — even back in 2016, Ubuntu alone was reported to have over 60 million images launched by docker users.↩3. For a thorough explanation of the utility of the above settings, the interested reader may refer to the KSPP project or to our kernel configuration glossary. In future posts, we will elaborate more on why many of these came to exist, as well as on possible ways to exploit the system in their absence.↩4. We should note that the numbers presented include all binaries across different versions of a software package. For instance, a dynamic module used by Python will be analyzed multiple times, separately for python-2.6, 2.7 etc. Our averages are presented across all packages.↩ Theofilos Petsios( Research Scientist ) Theofilos Petsios is a Research Scientist at Capsule8. His interests include systems and application security, binary analysis and privacy. Over the years, he has presented his research at various conferences internationally, including IEEE Security & Privacy, ACM CCS, ACM EuroSys and others. https://github.com/nettrino Capsule8 Labs Sursa: https://capsule8.com/blog/millions-of-binaries-later-a-look-into-linux-hardening-in-the-wild/ 1 Quote
