RE tools and books

gigiRoman · June 4, 2019

Digital Security Company Blog

Information Security

Reverse engineering

dukebarman August 15, 2017

Favorites: reverse engineering links

Sursa: https://m.habr.com/ru/company/dsec/blog/334832/

Hello!

Today we would like to share our list of materials on reverse engineering (RE). This list is very extensive, because our research department is primarily concerned with the tasks of RE. In our opinion, the selection of materials on the topic is good for the start, while it may be relevant for a long time.

We have been sending this list of links, resources, books for five years to people who would like to get into our research department, but they don’t yet pass by the level of knowledge or just begin their way in the field of information security. Naturally, this list, like most materials / selections, will need updating and updating in some time.

Funny fact: we were shown how some companies send out our list of materials from themselves, but only in a very old edition. And after this publication, they will finally be able to use its updated version with a clear conscience;)

So, let's go to the list of materials!

Topics
a. Reverse
b. Search for vulnerabilities (fuzzing)
c. Exploiting Vulnerabilities
d. Malware Analysis
Tools
a. IDA Pro
b. Radare2
c. WinDBG (Ollydbg / Immunity Debugger / x64dbg)
d. GDB
e. DBI
f. SMT
g. Python to automate
h. BAF (Binary Analysis Frameworks)
Architecture
a. x86-x86_64
b. ARM
OS
a. Windows
b. Linux
c. Mac OS (OSX) / iOS
d. Android
File Formats
a. PE
b. ELF
c. Mach-o
Programming
a. C / C ++
b. Assembler
Practice
a. War games

1. Topics

In this section, we will look at the main areas of RE application. Let's start directly from the reverse development process itself, move on to finding vulnerabilities and developing exploits, and, of course, let's get to malware analysis.

1.a Reverse engineering

Chris Kaspersky’s “The Art of Disassembling” is not new, but a very good and still up-to-date book from Chris with a good systematization of knowledge and excellent material;
" Practical Reverse Engineering: x86, x64, ARM, Windows Kernel, Reversing Tools, and Obfuscation " - a "new" book from several well-known information security specialists covering some new issues and themes that are missing from Chris's book;
" Reversal for Beginners " by Denis Yuryevich is a completely free book, already translated into many languages of the world. Here, probably, the most remarkable thing is the presence of interesting tasks after each chapter, while for several architectures at once;
" Practical RE tips " - an excellent webinar in English from Gynvael Coldwind, containing many useful tips and scripts about RE;
The resource "OPENSECURITYTRAINING.INFO " contains good educational lectures and videos on RE in English;
" Digging Through the Firmware " is a good series of Practical Reverse Engineering articles - useful articles for those who are just about to dive into the world of device firmware reversal;
" Training: Security of BIOS / UEFI System Firmware from Attacker and Defender Perspectives " - if you want to dive into the world of firmware security, UEFI BIOS, then you definitely need to familiarize yourself with these slides that were previously in paid training at leading security conferences;
CRYPTO101 - a little introduction to cryptography, without which it can not do.

1.b Vulnerability Scan

" Fuzzing: Brute Force Vulnerability Discovery" - although not a new book, it’s just right for understanding the basics of fuzzing. There is a translation into Russian, but it contains rather funny blunders;
" Automatic search for vulnerabilities in programs without source texts " - a good introductory material in Russian, presented at PHDays 2011;
" The Evolving Art of Fuzzing " - an article about the development of fuzzing;
" Modern Security Vulnerability Discovery " - a compilation of different techniques for finding vulnerabilities in one document;
" (State of) The Art of War: Offensive Techniques in Binary Analysis " - an all-in-one document on all existing vulnerability scan techniques;
" The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities " is far from a new, but still relevant, book on different approaches to finding vulnerabilities.

1.c Examples of exploiting found vulnerabilities

" Exploit Writing Tutorials by Corelan Team " ( translation ) - a famous series of posts on writing exploits and shellcodes, starting with the basics;
" Exploit Development Community " ( partial translation ) - a series of articles on writing a combat exploit for IE 10 and 11 versions;
" Modern Binary Exploitation " - materials from the RPISEC team from the training course they conducted at the Rensselaer Polytechnic Institute;
" Web-archive of the blog company Vupen " - submerged blog with examples of exploiting complex vulnerabilities in VirualBox, XEN, Firefox, IE10, Windows Kernel, Adobe Flash, Adobe Reader;
" Project Zero " - a blog from the research team of Google, where their experts often share interesting stories on the exploitation of various cool vulnerabilities;
" Browser mitigations against memory corruption vulnerabilities " - protection technologies used in popular browsers:
" SoK: Eternal War in Memory " is an excellent document that shows the attack model and describes various mechanisms to prevent exploitation at different stages for different types of vulnerabilities associated with memory corruption;
" Writing Exploits for Win32 Systems from Scratch " - a detailed article on writing an exploit from scratch for a vulnerability in the SLMAIL program;
Phrack - the famous hacker magazine Phrack. We recommend reading, first of all, the articles of the category "The Art of Exploitation";
" The Shellcoder's Handbook: Discovering and Exploiting Security Holes " is a legendary book on shellcode writing.

1.d Malware Analysis

" Practical Malware Labs " - source for the book " Practical Malware Analysis ";
" Malware Analyst's Cookbook and DVD: Tools and Techniques for Fighting Malicious Code " - we recommend this and the previous book with one set to those interested in this topic;
" Malware Analysis Tutorials: a Reverse Engineering Approach " ( translation ) is a rather long series of articles devoted to setting up an environment with subsequent analysis of malware in it;
" Course materials for Malware Analysis by RPISEC " - another course from RPISEC, only now about malware;
" Computer viruses and antiviruses. Programmer's view " - even though the book examines malicious programs starting from the DOS times, it will still be useful, because besides analyzing the code of such programs, the author shows examples of writing antiviruses for each specific case.

2. Necessary tools

Below are the popular tools used in RE.

2.a IDA Pro

" The IDA Pro Book: The Unofficial Guide to the World Popular Disassembler " is a book that will make your acquaintance with IDA Pro easy and relaxed
" TiGa's Video Tutorial Series on IDA Pro " - a selection of small HOW-TO videos using IDA Pro;
" Open Analysis Live " - in contrast to the previous selection on the use of IDA Pro, this newer and more updated. Mostly, malware analysis is considered.

2.b Radare2

" The radare2 book " - the main book on the use of the Radare2 framework for reverse;
" Radare2 Cheatsheet " - "cheat sheet" for the main teams;
" Radare Today - the blog of radare2 " - framework blog. There are not only news, but also practical examples.

2.c WinDBG (Ollydbg / Immunity Debugger / x64dbg)

Without knowledge of the principles of the debugger and the ability to use it, too, can not do. Below we look at debuggers for Windows OS, and in the next paragraph we will focus on the famous GDB. So, let's go:

Advanced Windows Debugging: Developing and Administering Reliable, Robust, and Secure Software - first of all, this book is useful for understanding and “catching” errors like heap damage;
" Inside Windows Debugging: A Practical Guide to Debugging and Tracing Strategies in Windows " - this edition will well complement the previous book;
“An introduction to cracking from scratch using OllyDbg” - unfortunately, the oldest resource wasm.ru was closed, but such a compilation is easily searched because it has been duplicated into many resources. In addition, "forks" began to appear on the network, only they are already using x64dbg or IDA.

2.d gdb

" gdb Debugging Full Example (Tutorial): ncurses " - a guide for using GDB;
" GEF - GDB Multi-Architecture Enhanced Features for Exploiters & Reverse-Engineers" - add-on GDB over the Python language, adds many useful new commands that will be useful for developing exploits;
" GEF Tutorials " is a series of screencasts on using GEF.

2.e DBI

Programmable debugging is today an indispensable approach in the arsenal of any reverser. And DBI is one of the tools. More details:

" Dynamic Binary Instrumentation inInformation Security " - this article has already collected some generalized information about DBI;
" Light And Dark Side Of Code Instrumentation " - this presentation will help you navigate in the varieties of various code tools and in what and when you can help with the analysis of programs.

2.f SMT

What is the SMT solver? In short, an SMT solver is a program that can solve logical formulas.

The basic idea of using SMT in the field of software security is to translate a program code or algorithm into a logical formula, and then use a SMT solver to test one or another property of this code.

In other words, SMT provides a mathematical tool for semantic code analysis.
SMT solvers have been used in our field for quite some time. They are well established for the following tasks:

search bugs (static analysis / fuzzing);
deobfuscation;
"home" cryptanalysis;
character execution (as an "engine");
There are also some successes in the field of automatic exploit generation (for example, ROP generation).

During this time, SMT lost the aura of mystery, more or less working tools for “ordinary” people appeared.

Below are sources that will help to plunge into the topic:

" SMT Solvers for Software Security, Sean Heelan, Rolf Rolles " - perhaps the first scientific work in which the application of SMT was proposed for solving software security problems. It gives an idea of where and how SMT can find its place in this area;
Z3 is one of the most popular and effective SMT solvers;
- Z3 wiki - project repository;
- " Getting Started with Z3: A Guide " - online tutorial, SMT-solver for experiments;
- Z3Py - binding in Python for Z3;
- " Experimenting with Z3 - Dead code elimination ";
- " Experimenting with Z3 - Proving opaque predicates ";
" Theorem prover, symbolic execution and practical reverse-engineering " - a good overview presentation, with examples of solving real-world problems and using Z3Py;
" Quick introduction into SAT / SMT solvers and symbolic execution " ( Russian version ) is a good book with interesting practical examples.
" An introduction to the use of SMT solvers " - review material.

2.g Python for Automation

Today, without basic knowledge of Python, it will be very difficult, because this programming language is considered the most popular means for automating various tasks in the field of information security (and not only). In addition, it is used in various utilities (for example, all the above utilities allow you to complement the functionality with the help of this PL):

" Gray Hat Python " ( translation ) is a great book that tells you how useful Python is in reverse;
" The Beginner's Guide to IDAPython " - a free book on IDAPython;
" Python Arsenal for Reverse Engineering " is a resource dedicated to various utilities and libraries for reverse engineering using Python.

2.h BAF (Binary Analysis Frameworks)

For a bit more advanced, we recommend paying attention to whole frameworks, which in their composition use the previously mentioned mechanisms and analysis tools for solving more complex problems. So, here they are:

" Overview and Usage of Binary Analysis Frameworks " - a small overview of BAF;

Some interesting frameworks / tools:

3. Architecture

We will cover only a few popular architectures.At the end of the article in the section with additional materials you will find information on many others (MIPS, PowerPC, etc.).

3.a x86-x86_64

" Intel 64 and IA-32 Architectures Software Developer Developers " - previously, such manuals were sent by mail, but because of the large amount of material in them, printing became expensive. Recommended as a desktop reference.

3.b ARM

Azeria Labs (ARM Assembly Basics & ARM Exploit Development) - a site with articles on the basics of ARM-assembler and the development of exploits for this architecture;
The course " Introduction to ARM " - a two-day video course on ARM-development and operation;
VisUAL - visualization of the work of ARM-commands.

4. OS

Knowledge of the principles of work of popular Operating Systems.

4.a Windows

" Windows Internals " - the fundamental book for understanding the work of Windows.
The following items, although mainly related to the exploitation of vulnerabilities in this OS, but allow you to learn more about the insides of Windows:

4.b linux

" Linux insides " is an analogue of the book Windows Internals, but only for OS such as Linux.
As in the case of Windows, the following topics are related to the development of exploits:

4.c Mac OS (OSX) / iOS

" Reverse Engineering Resources Mac and iOS " - a selection of materials on this topic.

4.d Android

" Android Hacker's Handbook " - probably the most popular book dedicated to the safety of the Android OS;
" Android Internals :: Power User's View " - a book that tells about the internal mechanisms of this OS. Due to recent leaks, the material appeared in the public domain, about which the author himself writes on his website and provides an opportunity to download the previous version.

5. Executable file formats

This section provides links explaining the details of popular executable file formats.

5.a PE

" PE sections ";
" PE Title ";
" Windows executable file format. PE32 and PE64 ";
" Computer viruses inside and out ."

5.b ELF

" Linux x64 Infection for Lamers (by a Lamer)."

5.c mach-o

" Parsing mach-o files "

The famous researcher corkami makes very useful and interesting "posters" with the scheme of various file formats, including those mentioned above. We recommend using them as a cheat sheet. A utility Kaitai Sctruct will help in the analysis.

6. Programming

One of our friends once said that a good reverser is 80% a good programmer. The ability to program and understand what is being done and why simplifies the process of researching someone else's program. Therefore, without programming in the reverse nowhere. And of course, the automation of routine tasks, as you probably already understood, is a very useful thing;)

6.a C / C ++

Modern Memory Safety: C / C ++ Vulnerability Discovery, Exploitation, Hardening is a great course with excellent examples. Just must have stuff for everyone.

6.b ASM

" A Crash Course in x86 Assembly for Reverse Engineers " - an "accelerated course" for diving in x86 Assembler, positioned as special for RE;
" Assembly Programming Tutorial " - assembly programming manual, with the ability to run examples online as you study;
" Assembler. 2nd edition " - it is recommended to use as a reference;
" x86 Assembly Guide " - online version.

7. Practice

This section provides links to virtual machines and online resources to practice.

7.a War Games

SmashTheStack Wargaming Network - this multi-wargame network is maintained by volunteers and is available online. We recommend starting with it;
BinTut - local wargame;
Reversing Workshop - a master class on solving tasks from the annual competition "The Flare On Challenge" for 2016;
Exploit-Challenges - a selection of vulnerable ARM binary files;
ARM Reverse Engineering Exercises - the original repository "disappeared", but one of the forks was found on the github expanses;
CTF Time - here you can find out the schedule of future CTF-events and read the solutions of the past.

And finally, a few links with a large number of materials on the above topics:

Selection, generally devoted to the field of information security
Pro exploitation of vulnerabilities
About reverse engineering:
- Awesome-reversing
- REMath Resource Overview
About the exploitation of vulnerabilities in Windows
About phasing
Malware Analysis
And many more different " awesome " collections.

+35

37115

+35

38.3k371

27

Karma

0

Rating

Boris Ryutin @dukebarman

Security researcher

13 subscribers

Share publication

Comments 15

RELATED PUBLICATIONS

August 24, 2015

Sign In