In terms of software, reverse engineering is the process of researching a program to obtain closed information about how it works and what algorithms it uses. While software reversing can be used for legal purposes, in particular, malware analysis or undocumented system research, it’s generally considered to be used by hackers for illegal activities.
Apriorit Research and Reversing team decided to share their professional experience and provide a list of popular simple and advanced techniques you can use to protect your software from illegal reversing. And of course, being ethical hackers, they use this opportunity to flex their muscle and show how an experienced reverser can bypass these protections (including several code samples).
Our reversers note less effective and more effective protection techniques giving some insights about choosing your own set.
Who This Article is Intended For
This article is for all software developers and reverse engineers who are interested in anti-reverse engineering techniques. To understand all the examples and anti-debugging technologies mentioned here, you’ll need Assembler knowledge, some WinDbg experience, and experience developing for Windows using API functions.
Anti-Debugging Method Introduction
There are several approaches to analyzing software:
- Data exchange analysis using a packet sniffer to analyze data exchanged over a network.
- Software binary code disassembly to get its listing in assembly language.
- Decompilation of binary or byte-code to recreate source code in a high-level programming language.
This article considers popular anti-cracking and anti-reverse engineering protection techniques, namely anti-debugging methods in Windows. We should mention right at the beginning that it’s impossible to completely protect software from being reverse engineered. The main goal of various anti-reverse engineering techniques is simply to complicate the process as much as possible.
The best way to be prepared for an attack is to know where one could come from. This article presents popular anti-debugging techniques, starting from the simplest, and notes how to bypass them. We won’t consider different theories of software protection, only practical examples.
PEB (Process Environment Block)
PEB is a closed structure used inside the Windows operating system. Depending on the environment, you need to get the PEB structure pointer in different ways. Below, you can find an example of how to obtain the PEB pointer for x32 and x64 systems:
How to bypass the IsDebuggerPresent check
To bypass the IsDebuggerPresent check, set
BeingDebugged to 0 before the checking code is executed. DLL injection can be used to do this:
Checking for the presence of a debugger in the
main function is not the best idea, as this is the first place a reverser will look when viewing a disassembler listing. Checks implemented in
main can be erased by
NOP instructions thus disarming the protection. If the CRT library is used, the main thread will already have a certain call stack before transfer of control to the
main function. Thus a good place to perform a debugger presence check is in the TLS Callback. Callback function will be called before the executable module entry point call.
In Windows NT, there’s a set of flags that are stored in the global variable
NtGlobalFlag, which is common for the whole system. At boot, the
NtGlobalFlag global system variable is initialized with the value from the system registry key:
How to bypass the NtGlobalFlag check
To bypass the NtGlobalFlag check, just performing reverse the actions that we took before the check; in other words, set the the
NtGlobalFlag field of the PEB structure of the debugged process to 0 before this value is checked by the anti debugging protection.
NtGlobalFlag and IMAGE_LOAD_CONFIG_DIRECTORY
The executable can include the
IMAGE_LOAD_CONFIG_DIRECTORY structure, which contains additional configuration parameters for the system loader. This structure is not built into an executable by default, but it can be added using a patch. This structure has the
GlobalFlagsClear field, which indicates which flags of the
NtGlobalFlag field of the PEB structure should be reset. If an executable was initially created without the mentioned structure or with
GlobalFlagsClear = 0, while on the disk or in the memory, the field will have a non-zero value indicating that there’s a hidden debugger working. The code example below checks the
GlobalFlagsClear field in the memory of the running process and on the disk thus illustrating one of the popular anti debugging techniques:
How to bypass the NtQueryInformationProcess checks
Bypassing the NtQueryInformation Process checks pretty simple. The values returned by the
NtQueryInformationProcess function should be changed to values that don’t indicate the presence of a debugger:
ProcessBasicInformation, change the
InheritedFromUniqueProcessIdvalue to the ID of another process, e.g. explorer.exe
Breakpoints is the main tool provided by debuggers. Breakpoints allow you to interrupt program execution at a specified place. There are two types of breakpoints:
- Software breakpoints
- Hardware breakpoints
It’s very hard to reverse engineer software without breakpoints. Popular anti-reverse engineering tactics are based on detecting breakpoints, providing a series of corresponding anti-debugging methods.
In the IA-32 architecture, there’s a specific instruction – int 3h with the 0xCC opcode – that is used to call the debug handle. When the CPU executes this instruction, an interruption is generated and control is transferred to the debugger. To get control, the debugger has to inject the int 3h instruction into the code. To detect a breakpoint, we can calculate the checksum of the function. Here’s an example:
How to bypass a software breakpoint check
There’s no universal approach for bypassing a software breakpoint check. To bypass this protection, you should find the code calculating the checksum and substitute the returned value with a constant, as well as the values of all variables storing function checksums.
In the x86 architecture, there’s a set of debug registers used by developers when checking and debugging code. These registers allow you to interrupt program execution and transfer control to a debugger when accessing memory to read or write. Debug registers are a privileged resource and can be used by a program only in real mode or safe mode with privilege level CPL=0. There are eight debug registers DR0-DR7:
- DR0-DR3 – breakpoint registers
- DR4 & DR5 – reserved
- DR6 – debug status
- DR7 – debug control
DR0-DR3 contain linear addresses of breakpoints. Comparison of these addresses is performed before physical address translation. Each of these breakpoints is separately described in the DR7 register. The DR6 register indicates, which breakpoint is activated. DR7 defines the breakpoint activation mode by the access mode: read, write, or execute. Here’s an example of a hardware breakpoint check: