Saturday, August 5, 2017

Statically Compile Tor

The Onion Router (Tor) is a great open-source tool to help protect your privacy online. It's useful for journalists in hostile nations, and allows us the freedom to find information online without being tracked via IP address. It does this by bouncing your connections through multiple encrypted routers on the Tor network to hide the source of the request.

At Nullable Security, we sometimes use Tor as a transport mechanism for client penetration tests. We had a customer who wanted to simulate how an APT-style attack would look like when it originates from, or connects to, the Tor network. Most organizations don't have the level of network awareness to alert on such communications, and Tor traffic tends to go unnoticed. Whenever we use the Tor client in a customer engagement, we always want the software to be as up-to-date and portable as possible. And we of course want to be transparent on the software we're using in their network. This post will describe how we wrap up Tor for easy deployment.

The Tor software comes in three major forms: a bowser bundle, an expert bundle, and source code. We'll be focusing on the Windows variants of Tor for this post. Most people use the browser bundle, which is a combination of the Tor client and a hardened version of Firefox. The expert bundle comes with just the Tor client. And source code contains all the code for Tor and it's accessories. We have little use for the full browser bundle because we don't usually need a graphical browser for our pentesting tools. All we really need is a locally listening Tor socket, which the expert bundle gives us. But when you download the Tor expert bundle, you'll see that it contains two top-level folders: Data & Tor. Data contains geoip data, and isn't required for the Tor client to run. The Tor folder contains two executables and eight DLLs. Of the two executables, we only care about tor.exe, which needs six of those DLLs. But we only want to deploy the single tor.exe, and not all the required DLLs. To solve this, we will statically compile the Tor source code, and include all the necessary DLL code into a single Tor binary. This also lets us use the most modern source code for Tor and its dependencies. We'll begin by getting the necessary pieces of software we'll need to compile Tor.

MSYS2 is a software build environment similar to Cygwin, which we'll use to build Tor on Windows 10. Run the MSYS2 installer and accept all the defaults. We used the 64-bit 20161025 version of MSYS2. This will open the MSYS command prompt, where we will install some packages needed to compile Tor. The pacman command will be used to fetch updates and dependencies. Run this command twice to ensure everything updated.

   pacman -Syu

Now pull the build environment.

   pacman -S msys/make msys/perl msys/tar
   pacman -S mingw32/mingw-w64-i686-binutils msys/binutils
   pacman -S mingw32/mingw-w64-i686-gcc
   pacman -S mingw32/mingw-w64-i686-make
   pacman -S msys/pkg-config mingw32/mingw-w64-i686-pkg-config

Open the mingw32 console (C:\msys64\msys2_shell.cmd -mingw32) and enter these commands to download the Tor source code and it's dependencies.

   mkdir openssl && mkdir libevent && mkdir zlib && mkdir tor
   tar xvf openssl-1.0.2l.tar.gz -C openssl
   tar xvf zlib-1.2.11.tar.gz -C zlib
   tar xvf tor- -C tor

Set a few build environment variables

   export INCLUDE_PATH="/mingw32/include:/mingw32/i686-w64-mingw32/include:$INCLUDE_PATH"
   export LIBRARY_PATH="/mingw32/lib:/mingw32/i686-w64-mingw32/lib:$LIBRARY_PATH"
   export BINARY_PATH="/mingw32/bin:/mingw32/i686-w64-mingw32/bin:$BINARY_PATH"

Compile zlib

   cd ~/zlib/zlib-1.2.11
   make -f win32/Makefile.gcc

Compile libevent

   cd ~/libevent/libevent-2.1.8-stable/
   ./configure --prefix="$HOME/libevent/install" --enable-static --disable-shared
   make && make install-strip

Compile OpenSSL

   cd ~/openssl/openssl-1.0.2l
   LDFLAGS="-static" ./Configure no-shared no-zlib no-asm --prefix="$HOME/openssl/install" -static mingw
   make depend && make && make install
Compile Tor

   cd ~/tor/tor-
   export LDFLAGS="-static -L $HOME/openssl/install/lib -L $HOME/libevent/install/lib -L $HOME/zlib/zlib-1.2.11 -L /mingw32/lib -L /mingw32/i686-w64-mingw32/lib"
   export CFLAGS="-I $HOME/openssl/install/include -I $HOME/zlib/zlib-1.2.11 -I $HOME/libevent/install/include"
   export LIBRARY_PATH="$HOME/openssl/install/lib:$HOME/libevent/install/lib:$HOME/zlib/zlib-1.2.11:/mingw32/lib:/mingw32/i686-w64-mingw32/lib"
   export INCLUDE_PATH="$HOME/openssl/install/include:$HOME/zlib/zlib-1.2.11:$HOME/libevent/install/include:/mingw32/include:/mingw32/i686-w64-mingw32/include"
   export BINARY_PATH="/mingw32/bin:/mingw32/i686-w64-mingw32/bin"
   export PKG_CONFIG_PATH="$HOME/openssl/install/lib/pkgconfig:$PKG_CONFIG_PATH"
   export LIBS="-lcrypt32"
   ./configure --disable-gcc-hardening --enable-static-tor --prefix="$HOME/tor/install" --with-libevent-dir="$HOME/libevent/install/lib" --with-openssl-dir="$HOME/openssl/install/lib" --with-zlib-dir="$HOME/zlib/zlib-1.2.11"
   make && make install-strip

And now we have a tor.exe binary in our MSYS2 home build directory. You can find that at the default path of C:\msys64\home\<username>\tor\install\bin. Here's a demonstration of Tor successfully bootstrapping to the network, using only the newly compiled tor.exe binary. And a bonus attribute about your new binary - since it's custom compiled, can be renamed/packed/crypted, and made to listen on any port - it's more likely to bypass AV signatures

This executable can be passed a standard torrc configuration file, and may be easily embedded in other malware. For Blue-Team'ers, the most effective way to detect something like this is to watch for Tor traffic on the wire. Creating firewall blacklists and alerts for traffic to known Tor entry guard nodes will show you which hosts on your network may be running Tor enabled software. Monitoring for Tor exit node traffic will show you connections originating from the Tor network. You can retrieve IP lists from the Tor project and other third-parties here.

Sunday, July 23, 2017

Fuzzing Nginx

Nginx is a popular web server software used by over 130 million websites. Of the 10,000 busiest websites, most are running on Nginx. Due to its vast deployment, the importance of this software cannot be overstated. Because of this, we have decided to evaluate the ruggedness of the product to search for unknown security vulnerabilities. This post will describe how to set up and use a fuzzing environment to search for bugs in Nginx.

Fuzzing is the technique of sending malformed data to a piece of software in order to understand how it reacts. American Fuzzy Lop (AFL) will be our primary fuzzer. And we'll need a few hacks to make AFL and Nginx play nice. Fuzzing of Nginx appears infrequently, so maybe we'll find some good bugs by doing this. Google's oss-fuzz project will eventually target Nginx, but at the moment they appear to have made little progress.

To begin, we need to set up a Linux environment to test on. For our purposes, we will be running Debian 9 (Stretch) installed in a VM. But this can be done on bare-metal machines or cloud VPSs too. Once Debian is fully patched and running on your machine of choice, you'll need to install/compile a few pre-requisite packages. The main fuzzing components we use are,
Let's create our working directory, and get the software we'll need.

    mkdir /opt/fuzz /opt/fuzz/tests /opt/fuzz/results && cd /opt/fuzz
    apt install unzip build-essential clang zlib1g-dev libpcre3 libpcre3-dev libbz2-dev libssl-dev libini-config-dev llvm-3.8 llvm-3.8-dev llvm-3.8-runtime -y
    tar xvf nginx-1.12.1.tar.gz
    tar xvf afl-latest.tgz

Now we build AFL and it's optional afl-clang-fast wrapper.

    cd /opt/fuzz/afl-2.49b/ && make
    cd ./llvm_mode/ && LLVM_CONFIG=/usr/bin/llvm-config-3.8 make
    cd ../ && sudo make install
In order to have Nginx quickly execute our test cases, we need to patch the software to exit after performing exactly one HTTP request. This allows the Nginx process space to be in a clean state for each test case, which will help to correlate bugs and inputs. It will also prevent Nginx from performing other actions that we do not want to test. Begin by editing the file /opt/fuzz/nginx-1.12.1/src/os/unix/ngx_process_cycle.c. On line 309, there is a call to the ngx_process_events_and_timers() function, which will processes the incoming event. We want Nginx to perform this call, and then validate it's state, before we exit the program. But interestingly, Nginx considers both the incoming request and the outgoing response, to each be a single "event." So to see the output of our fuzzed HTTP request, we need to allow this event processing to happen twice. To get around this, add a counter to the for loop, which will check for two iterations before it exits. A "run_count" variable is initialized before the for loop, and checked after each iteration. The code in red below is what needs to be added.

 309     static volatile int run_count = 0;
 310     for ( ;; ) {
 311         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle     ");
 313         ngx_process_events_and_timers(cycle);


 339         if (ngx_reopen) {
 340             ngx_reopen = 0;
 341             ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
 342             ngx_reopen_files(cycle, (ngx_uid_t) -1);
 343         }
 344         if (run_count >= 1) exit(0);
 345         run_count += 1;
 346      }

Next step is to build Nginx, but using the AFL Clang Fast wrapper (afl-clang-fast). This wrapper is a drop-in Clang replacement, which allows AFL to perform instrumentation on the newly compiled Nginx binary. Afl-clang-fast is a true compiler-level instrumentation, instead of the more crude assembly-level rewriting approach taken by afl-gcc and afl-clang. This gives AFL the ability to see when different code paths are executed for each of it's fuzz test cases. Nginx will also be compiled with the "select_module" enabled, which forces the server to work with the select() method to handle connections. This makes the binary easier to profile, as this is a standard Linux syscall.

    cd /opt/fuzz/nginx-1.12.1
    CC=/usr/local/bin/afl-clang-fast ./configure --prefix=/opt/fuzz/nginx --with-select_module
    make && make install

Nginx needs to be configured so that it's friendly to the single-request-then-exit style fuzzing we will perform. Edit the file /opt/fuzz/nginx/conf/nginx.conf and add these lines at the top. This prevents Nginx from forking and running as a service.

    master_process off;
    daemon off;

In the same file, add the red lines into the "events" config block, and edit the "server" block to make Nginx listen on 8080 which allows it run as a non-root user. This also tells Nginx to handle one request at a time, and use the select() syscall we enabled earlier.

    events {
        worker_connections  1024;
        use select;
        multi_accept off;


    server {
        listen       8080;
        server_name  localhost;

The web server is compiled, configured, and nearly ready to fuzz. But there's a limitation in AFL that needs to be worked around. AFL primarily operates on files, and was not designed to fuzz network sockets - which is what we need to talk to Nginx. To get around this, Nginx needs to be able to talk over stdin / stdout so that we can feed in AFL's tests. Preeny is a collection of utilities that takes advantage of LD_Preload hooking to do all kinds of crazy things to other binaries. Specifically there is a utility called "desock" which will channel a socket to the console. This utility will bridge the gap between Nginx and AFL. Compile and load Preeny using these commands.

    cd /opt/fuzz/preeny-master/ && make

The compilation will create a directory in the preeny-master/ folder with the architecture of your machine. It will contain a file called which we use to hook Nginx. Let's copy that to our main fuzz directory for ease of access.

    cp /opt/fuzz/preeny-master/x86_64-pc-linux-gnu/ /opt/fuzz/

This hook command will launch the target Nginx server, and we'll use this again.

    LD_PRELOAD=/opt/fuzz/ /opt/fuzz/nginx/sbin/nginx

After running this command, you'll notice that the terminal seems to hang. This is because Nginx is now waiting for input on stdin. Test this by typing in "GET /" to your terminal, to see Nginx's response:

    GET /
    <!DOCTYPE html>

    <title>Welcome to nginx!</title>

The server should close immediately after issuing the response.

Creating and organizing input test cases for AFL is very important to the accuracy and speed of the fuzz job. You want to give AFL good context so that it can learn from those input test cases. Start by creating a single, very simple HTTP test case in the file /opt/fuzz/tests/test1.txt with this HTTP request as its contents. And don't forget to add two new-line characters at the end of this file to terminate the HTTP protocol.

   GET / HTTP/1.1
   Accept: text/html, application/xhtml+xml, */*
   Accept-Language: en-US
   User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
   Accept-Encoding: gzip, deflate
   Connection: Keep-Alive
   Cookie: A=asdf1234

Test this by passing it to a new hooked instance of Nginx.

   cd /opt/fuzz/ 
   LD_PRELOAD=./ ./nginx/sbin/nginx < ./tests/test1.txt

Now that we have a command that runs our fully-instrumented Nginx server, let's feed that to AFL.

    LD_PRELOAD=./ afl-fuzz -i tests -o results nginx/sbin/nginx

And now we let AFL find some bugs..

Future Work:
  • Implement AFL persistence mode