Sunday, July 23, 2017

Fuzzing Nginx

Nginx is a popular web server software used by over 130 million websites. Of the 10,000 busiest websites, most are running on Nginx. Due to its vast deployment, the importance of this software cannot be overstated. Because of this, we have decided to evaluate the ruggedness of the product to search for unknown security vulnerabilities. This post will describe how to set up and use a fuzzing environment to search for bugs in Nginx.

Fuzzing is the technique of sending malformed data to a piece of software in order to understand how it reacts. American Fuzzy Lop (AFL) will be our primary fuzzer. And we'll need a few hacks to make AFL and Nginx play nice. Fuzzing of Nginx appears infrequently, so maybe we'll find some good bugs by doing this. Google's oss-fuzz project will eventually target Nginx, but at the moment they appear to have made little progress.

To begin, we need to set up a Linux environment to test on. For our purposes, we will be running Debian 9 (Stretch) installed in a VM. But this can be done on bare-metal machines or cloud VPSs too. Once Debian is fully patched and running on your machine of choice, you'll need to install/compile a few pre-requisite packages. The main fuzzing components we use are,
Let's create our working directory, and get the software we'll need.

    mkdir /opt/fuzz /opt/fuzz/tests /opt/fuzz/results && cd /opt/fuzz
    apt install unzip build-essential clang zlib1g-dev libpcre3 libpcre3-dev libbz2-dev libssl-dev libini-config-dev llvm-3.8 llvm-3.8-dev llvm-3.8-runtime -y
    tar xvf nginx-1.12.1.tar.gz
    tar xvf afl-latest.tgz

Now we build AFL and it's optional afl-clang-fast wrapper.

    cd /opt/fuzz/afl-2.49b/ && make
    cd ./llvm_mode/ && LLVM_CONFIG=/usr/bin/llvm-config-3.8 make
    cd ../ && sudo make install
In order to have Nginx quickly execute our test cases, we need to patch the software to exit after performing exactly one HTTP request. This allows the Nginx process space to be in a clean state for each test case, which will help to correlate bugs and inputs. It will also prevent Nginx from performing other actions that we do not want to test. Begin by editing the file /opt/fuzz/nginx-1.12.1/src/os/unix/ngx_process_cycle.c. On line 309, there is a call to the ngx_process_events_and_timers() function, which will processes the incoming event. We want Nginx to perform this call, and then validate it's state, before we exit the program. But interestingly, Nginx considers both the incoming request and the outgoing response, to each be a single "event." So to see the output of our fuzzed HTTP request, we need to allow this event processing to happen twice. To get around this, add a counter to the for loop, which will check for two iterations before it exits. A "run_count" variable is initialized before the for loop, and checked after each iteration. The code in red below is what needs to be added.

 309     static volatile int run_count = 0;
 310     for ( ;; ) {
 311         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle     ");
 313         ngx_process_events_and_timers(cycle);


 339         if (ngx_reopen) {
 340             ngx_reopen = 0;
 341             ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
 342             ngx_reopen_files(cycle, (ngx_uid_t) -1);
 343         }
 344         if (run_count >= 1) exit(0);
 345         run_count += 1;
 346      }

Next step is to build Nginx, but using the AFL Clang Fast wrapper (afl-clang-fast). This wrapper is a drop-in Clang replacement, which allows AFL to perform instrumentation on the newly compiled Nginx binary. Afl-clang-fast is a true compiler-level instrumentation, instead of the more crude assembly-level rewriting approach taken by afl-gcc and afl-clang. This gives AFL the ability to see when different code paths are executed for each of it's fuzz test cases. Nginx will also be compiled with the "select_module" enabled, which forces the server to work with the select() method to handle connections. This makes the binary easier to profile, as this is a standard Linux syscall.

    cd /opt/fuzz/nginx-1.12.1
    CC=/usr/local/bin/afl-clang-fast ./configure --prefix=/opt/fuzz/nginx --with-select_module
    make && make install

Nginx needs to be configured so that it's friendly to the single-request-then-exit style fuzzing we will perform. Edit the file /opt/fuzz/nginx/conf/nginx.conf and add these lines at the top. This prevents Nginx from forking and running as a service.

    master_process off;
    daemon off;

In the same file, add the red lines into the "events" config block, and edit the "server" block to make Nginx listen on 8080 which allows it run as a non-root user. This also tells Nginx to handle one request at a time, and use the select() syscall we enabled earlier.

    events {
        worker_connections  1024;
        use select;
        multi_accept off;


    server {
        listen       8080;
        server_name  localhost;

The web server is compiled, configured, and nearly ready to fuzz. But there's a limitation in AFL that needs to be worked around. AFL primarily operates on files, and was not designed to fuzz network sockets - which is what we need to talk to Nginx. To get around this, Nginx needs to be able to talk over stdin / stdout so that we can feed in AFL's tests. Preeny is a collection of utilities that takes advantage of LD_Preload hooking to do all kinds of crazy things to other binaries. Specifically there is a utility called "desock" which will channel a socket to the console. This utility will bridge the gap between Nginx and AFL. Compile and load Preeny using these commands.

    cd /opt/fuzz/preeny-master/ && make

The compilation will create a directory in the preeny-master/ folder with the architecture of your machine. It will contain a file called which we use to hook Nginx. Let's copy that to our main fuzz directory for ease of access.

    cp /opt/fuzz/preeny-master/x86_64-pc-linux-gnu/ /opt/fuzz/

This hook command will launch the target Nginx server, and we'll use this again.

    LD_PRELOAD=/opt/fuzz/ /opt/fuzz/nginx/sbin/nginx

After running this command, you'll notice that the terminal seems to hang. This is because Nginx is now waiting for input on stdin. Test this by typing in "GET /" to your terminal, to see Nginx's response:

    GET /
    <!DOCTYPE html>

    <title>Welcome to nginx!</title>

The server should close immediately after issuing the response.

Creating and organizing input test cases for AFL is very important to the accuracy and speed of the fuzz job. You want to give AFL good context so that it can learn from those input test cases. Start by creating a single, very simple HTTP test case in the file /opt/fuzz/tests/test1.txt with this HTTP request as its contents. And don't forget to add two new-line characters at the end of this file to terminate the HTTP protocol.

   GET / HTTP/1.1
   Accept: text/html, application/xhtml+xml, */*
   Accept-Language: en-US
   User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
   Accept-Encoding: gzip, deflate
   Connection: Keep-Alive
   Cookie: A=asdf1234

Test this by passing it to a new hooked instance of Nginx.

   cd /opt/fuzz/ 
   LD_PRELOAD=./ ./nginx/sbin/nginx < ./tests/test1.txt

Now that we have a command that runs our fully-instrumented Nginx server, let's feed that to AFL.

    LD_PRELOAD=./ afl-fuzz -i tests -o results nginx/sbin/nginx

And now we let AFL find some bugs..

Future Work:
  • Implement AFL persistence mode