Keeping progress of huge wordlists with fuzzing tools like wfuzz, THC Hydra & Patator using tcpdump and grep

Introduction

One of the non-trivial tasks when dealing with bulky wordlists like rockyou.txt is that tools either can not handle them, or the ones which can, do not usually have a progress bar to keep track of how far the current wordlist has been exhausted.

In this post I’ll be providing a quick copy-paste one-liners and explaining the methodology on how to implement an improvised progress tracker with tools like hydra, wfuzz and patator. In my example I will be using patator as it has no limitation on resources and I like to use it when I need to go > 100 threads (for refernce hydra supports up to 64).

Please note that I am aware that patator already has a progress bar implemented, but if you want more flexibility and to show the exact word being tried in real time I will show how to do this using tcpdump and console commands like grep, as the method below could easily be ported to track progress on any other fuzzing tool sending an HTTP GET or HTTP POST request.

Below are the usual obstacles when working with big wordlists:

  • Wfuzz does not keep a memory buffer and bigger wordlists would crash it
  • Burp Suite, regardless of the version used – either community or professional, does not have the option to keep a memory buffer in the HTTP history tab. Piping thousands of requests through burp would eventually crash it.
  • THC Hydra is limited for use in up to 64 threads, and reports too many false positives and the tool is lacking flexibility – it would not report a non-returned / timed out http request properly
  • Patator does not have these limitations, and already has a progress bar imlemented w(hit enter while running the tool to see it)
  • All of these tools lack a progress bar and a reliable way to track progress

The developers of Burp Suite state themselves that the tool is not designed to work with huge wordlists:

Intruder isn’t really designed to such large wordlists. It’s intended as an interactive application, where you’ll be manually working with the results. For really large wordlists you’re generally better using dedicated brute force software like Hydra.

So what’s the solution? If you are on a brute-force that’s planned to take days, you would need to get creative. Chances are to either alter the tool and implement a customization by forking their git repo (you could also submit a git request afterwards), or pipe output through bash .

Methodology and step-by-step commands explanation

  1. Run a tool of choice, like wfuzz (patator is shown in the example – once again, the tool has already a progress an functionality implemented, you do not to do this unless you want the current word shown while brute forcing)
  2. Catch the request using tcpdump using the –immediate mode and -w options. You may need to adjust the command to your requirements – in the example provided, port 8080 is used and POST data being captured. 
tcpdump -i ens160 -w output.tcpdump --immediate-mode -l -s 0 -A 'tcp dst port 8080 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354'

3. Implement a bash sequence of commands grepping the output of the fuzzed parameter. In this case, we are doing the following things:

  • grep the j_password parameter from tcpdump’s output stored in the output.tcpdump file
  • use cut to separate results using ‘=’ as a separator and get the third result, then do the same using ‘&’ as a separator and get the first result – this will get the exact words the current thread pool is trying (i.e. if running at 10 threads you should get 10 words per command run)
  • tail the thread pool results and get the last word – this is the word that the tool has used in the last thread request on the thread pool
  • now that we have the word itself, we need to get the exact line of the word within the wordlist, so we could calculate progress as percentage:
grep -nw -f - "$wordlist"

The above grep command would get the current word by parsing the exact result match (-w*) from stdin(-f –), its line number (-n)
* this is to avoid getting broad matches, for example trying to match the word “force” would match “force1337” as well unless the -w option is provided

grep -a j_password output.tcpdump | cut -d '=' -f3 | cut -d '&' -f1 | tail -n1 | grep -nw -f - "$wordlist"; current_c=echo "$current_s" | cut -d ':' -f1

As we have the value of the current word and its line number at this point, we could calculate the current progress by getting the progress percentage from the total wordlist count:

total_w=$(wc -l "$wordlist" | cut -d ' ' -f1) # Get the total words in the wordlist
current_c="$current_s" | cut -d ':' -f1 # from the current_s string, getting just the count

# To calculate progress as a percentage, we use the current count muptiplied by 100 and then divided by the total (the vice-versa version wont work as bash arithmetic expansion does not support dividing lower by a higher value) 

progress=$(("$current_c*100/$total_w")); echo -ne $current_s "\t $progress%\r") 2>/dev/null; done

Then, we need to just create an infinite bash loop and use \r to replace the current line in the console instead of appending it (pseudo-code provided below): 

while true; do 
* all of the above commands *
progress=$(("$current_c*100/$total_w"))
echo -ne $current_s "\t $progress%\r") 2>/dev/null 
# we need to send err to /dev/null to keep the terminal free of errors, as the overall method described would lead to some errors
done

In case you need to stop the brute-force process and continue later, you could cut the wordlist from the current word using vim or sed:

for vim:

vi rockyou.txt
:/currentwordhere
:1,.d
:wq

sed (I haven’t tested this, it’s from stackexchange):

sed 's/^.(currentwordhere).*$/\1/' rockyou.txt

The full commands assembly is provided in the TLDR section below.

TLDR: I’m here just for the commands (copy paste version)

Note: this is an example of Brute-forcing an old version of Jenkins, which lacks any brute-force rate limiting protections.

Convert rockyou.txt (some tools are not able to parse symbols in its default encoding):

iconv -f latin1 -t ASCII//TRANSLIT /opt/rockyou.txt -o /opt/rockyou_patator.txt
iconv -t utf8 /opt/rockyou.txt -o /opt/rockyou_patator.txt

Terminal A (patator)

patator http_fuzz url=http://10.0.0.37:8080/j_acegi_security_check method=POST body='j_username=admin&j_password=FILE0&from=%2F&Submit=Sign+in' 0
=rockyou2_patator.txt follow=0 accept_cookie=1 -x ignore:fgrep='loginError' --start=856900 -t 140 -l patator

terminal B (tcpdump)

(tcpdump): tcpdump -i ens160 -w output.tcpdump --immediate-mode -l -s 0 -A 'tcp dst port 8080 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354'

terminal C (progress):

while true; do (wordlist="rockyou2_patator.txt"; total_w=wc -l "$wordlist" | cut -d ' ' -f1; current_s=grep -a j_password output.tcpdump | cut -d '=' -f3 | cut -d '&' -f1 | tail -n1 | grep -nw -f - "$wordlist"; current_c=echo "$current_s" | cut -d ':' -f1; progress=$(("$current_c*100/$total_w")); echo -ne $current_s "\t $progress%\r") 2>/dev/null; done

OR (more stats):

while true; do (wordlist="rockyou2_patator.txt"; total_w=wc -l "$wordlist" | cut -d ' ' -f1; current_s=grep -a j_password output.tcpdump | cut -d '=' -f3 | cut -d '&' -f1 | tail -n1 | grep -nw -f - "$wordlist"; current_c=echo "$current_s" | cut -d ':' -f1; progress=$(("$current_c*100/$total_w")); echo -ne "Current: $current_s \t\t Progress: $progress% \t\t\t\t Total: $total_w\r") 2>/dev/null; done

Here’s how it looks:

If you want to use an alternative tool like wfuzz you will have to adapt the strings and the grep command to the tool’s output.

Here’s how patator’s original stats progress output looks (just for reference):

Quick tcpdump one-liners

POST

tcpdump -i ens160 -w output.tcpdump --immediate-mode -l -s 0 -A 'tcp dst port 8080 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354' | grep -E -i -a '/|password|Host:'

GET

tcpdump -i ens160 --immediate-mode -l -s 0 -A 'tcp dst port 8080 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' | grep -E -i -a '/|password|Host:'

Both

tcpdump -i ens160 -s 0 -A 'tcp dst port 8080 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354' | grep -E -i -a '/|password|Host:'

Conclusion

Working with overly sized wordlists could be a tedious obstacle if brute-forcing is a big part of your regular work, as most tools do not have output buffering and well-designed memory management for the purpose. However, using a few bash commands and tcpdump, we could easily keep track of the current progress on the wordlist and fine-tune our brute-force attack to have as a reference on how to adjust threads, or if we want to just to have an overall idea and more insight to the brute-force process.

Resources:
TCPDump Capture HTTP GET/POST requests
– https://www.middlewareinventory.com/