John The Ripper (JTR) - Tweak That Attack!

I decided to blog about an overview of few methods and concepts I used for cracking hashes during DEFCON 2011, Crack Me If You Can. It felt good to win the contest and as a takeaway, there is a need to push the envelope of cracking hashes.

In this post, I will talk about JTR.

You are all familiar with JTR if you've been cracking hashes for quite sometime. I wanted to draw attention to certain features of JTR which will help you gain a better grasp at how it works and how it can be used more efficiently.

Important Files:

There are several important files which must be kept secure:

john.pot -> The well known pot file of jtr. Every time, jtr cracks a hash, it records the result in hash:pass format in a file called john.pot. It helps JTR to skip these hashes in future cracking processes should they reappear in the hash list.

We all know this much about john.pot. There are ways in which john.pot can be used to customize our cracking process. It can be parsed to do character frequency analysis of already cracked hashes and prepare custom charsets. More on that later.

john.rec -> This is the default session file of jtr. When you run jtr without explicitly specifying a session name, it creates a file called john.rec for the current session. This file has the extension ".rec". So, any file in your $john/run directory with extension ".rec" is the session file. They hold the command line parameters which we passed to JTR in that session. It helps JTR in resuming the cracking process the next time.

The name of this file is the same as the name given to the session at command line.

Code:

./john -w:wordlist.txt --format=raw-MD5 --session=cracking1 hashes.txt

Once we run this command, 2 new files called cracking1.rec and cracking1.log are created in JTR's home directory.

It's also a good reference for us. For instance, if we have 10 different sessions created called "cracking1" to "cracking10". Now if after 3 days, you resume any of these cracking sessions, you'll most likely not remember what these sessions do. So, quickly open the .rec file corresponding to your cracking session and read the CLI parameters.

john.log -> This file holds the cracking process details, the sequence of chars JTR has already tried, are stored in this file. It's size depends on the duration for which the cracking process was run. Again, this is another file which needs to be kept secure.

You can have multiple .rec and .log files, since they are specific to the session. Their names are used by JTR to correlate them.

charset files -> What makes JTR so efficient is it's ability to perform character frequency analysis on a list of words based on the frequency of occurrence of characters in a specific position. Based on this, JTR can generate words and increase the probability of cracking hashes. The list of words used by JTR to perform character frequency analysis has to be provided by us and is usually the hashes cracked so far during a session.

Charset files have ".chr" extension. By default, JTR provides us several charset files like all.chr, digits.chr, alnum.chr, lanman.chr. You cannot read the contents of these files by opening them with a text editor. They are used by JTR. However, we are given the option to create our own custom charset files.

How does it create this custom charset file?

By default, JTR parses the john.pot file and looks up all the passwords. It then performs character frequency calculation on these passwords and generates the custom charset file.

However, we would like to use our own file of cracked hashes. Here's a quick way of doing this.

Let's say we are cracking the hashfile, hashes.txt which contains a list of MD5 hashes. After cracking around 100 hashes, I decide to speed up the cracking process by fingerprinting the passwords already cracked so far.

So, I list down the cracked hashes in the console.

Code:

./john -show --format=raw-MD5 hashes.txt

It gives a list of all the cracked hashes in user:pass format.

Code:

./john -show --format=raw-MD5 hashes.txt > parseMe.txt

This will redirect the above output to parseMe.txt

Now, we need to filter out all the usernames and keep only the passwords in the format, :pass as opposed to user:pass.

It can be done quickly using Perl,

A sample script:

Code:

#!/usr/bin/perl

use strict;
use warnings;

$input=$ARGV[0];
$output=$ARGV[1];

chomp $output;

my ($user,$pass);

open(INPUT,\"<$input\") or die (\"Couldn't open the file with error $!\n\");
open(OUTPUT,\">$output\");

while(<INPUT>)
{
   chomp $_;
   ($user,$pass)=split /:/,$_;
   print OUTPUT \":\".$pass.\"\n\";
}

close(INPUT):
close(OUTPUT);

It will read each entry from the passfile in user:pass format and convert it to :pass format storing the result in an output file.

First thing, take a backup of the john.pot file somewhere outside the JTR's directory and delete it from inside run folder. Reason being, we are going to have a new john.pot file after this perl script is run and this will be used to prepare our custom charset.

Let's say, this script is called, Charset.pl. You need to call this script now as:

Code:

perl charset.pl parseMe.txt john.pot

Now, we have our specific patterns in john.pot file.

Code:

./john --make-charset=specific.chr

JTR will quickly parse the john.pot file to prepare our specific.chr file. We need to add an entry for this custom charset file inside john.conf which will enable JTR to use it.

Create a new section in JTR's john.conf as follows:

Code:

[Incremental:Specific]
FILE=$JOHN/specific.chr
MinLen= 1
MaxLen= 8
CharCount=

Min and Max lengths are of course going to depend on your requirements.

Now, to use our custom charset file, specific.chr in our next cracking session:

Code:

./john -i:Specific --format=raw-MD5 --session=cracking2 hashes.txt

Notice, the name "Specific" which I passed on the command line. This needs to be the same as what was defined in the john.conf file's incremental section.

john.conf -> Among all the files, this will be the most often visited file during the cracking process. All the tweaks need to be made in this file to increase the efficiency of our cracking process. A good understanding of the john.conf file helps in having a better command on JTR.

Sections

The JTR configuration file organizes information into separate sections. Each section has a unique function. Section Names are enclosed within Square Brackets.

There are different types of sections.

Rules sections: Any rule section name must have "List.Rules:" prefixed to it. The default rules section available in JTR are:

[List.Rules:Single] - This consists of all the rules which are applied to words during a single mode crack.

[List.Rules:Wordlist] - This consists of all the rules applied to a wordlist or dictionary when we pass the --rules parameter on command line.

[Incremental:Attack_Name] - Incremental sections are specific to the Incremental or Blind Bruteforce attacks performed in JTR. These are usually the last resort to hit those hard-to-find combinations of characters. Once we have exhausted other forms of attack, we go for this one. They run, until JTR has cracked all the hashes in the list or has finished trying all combinations generated from the charset files.

Here Attack_Name represents the name of the type of Incremental attack we are performing. It could be anything we want. What is important is the values under each Incremental Section. By default, JTR comes with 4 different sections of Incremental attack, All, Digits, Alpha, LanMan.

Each section requires following parameters:

FILE= Path to the charset file used by JTR to generate the words. This path is relative to JTR's home directory, $JOHN.
MinLen = Min length of the word generated by JTR using the charset
MaxLen = Maximum Length of the words generated by JTR with above charset
Charcount = The length of keyspace JTR uses to generate the characters. If we are targetting all the digits, lowercase and upper case characters of ASCII then, it will be 9+26+26 = 61.

There are other interesting sections as well like for external mode which we will cover in other series.

What combinations is JTR generating?

Once you have tweaked your cracking session by creating custom charset files, or by creating new rules inside john.conf file, it would be a good if could check what words JTR is generating using them. This will help us in confirming whether we got the desired results from our tweaking or not.

Let's say, I create a new rule in john.conf which will append 3 digits to the end of every word. Now, I want to run JTR such that it generates all these words by applying my custom rules and display them on the console.

This can be done easily using the --stdout option.

Code:

./john -w:wordlist.txt --rules=Add3Num --stdout

JTR will quickly populate the console with the list of generated word combinations.

Another nice way to use --stdout option is to pipe the output of JTR into another hash cracking application like oclhashcat-plus. This can be really useful as we are combining the processing power of both CPU and GPU.

CPU will generate the word combinations by applying word mangling rules and pass them to oclhashcat which will use the parallel processing power of GPU to crack the hashes.

Keyspace: Ever tried to crack those passwords with non ASCII Characters? If a password has characters that do not belong to the ASCII Charset, JTR will not be able to crack them. Reason being, JTR understands and generates characters which only belong to the ASCII Charset.

Each character can be represented as 8 bits or 1 byte. JTR uses only 7 bits for each character while generating words during a hash cracking process. Maximum combinations possible are: 2**7 = 128. You can use asciitable.com as a reference to verify that any ASCII Character will have a value between 0 and 127.

Some characters like accents, umlauts and Cyrillic characters which belong to languages such as Spanish, French, Russian, German are left out since they belong to the Unicode Charset. To enable recognition of these chars in JTR, you need to edit the configuration files of JTR and recompile it.

If you are on Windows, you usually run the executable the way it was provided. However, for our changes to be reflected, JTR needs to be recompiled.

In future series, I will cover:

How to crack passwords with Unicode Charset characters in them.
How to crack passwords of length greater than 8 using JTR.

There's a lot more to how we can tweak JTR and have more focused attacks which will be discussed in other series.