Bash Shell Scripting - 10 Seconds Guide

This Bash shell scripting guide is not a detailed study but a quick reference to the BASH syntax. So lets begin...

Common Environment Variables

PATH - Sets the search path for any executable command. Similar to the PATH variable in MSDOS.

HOME - Home directory of the user.

MAIL - Contains the path to the location where mail addressed to the user is stored. 

IFS - Contains a string of characters which are used as word seperators in the command line. The string normally consists of the space, tab and the newline characters. To see them you will have to do an octal dump as follows:

$ echo $IFS | od -bc

PS1 and PS2 - Primary and secondary prompts in bash. PS1 is set to $ by default and PS2 is set to '>' . To see the secondary prompt, just run the command :

$ ls |

... and press enter.

USER - User login name.

TERM - indicates the terminal type being used. This should be set correctly for editors like Vim to work correctly.

SHELL - Determines the type of shell that the user sees on logging in.

Note: To see what are the values held by the above environment variables, just do an echo of the name of the variable preceeded with a $. For example, if I do the following:

$ echo $USER
ravi

... I get the value stored in the environment variable USER.

Some Bash Shell Scripting Rules

• The first line in your script must be #!/bin/bash
  ... that is a # (Hash) followed by a ! (bang) followed by the path of the shell. This line lets the environment know the file is a shell script and the location of the shell.
• Before executing your script, you should make the script executable. You do it by using the following command:
  

  $ chmod ugo+x your_shell_script.sh

• The name of your shell script must end with a .sh . This lets the user know that the file is ashell script. This is not compulsary but is the norm.

Conditional Statements

'If' Statement

The 'if' statement evaluates a condition which accompanies its command line.
syntax:

if condition_is_true
then
   execute commands
else
   execute commands
fi

'if' condition also permits multiway branching. That is you can evaluate more conditions if the previous condition fails.

if condition_is_true
then
   execute commands
elif another_condition_is_true
then
   execute commands
else
   execute commands
fi

Example :

if grep "linuxhelp" thisfile.html
then
   echo "Found the word in the file"
else
   echo "Sorry no luck!"
fi

If's Companion - Test

test is an internal feature of the shell. 'test' evaluates the condition placed on its right, and returns either a true or false exit status. For this purpose, 'test' uses certain operators to evaluate the condition. They are as follows:

Relational Operators

• -eq - Equal to
• -lt - Less than
• -gt - Greater than
• -ge - Greater than or Equal to
• -le - Less than or Equal to

File related tests

• -f file - True if file exists and is a regular file.
• -r file - True if file exists and is readable.
• -w file - True if file exists and is writable.
• -x file - True if file exists and is executable.
• -d file - True if file exists and is a directory.
• -s file - True if file exists and has a size greater than zero.

String tests

• -n str - True if string str is not a null string.
• -z str - True if string str is a null string.
• str1 == str2 - True if both strings are equal.
• str - True if string str is assigned a value and is not null.
• str1 != str2 - True if both strings are unequal.
• -s file - True if file exists and has a size greater than zero.

Test also permits the checking of more than one expression in the same line.

• -a  - Performs the AND function
• -o  - Performs the OR function

A few Example snippets of using test

test $d -eq 25 && echo $d

... which means, if the value in the variable d is equal to 25, print the value. Otherwise don't print anything.

test $s -lt 50 && do_something

if [ $d -eq 25 ]
then
echo $d
fi

In the above example, I have used square brackets instead of the keyword test - which is another way of doing the same thing.

if [ $str1 == $str2 ]
then
do something
fi

if [ -n "$str1" -a -n "$str2" ]
then
echo 'Both $str1 and $str2 are not null'
fi

... above, I have checked if both strings are not null then execute the echo command.

Things to remember while using test

1. If you are using square brackets [] instead of test, then care should be taken to insert a space after the [ and before the ].
2. test is confined to integer values only. Decimal values are simply truncated.
3. Do not use wildcards for testing string equality - they are expanded by the shell to match the files in your directory rather than the string.

Case Statement

Case statement is the second conditional offered by the shell.
Syntax:

case expression in
pattern1) execute commands ;;
pattern2) execute commands ;;
...
esac

The keywords here are in, case and esac. The ';;' is used as option terminators. The construct also uses ')' to delimit the pattern from the action.


Example:

...
echo "Enter your option : "
read i;

case $i in
1) ls -l ;;
2) ps -aux ;;
3) date ;;
4) who ;;
5) exit
esac

Note: The last case option need not have ;; but you can provide them if you want.


Here is another example:

case `date |cut -d" " -f1` in
Mon) commands ;;
Tue) commands ;;
Wed) commands ;;
...
esac

Case can also match more than one pattern with each option.You can also use shell wild-cards for matching patterns.

...
echo "Do you wish to continue? (y/n)"
read ans

case $ans in
Y|y) ;;
[Yy][Ee][Ss]) ;;
N|n) exit ;;
[Nn][Oo]) exit ;;
*) echo "Invalid command"
esac

In the above case, if you enter YeS, YES,yEs and any of its combinations, it will be matched.

This brings us to the end of conditional statements.

Looping Statements

while loop
Syntax :

while condition_is_true
do
execute commands
done

Example:

while [ $num -gt 100 ]
do
sleep 5
done

while :
do
execute some commands
done

The above code implements a infinite loop. You could also write 'while true' instead of 'while :' .
Here I would like to introduce two keywords with respect to looping conditionals. They are break and continue.
break - This keyword causes control to break out of the loop.
continue - This keyword will suspend the execution of all statements following it and switches control to the top of the loop for the next iteration.

until loop
Until complements while construct in the sense that the loop body here is executed repeatedly as long as the condition remains false.
Syntax:

until false
do
execute commands
done

Example:

...
until [ -r myfile ]
do
sleep 5
done

The above code is executed repeatedly until the file myfile can be read.

for loop
Syntax :

for variable in list
do
execute commands
done

Example:

...
for x in 1 2 3 4 5
do
echo "The value of x is $x";
done

Here the list contains 5 numbers 1 to 5. Here is another example:

for var in $PATH $MAIL $HOME
do
echo $var
done

Suppose you have a directory full of java files and you want to compile those. You can write a script like this:

...
for file in *.java
do
javac $file
done

Note: You can use wildcard expressions in your scripts.

Special Symbols Used In BASH Scripting

• $* - This denotes all the parameters passed to the script at the time of its execution. Which includes $1, $2 and so on.
• $0 - Name of the shell script being executed.
• $# - Number of arguments specified in the command line.
• $? - Exit status of the last command.

The above symbols are known as positional parameters. Let me explain the positional parameters with the aid of an example. Suppose I have a shell script called my_script.sh . Now I execute this script in the command line as follows :

$ ./my_script.sh linux is a robust OS

... as you can see above, I have passed 5 parameters to the script. In this scenario, the values of the positional parameters are as follows:

• $* - will contain the values 'linux','is','a','robust','OS'.
• $0 - will contain the value my_script.sh - the name of the script being executed.
• $# - contains the value 5 - the total number of parameters.
• $$ - contains the process ID of the current shell. You can use this parameter while giving unique names to any temporary files that you create at the time of execution of the shell.
• $1 - contains the value 'linux'
• $2 - contains the value 'is'

... and so on.

The Set And Shift Statements

set - Lets you associate values with these positional parameters .
For example, try this:

$ set `date`
$ echo $1
$ echo $*
$ echo $#
$ echo $2

shift - transfers the contents of a positional parameter to its immediate lower numbered one. This goes on as many times it is called.

Example :

$ set `date`
$ echo $1 $2 $3
$ shift
$ echo $1 $2 $3
$ shift
$ echo $1 $2 $3

To see the process Id of the current shell, try this:

$ echo $$
2667

Validate that it is the same value by executing the following command:

$ ps -f |grep bash

Make Your BASH Shell Script Interactive

read statement
Make your shell script interactive. read will let the user enter values while the script is being executed. When a program encounters the read statement, the program pauses at that point. Input entered through the keyboard id read into the variables following read, and the program execution continues.
Eg:

#!/bin/sh
echo "Enter your name : "
read name
echo "Hello $name , Have a nice day."

Exit Status Of The Last Command

Every command returns a value after execution. This value is called the exit status or return value of the command. A command is said to be true if it executes successfully, and false if it fails. This can be checked in the script using the $? positional parameter.

Implementation of SHA512-crypt vs MD5-crypt

If you have a new installation, you’re probably using SHA512-based passwords instead of the older MD5-based passwords described in detail in the previous post, which I’ll assume you’ve read. sha512-crypt is very similar to md5-crypt, but with some interesting differences.

Since the implementation of sha512 is really less interesting than the comparison with md5-crypt, I’ll describe it by striking out the relevant parts of the md5-crypt description and writing in what sha512-crypt does instead.

Like md5-crypt, it can be divided into three phases. Initialization, loop, and finalization.

1. Generate a simple md5 sha512 hash based on the salt and password
2. Loop 1000 5000 times, calculating a new sha512 hash based on the previous hash concatenated with alternatingly the hash of the password and the salt. Additionally, sha512-crypt allows you to specify a custom number of rounds, from 1000 to 999999999
3. Use a special base64 encoding on the final hash to create the password hash string

The main differences are the higher number of rounds, which can be user selected for better (or worse) security, the use of the hashed password and salt in each round, rather than the unhashed ones, and a few tweaks of the initialization step.

Here’s the real sha512-crypt initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 16chars) , and “magic” the string “$1$”
2. Start by computing the Alternate sum, sha512(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte the Alternate sum
      • If it’s unset, append the first byte of the password
4. New: Let S_factor be 16 + the first byte of Intermediate₀
5. New: Compute the S bytes, length(salt) bytes of sha512(salt, concatenated S_factor times).
6. New: Compute the P bytes, length(password) bytes of sha512(password), repeated as necessary

Step 3.5 — which was very strange in md5-crypt — now makes a little more sense. We also calculated the S bytes and P bytes, which from here on will be used just like salt and password was in md5-crypt.

From this point on, the calculations will only involve the password P bytes, salt S bytes, and the Intermediate₀ sum. Now we loop 5000 times (by default), to stretch the algorithm.

• For i = 0 to 4999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password P bytes
  3. If i is not divisible by 3, salt S bytes
  4. If i is not divisible by 7, password P bytes
  5. If i is even, password P bytes
  6. If i is odd, Intermediate_i
  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₅₀₀₀. Let’s call this the Final sum. Since sha512 is 512bit, this is 64 bytes long.

The bytes will be rearranged, and then encoded as 86 ascii characters using the same base64 encoding as md5-crypt.

1. Output the magic, “$6$”
2. New: If using a custom number of rounds, output “rounds=12345$”
3. Output the salt
4. Output a “$” to separate the salt from the encrypted section
5. Pick out the 64 bytes in this order: 63 62 20 41 40 61 19 18 39 60 59 17 38 37 58 16 15 36 57 56 14 35 34 55 13 12 33 54 53 11 32 31 52 10 9 30 51 50 8 29 28 49 7 6 27 48 47 5 26 25 46 4 3 24 45 44 2 23 22 43 1 0 21 42
   • For each group of 6 bits (there’s 86 groups), starting with the least significant
     • Output the corresponding base64 character with this index

And yes, I do have a shell script for this as well: sha512crypt. This one takes about a minute to generate a hash, due to the higher number of rounds. However, it doesn’t support custom rounds.

I hope these two posts have provided an interesting look at two exceedingly common, but often overlooked, algorithms!

 sha512crypt

#!/bin/bash

# sha512-crypt
# sha512-crypt for GNU and Bash

b64="./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

stringToNumber() { 
    expression=0
    for((i=0; i<${#1}; i++))
    do
        expression=$(printf '(%s)*256+%d' "$expression" "'${1:$i:1}")
    done
    bc <<< "$expression"
}

# Turn some string into a \xd4\x1d hex string
stringToHex() { 
    for((i=0; i<${#1}; i++))
    do
        printf '\\x%x' "'${1:i:1}"
    done
}

# Turn stdin into a \xd4\x1d style sha512 hash
sha512hex() { 
    sum=$(sha512sum) 
    read sum rest <<< "$sum" # remove trailing dash
    hex=$(sed 's/../\\x&/g' <<< "$sum")
    echo "$hex"
}

# Turn an integer into a crypt base64 string with n characters
intToBase64() { 
    number=$1
    n=$2
    for((j=0; j>=1)) 
        do
            if (( i & 1 ))
            then
                printf "$alternate"
            else 
                printf "$password" 
            fi
        done

    } | sha512hex
    )
    firstByte=$(hexToInt $(getBytes "$intermediate" 0))

    p_bytes=$(for((i=0; i<$passwordLength; i++)); do printf "$password"; done | sha512hex | head -c $((passwordLength*4)) )
    s_bytes=$(for((i=0; i<16+${firstByte}; i++)); do printf "$salt"; done  | sha512hex | head -c $((saltLength*4)) )


    for((i=0; i<5000; i++))
    do
        intermediate=$({
            (( i & 1 )) && printf "$p_bytes" || printf "$intermediate"
            (( i % 3 )) && printf "$s_bytes"
            (( i % 7 )) && printf "$p_bytes"
            (( i & 1 )) && printf "$intermediate" || printf "$p_bytes"
        } | sha512hex)
    done

    # Rearrange the bytes and crypt-base64 encode them
    hex=$(base64EncodeBytes 86 "$intermediate" \
        63  62 20 41  40 61 19  18 39 60  59 17 38  37 58 16  15 36 57  56 14 35 \
            34 55 13  12 33 54  53 11 32  31 52 10   9 30 51  50  8 29  28 49  7 \
             6 27 48  47  5 26  25 46  4   3 24 45  44  2 23  22 43  1   0 21 42)

    printf "%s$salt\$%s\n" "$magic" "$hex" 

}


if [[ $# < 1 ]] 
then
    echo "Usage: $0 password [salt]" >&2 
    exit 1
fi

password=$(stringToHex "$1")
salt=$(stringToHex "$2")
[[ -z $salt ]] && salt=$(tr -cd 'a-zA-Z0-9' < /dev/urandom | head -c 16) 

doHash "$password" "$salt" '$6$'

Password hashing with MD5-crypt in relation to MD5

If you haven’t reinstalled recently, chances are you’re using MD5-based passwords. However, the password hashes you find in /etc/shadow look nothing like what md5sum returns.

Here’s an example:

/etc/shadow:
$1$J7iYSKio$aEY4anysz.gtXxg7XlL6v1

md5sum:
7c6483ddcd99eb112c060ecbe0543e86

What’s the difference in generating these hashes? Why are they different at all?

Just running md5sum on a password and storing that is just marginally more secure than storing the plaintext password.

Thanks to GPGPUs, a modern gaming rig can easily try 5 billion such passwords per second, or go over the entire 8-character alphanumeric space in a day. With rainbow tables, a beautiful time–space tradeoff, you can do pretty much the same in 15 minutes.

MD5-crypt employs salting to make precomputational attacks exponentially more difficult. Additionally, it uses stretching to make brute force attacks harder (but just linearly so).

As an aside, these techniques were used in the original crypt from 1979, so there’s really no excuse to do naive password hashing anymore. However, at that time the salt was 12 bits and the number of rounds 25 — quite adorable in comparison with today’s absolute minimum of 64 bits and 1000 rounds.

The original crypt was DES based, but used a modified algorithm to prevent people from using existing DES cracking hardware. MD5-crypt doesn’t do any such tricks, and can be implemented in terms of any MD5 library, or even the md5sum util.

As regular reads might suspect, I’ve written a shell script to demonstrate this: md5crypt. There are a lot of workarounds for Bash’s inability to handle NUL bytes in strings. It takes 10 seconds to generate a hash, and is generally awful..ly funny!

Let’s first disect a crypt hash. man 3 crypt has some details.

If salt is a character string starting with the characters
"$id$" followed by a string terminated by "$":

       $id$salt$encrypted

then instead of using the DES machine, id  identifies  the
encryption  method  used  and this then determines how the
rest of the password string is interpreted.  The following
values of id are supported:

       ID  | Method
       -------------------------------------------------
       1   | MD5
       2a  | Blowfish (on some Linux distributions)
       5   | SHA-256 (since glibc 2.7)
       6   | SHA-512 (since glibc 2.7)

Simple and easy. Split by $, and then your fields are Algorithm, Salt and Hash.

md5-crypt is a function that takes a plaintext password and a salt, and generate such a hash.

To set a password, you’d generate a random salt, input the user’s password, and write the hash to /etc/shadow. To check a password, you’d read the hash from /etc/shadow, extract the salt, run the algorithm on this salt and the candidate password, and then see if the resulting hash matches what you have.

md5-crypt can be divided into three phases. Initialization, loop, and finalization. Here’s a very high level description of what we’ll go through in detail:

1. Generate a simple md5 hash based on the salt and password
2. Loop 1000 times, calculating a new md5 hash based on the previous hash concatenated with alternatingly the password and the salt.
3. Use a special base64 encoding on the final hash to create the password hash string

Put like this, it relatively elegant. However, there are a lot of details that turn this from elegant to eyerolling.

Here’s the real initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 chars), and “magic” the string “$1$”
2. Start by computing the Alternate sum, md5(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte
      • If it’s unset, append the first byte of the password

I know what you’re thinking, and yes, it’s very arbitrary. The latter part was most likely a bug in the original implementation, carried along as UNIX issues often are. Remember to stay tuned next week, when we’ll compare this to SHA512-crypt as used on new installations!

From this point on, the calculations will only involve the password, salt, and Intermediate₀sum. Now we loop 1000 times, to stretch the algorithm.

• For i = 0 to 999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password
  3. If i is not divisible by 3, salt
  4. If i is not divisible by 7, password
  5. If i is even, password
  6. If i is odd, Intermediate_i
  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₁₀₀₀. Let’s call this the Final sum. Since MD5 is 128bit, this is 16 bytes long.

The bytes will be rearranged, and then encoded as 22 ascii characters with a special base64-type encoding. This is not the same as regular base64:

Normal base64 set:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

Crypt base64 set:
./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Additionally, there is no padding. The leftover byte will be encoded into 2 base64 ascii characters.

1. Output the magic
2. Output the salt
3. Output a “$” to separate the salt from the encrypted section
4. Pick out the 16 bytes in this order: 11 4 10 5 3 9 15 2 8 14 1 7 13 0 6 12.
   • For each group of 6 bits (there are 22 groups), starting with the least significant
     • Output the corresponding base64 character with this index

Congratulations, you now have a compatible md5-crypt hash!

As you can see, it’s quite far removed from a naive md5(password) attempt.

Fortunately, one will only ever need this algorithm for compatibility. New applications can use the standard PBKDF2 algorithm, implemented by most cryptography libraries, which does the same thing only in a standardized and parameterized way.

As if this wasn’t bad enough, the next post next week will be more of the same, but with SHA512-crypt!

md5cript.sh

#!/bin/bash
# md5-crypt for GNU and Bash

b64="./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

stringToNumber() { 
    expression=0
    for((i=0; i<${#1}; i++))
    do
        expression=$(printf '(%s)*256+%d' "$expression" "'${1:$i:1}")
    done
    bc <<< "$expression"
}

# Turn some string into a \xd4\x1d hex string
stringToHex() { 
    for((i=0; i<${#1}; i++))
    do
        printf '\\x%x' "'${1:i:1}"
    done
}

# Turn stdin into a \xd4\x1d style md5 hash
md5hex() { 
    sum=$(md5sum) 
    read sum rest <<< "$sum" # remove trailing dash
    hex=$(sed 's/../\\x&/g' <<< "$sum")
    echo "$hex"
}

# Turn an integer into a crypt base64 string with n characters
intToBase64() { 
    number=$1
    n=$2
    for((j=0; j>=1)) 
        do
            if (( i & 1 ))
            then
                printf '\x00' 
            else 
                printf "$password" | head -c 1
            fi
        done

    } | md5hex
    )

    for((i=0; i<1000; i++))
    do
        intermediate=$({
            (( i & 1 )) && printf "$password" || printf "$intermediate"
            (( i % 3 )) && printf "$salt"
            (( i % 7 )) && printf "$password"
            (( i & 1 )) && printf "$intermediate" || printf "$password"
        } | md5hex)
    done

    # Rearrange the bytes and crypt-base64 encode them
    encoded=$(base64EncodeBytes 22 "$intermediate" 11  4 10 5  3 9 15  2 8 14  1 7 13  0 6 12)

    printf "%s$salt\$%s\n" "$magic" "$encoded" 

}


if [[ $# < 1 ]] 
then
    echo "Usage: $0 password [salt]" >&2 
    exit 1
fi

password=$(stringToHex "$1")
salt=$(stringToHex "$2")
[[ -z $salt ]] && salt=$(tr -cd 'a-zA-Z0-9' < /dev/urandom | head -c 8) 

doHash "$password" "$salt" '$1$'

What’s in a SSH RSA key pair?

You probably have your own closely guarded ssh key pair. Chances are good that it’s based on RSA, the default choice in ssh-keygen.

RSA is a very simple and quite brilliant algorithm, and this article will show what a SSH RSA key pair contains, and how you can use those values to play around with and encrypt values using nothing but a calculator.

RSA is based on primes, and the difficulty of factoring large numbers. This post is not meant as an intro to RSA, but here’s a quick reminder. I’ll use mostly the same symbols as Wikipedia: you generate two large primes, p and q. Let φ = (p-1)(q-1). Pick a number e coprime to φ, and let d ≡ e^-1 mod φ.

The public key is then (e, n), while your private key is (d, n). To encrypt a number/message m, let the ciphertext c ≡ m^e mod n. Then m ≡ c^d mod n.

This is very simple modular arithmetic, but when you generate a key pair with ssh-keygen, you instead get a set of opaque and scary looking files, id_rsa and id_rsa.pub. Here’s a bit from the private key id_rsa (no passphrase):


-----BEGIN RSA PRIVATE KEY-----
MIIBygIBAAJhANj3rl3FhzmOloVCXXesVPs1Wa++fIBX7BCZ5t4lmMh36KGzkQmn
jDJcm+O9nYhoPx6Bf+a9yz0HfzbfA5OpqQAyC/vRTVDgHhGXY6HFP/lyWQ8DRzCh
tsuP6eq9RYHnxwIBIwJhAKdf+4oqqiUWOZn//vXrV3/19LrGJYeU

...
-----END RSA PRIVATE KEY-----

How can we get our nice RSA parameters from this mess?

The easy way is with openssl: (I apologize in advance for all the data spam in the rest of the article).


root@suresh ~/.ssh $ openssl rsa -text -noout < id_rsa
Private-Key: (768 bit)
modulus:
     00:d8:f7:ae:5d:c5:87:39:8e:96:85:42:5d:77:ac:
     54:fb:35:59:af:be:7c:80:57:ec:10:99:e6:de:25:
     ...
publicExponent: 35 (0x23)
privateExponent:
     00:a7:5f:fb:8a:2a:aa:25:16:39:99:ff:fe:f5:eb:
     57:7f:f5:f4:ba:c6:25:87:94:48:64:93:fb:3d:a7:
     ...
prime1:
    ...
prime2:
    ...
exponent1:
    ...
exponent2:
    ...
coefficient:
    ...

Here, modulus is n, publicExponent is e, privateExponent is d, prime1 is p, prime2 is q, exponent1 is d_P from the Wikipedia article, exponent2 is d_Q and coefficient is q_Inv.

Only the first three are strictly required to perform encryption and decryption. The latter three are for optimization and the primes are for verification.

It’s interesting to note that even though the private key from RSA’s point of view is (d,n), the OpenSSH private key file includes e, p, q and the rest as well. This is how it can generate public keys given the private ones. Otherwise, finding e given (d,n) is just as hard as finding d given (e,n), except e is conventionally chosen to be small and easy to guess for efficiency purposes.

If we have one of these hex strings on one line, without colons, and in uppercase, then bc can work on them and optionally convert to decimal.


# If you don't want to do this yourself, see end for a script
root@suresh ~/.ssh $ { echo 'ibase=16'; cat | tr -d ':\n ' | tr a-f A-F; echo; } | bc

00:d8:f7:ae:5d:c5:87:39:8e:96:85:42:5d:77:ac:
54:fb:35:59:af:be:7c:80:57:ec:10:99:e6:de:25:
98:c8:77:e8:a1:b3:91:09:a7:8c:32:5c:9b:e3:bd:
….
Ctrl-d to end input
13158045936463264355006370413708684112837853704660293756254884673628\
63292…

We also need a power-modulo function, since b^e % m is unfeasibly slow if you go by way of b^e. Luckily, bc is programmable.


root@suresh ~/.ssh $ bc
bc 1.06.94
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.

# Our powermod function:
define pmod(b,e,m) { if(e == 0 ) return 1; if(e == 1) return b%m; rest=pmod(b^2%m,e/2,m); if((e%2) == 1) return (b*rest)%m else return rest; }

#Define some variables (this time unabbreviated)
n=13158045936463264355006370413708684112837853704660293756254884673628\
63292777770859554071108633728590995985653161363101078779505801640963\
48597350763180843221886116453606059623113097963206649790257715468881\
4303031148479239044926138311

e=35

d=10150492579557375359576342890575270601332058572166512326253768176799\
23111571423234513140569517447770196903218153051479115016036905320557\
80231250287900874055062921398102953416891810163858645414303785372309\
5688315939617076008144563059

# Encrypt the number 12345
c=pmod(12345, e, n)

# Show the encrypted number
c
15928992191730477535088375321366468550579140816267293144554503305092\
03492035891240033089011563910196180080894311697511846432462334632873\
53515625


#Decrypt the number
pmod(c, d, n)
12345

Yay, we’ve successfully encrypted and decrypted a value using real life RSA parameters!

What’s in the public key file, then?

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEA2PeuXcWHOY6WhUJdd6xU+zVZr758gFfsEJnm3iWYyHfoobORCaeMMlyb472diGg/HoF/5r3LPQd/Nt8Dk6mpADIL+9FNUOAeEZdjocU/+XJZDwNHMKG2y4/p6r1FgefH suresh@suresh.spam

This is a very simple file format, but I don’t know of any tools that will decode it. Simply base64-decode the middle string, and then read 4 bytes of length, followed by that many bytes of data. Repeat three times. You will then have key type, e and n, respectively.

Mine is 00 00 00 07, followed by 7 bytes “ssh-rsa”. Then 00 00 00 01, followed by one byte of 0×23 (35, our e). Finally, 00 00 00 61 followed by 0×61 = 97 bytes of our modulus n.

If you want to decode the private key by hand, base64-decode the middle bit. This gives you an ASN.1 encoded sequence of integers.

This is an annotated hex dump of parts of a base64-decoded private key

30 82 01 ca   - Sequence, 0x01CA bytes
    02 01: Integer, 1 byte
        00
    02 61:    - Integer, 0x61 bytes (n).
        00 d8 f7 ae 5d c5 87 39 8e 96 ... Same as from openssl!
    02 01:  - Integer, 1 byte, 0x23=35 (e)
        23
    02 61  - Integer, 0x61 bytes (d)
        00 a7 5f fb 8a 2a aa 25 16 39 ...
    ...

Here’s a bash script

#!/bin/bash

set -e

die() { 
    echo "$@" >&2
    exit 1
}

if [[ -z $1 ]] 
then
    die "Usage: $0 private_rsa_key"
fi

sshkey=$(< $1) 
if [[ $sshkey != "-----BEGIN RSA PRIVATE KEY-----"* ]]
then
    die "This does not appear to be an RSA private key"
fi

if [[ $sshkey == *ENCRYPTED* ]] 
then
    echo "Key is encrypted, using openssl to decrypt"
    sshkey=$(openssl rsa <<< "$sshkey")
fi

base64key=$(sed '1d; $d;' <<< "$sshkey") #the base64 data
decdump=$(base64 -d <<< "$base64key" | od -t u1 | sed -e 's/^[^ ]*//')
decbytes=$(echo $decdump | tr ' ' '\n') #decimal bytes, one on each line

readInt() { 
    read byte 
#    echo "read type $byte" >&2
    (( (byte & 0x1F) != 2 )) && \
        die "$byte doesn't encode an integer :O"

    length=$(readLength)
#    echo "read length $length" >&2
    exp="0"
    for((i=0; i&2
        exp="($exp)*256+$b"
    done
    bc <<< "$exp" | tr -d '\n\\ '
}

readLength() { 
    local bytes length n i 
    read bytes

    n=$((bytes&0x7F))
    if (( bytes & 0x80 ))
    then
        length=0
        for((i=0; i /dev/null # the sequence length
    unknown=$(readInt)
    n=$(readInt) #modulo
    e=$(readInt) #public exponent
    d=$(readInt) #private exponent
    p=$(readInt) #first prime
    q=$(readInt) #second prime
    exp1=$(readInt) # d mod p-1
    exp2=$(readInt) # d mod q-1
    c=$(readInt) || c=0 # p^1 mod q

    length=$(bc -l <<< "scale=20; v=l($n)/l(2); scale=1; v/1")
    echo "# Key is $length bits long"
    echo "# Input for bc:"
    echo "n=$n" 
    echo "p=$p" 
    echo "q=$q" 
    echo "e=$e"
    echo "d=$d"
    echo "c=$c"
    echo "exp1=$exp1"
    echo "exp2=$exp2"
    echo "scale=0"
    echo "define pmod(b,e,m) { if(e == 0 ) return 1; if(e == 1) return b%m; rest=pmod(b^2%m,e/2,m); if((e%2) == 1) return (b*rest)%m else return rest;  }"
    echo "define encrypt(message) { return pmod(message, e, n); }"
    echo "define decrypt(message) { return pmod(message, d, n); }"
    echo "define verify() { return n == p*q && (d*e)%((p-1)*(q-1)) == 1 && exp1 == d % (p-1) && exp2 == d % (q-1) && (c*q)%p == 1; }"
    echo "# End bc"
) <<< "$decbytes"  

that will decode a private key and output variable definitions and functions for bc, so that you can play around with it without having to do the copy-paste work yourself. It decodes ASN.1, and only requires OpenSSL if the key has a passphrase.

When run, and its output pasted into bc, you will have the variables n, e, d, p, q and a few more, functions encrypt(m) and decrypt(c), plus a verify() that will return 1 if the key is valid. These functions are very simple and transparent.

Enjoy!