Thursday, January 19, 2012

Turbocharge PuTTY with 12 Powerful Add-Ons – Softwares

1. PuTTY Connection Manager

PuTTYCM gives a nice feature to arrange several PuTTY sessions in tabs . While starting PuTTYCM for the first time, you should specify the location of the original PuTTY. This requires .NET 2.0 to be installed on the windows system. Following screen-shot displays three putty sessions in tabs within the same window.
Note: If the PuTTY Connection Manager opens the original PuTTY in a separate window, instead of opening as a TAB, please go to Tools -> Options -> Select the check-box “Enable additional timing for PuTTY capture (ms)” -> set the value to 300 ms. This will open the PuTTY window inside the TAB as shown below.
PuTTY Connection Manager - Multiple Tab
Fig – PuTTY Connection Manager with multiple Tabs

2. PuTTYcyg

Cygwin users will absolutely love PuTTYcyg. This lets you use PuTTY as a local cygwin terminal. If you use cygwin on your windows, I’m sure you’ll hate the default MS-DOS looking cygwin window. Using PuTTYcyg, you can run cygwin inside PuTTY. I love this add-on and use it for my cygwin on Windows.
On PuTTYcyg, click on cygterm radio button in the Connection type. Enter – (hyphen) in the “Command (use – for login shell“, to connect to the cygwin on the windows laptop using PuTTY as shown below.
PuTTYcyg Cygterm radio-button
Fig – PuTTYcyg with Cygterm option

3. PuTTYtray

Using PuTTYtray, you can minimize the PuTTY window to the system tray on windows. By default, original PuTTY stores the session information in the registry. This is painful, when you want to transfer PuTTY sessions from one laptop to another. In PuTTYtray, there is an additional radio button “Sessions from file” as shown below, that will let you store session information in a file.
Fig – PuTTYtray with “Session from file” option

4. PuttyTabs

PuttyTabs provides a floating bar, that will display the open PuTTY sessions in TABs. Clicking on one of the tabs will bring the respective PuTTY session to the foreground. While starting PuTTYTabs for the first time, you should specify location of the original PuTTY. It reads the windows registry to get all the available PuTTY sessions. This also requires .NET 2.0 to be installed on the windows system. Following screen-shot displays three putty sessions arranged in tab.

PuTTYTabs Screenshot
Fig – PuTTYTabs with multiple Tabs

5. Quest PuTTY

Quest Software modified the PuTTY to add Active Directory (GSSAPI Kerberos) single sign-on feature. Quest PuTTY uses Microsoft’s Security Service Provider Interface (SSPI), which is Microsoft’s version of the GSSAPI, with which it is wire compatible. This version of PuTTY adds a new menu-item called GSSAPI, under Connection -> SSH, as shown below.
Quest PuTTY with GSSAPI
Fig – Quest PuTTY with GSSAPI option

6. Modified PuTTY

This modified PuTTY stores the PuTTY sessions in folder instead of storing it in the registry. If you already have sessions stored in the registry, it will display those also. The sessions stored in registry will be marked as [registry] as shown below. When you create a session using this PuTTY, this creates a sub-folder called session in the same folder where putty.exe is located, to store all the sessions in the file.
Modified Putty
Fig – Modified Putty displaying both registry and file sessions

7. PocketPuTTY

PocketPuTTY runs on Windows Mobile 2003/5.0 operating system. After I got my blackberry, I have dumped my Dell Axim that was running on Windows Mobile. So, I have not tried PocketPuTTY myself. If you’ve used PocketPuTTY or other mobile version of PuTTY, please leave your feedback.
PocketPuTTY UI
Fig – PocketPuTTY for Windows Mobile

8. portaPuTTY

portaPuTTY is a modified version of the PuTTY that stores the session information in a file by default, instead of storing it in the windows registry. The session files are stored under .putty/sessions folder. The .putty folder is created under the same directory where the putty.exe is located.

9. PuTTY Portable

PuTTY Portable is part of PortableApps suite. Use this to launch PuTTY from the USB drive and carry the sessions along with you.

10. PuTTY Launchy Plugin

If you are using Launchy, the open source keystroke launcher for windows, you can use Putty Launchy Plugin, to launch putty sessions from LaunPluginchy very easily. i.e you can type “ssh” or “putty” followed by tab or space to list all of your PuTTY sessions. Once you select a particular session, Launchy will automatically launch that particular PuTTY session.
PuTTY Launchy Plugin
Fig – PuTTY Launchy Plugin. Type ssh followed by tab.

11. PuTTY Session Manager

PuTTY Session Manager will let you organize the PuTTPluginY sessions into folders and assign hotkeys. This requires Microsoft .NET 2.0. Right click on the PSM icon in the system track and select “Session Hotkeys” to assign hot-keys for PuTTY session as shown below.
PuTTY Session Manager Hot Key Assignment
Fig – PuTTY Session Manager with session hot-key
To create a folder, right click on a particular PuTTY session -> Session Management -> New Folder. To move a existing session to a folder, just drag the session and drop to the corresponding folder.
PSM Session List with folders
Fig – PuTTY Session Manager with sessions inside sub-folder

12. PuTTY Command Sender

PuTTYCS is very helpful little tool that can boost your productivity by eliminating repetitive tasks performed on different servers. Using PuTTYCS, you can send a unix command to multiple PuTTY windows at the same time. You can use this to backup files, view log files, start and stop processes, copying file etc., on multiple servers, just by executing the command once, as shown below.
PuTTY Command Sender
Fig – PuTTYCS sends unix command to multiple PuTTY session

Mergecap and Tshark: Merge Packet Dumps and Analyze Network Traffic

Mergecap is a packet dump combining tool, which will combine multiple dumps into a single dump file. Based on timestamp, the packets are written into the output file in an orderly manner. By default the output file is written in the libpcap format. However using mergecap options, we can generate output in various different format including those that are supported by wireshark tool.

mergecap is available in the wireshark package. Make sure wireshark/ethereal package is installed to use the mergecap.

Combine two dump files into a single output_dump file

Combines input_dump1 and input_dump2 capture files and writes into output_dump file.
# mergecap -v input_dump1 input_dump2 -w output_dump

In this example, input_dump2 contains the packets which are captured after input_dump1. The output_dump will contain intput_dump2 packets in the beginning followed by intput_dump1 packets.
# mergecap input_dump1 input_dump2 -w output_dump -a

Print output dump file to standard output

Combine two network dump files and print the output to the standard output instead of writing to a file.
# mergecap -v input1_dump input2_dump -w -

Print output file in a specific encapsulation format

Use option -T, to get the output file in the desired encapsulation format as shown below.
# mergecap -v -T ether -w merge_cap capture1 capture2 capture3

3. Merge packets of certain length

In this example, the output_dump contains the packets of maximum 100 bytes length.
# mergecap -v -s 100 dump1 dump2 dump3 -w output_dump

Tshark – Packet capture tool

Tshark is a powerful tool to capture network packets, which can be used to analyze the network traffic. It comes with wireshark network analyzer distribution.

Capture network capture continuously

The following example will capture the network packets continuously for 60 seconds. After 60 seconds of capture, it would stop automatically. capture_out contains the packets, which are flown in the network during the last 60 seconds.
# tshark -q -w capture_out -a duration:60
In the following example packets will be printed on the screen and simultaneously it will be written into the output file.
# tshark -S -q -w capture_out -a duration:10

Capture network statistics using tshark

To see how many packets are flowing in the network for a specific interval use the following command.
# tshark -q -w capture_duration1 -a duration:1 -z io,stat,1

Capture network packets for a specific host

Use the following example, to capture the packet flow for a particular host(transmitted and received packets). In this example, we could see that for every second how many packets are flown in the network for the host
#  tshark -S -q -w capture_duration6 -a duration:6 -z io,stat,1,ip.addr==
After capturing all the packets for 6 seconds duration, it will print the statistics as like the following,
145 packets dropped
19749 packets captured
IO Statistics
Interval: 1.000 secs
Column #0: ip.addr==
|   Column #0
Time              |frames|  bytes
000.000-001.000    2733    545242
001.000-002.000    2991    583374
002.000-003.000    3310    650716
003.000-004.000    3236    641896
004.000-005.000    3518    690860
005.000-006.000    3310    654988
006.000-007.000     638    122812

Capture network packets on a specific port

This example captures only the ssh packets.
# tshark -f “tcp port 22” -w capture_out

Capture network packets for specific duration

The following example will capture packets for specific duration (5 seconds), switch over to the next file when capture file size reaches certain size (1000KB).
# tshark -a filesize:1000 -a duration:5 -a files:5 -w ethcap1
Sample output capture filename with size:
ethcap1_00001_20090216174203 -   1000K
ethcap1_00002_20090216174205 -  1000K
ethcap1_00003_20090216174207 -  835K

Other tshark capture commands

Use option -c, to capture the packets upto certain packet count. The following example creates the ethcap1 file only with 10 packets.
# tshark -c 10  -w ethcap1

Use option -r to read network packets from as compressed file.
# tshark -r capture_dump.gz

Use option -r, to displays only specific packet types. The following example creates the file capture_dump only with the rtp packets in the network analyzer.
# tshark -R “rtp” -r capture_dump

Use the filter below to capture the tcp packets which are flowing in the port 1720.
# tshark -f “tcp port 1720”
The following example will capture packets that are coming either to the port 1720 or 1721.
# tshark -f  “port 1720 or port 1721”  -w capture_dump

By default, tshark will use eth0 device to do the packet capture. You can also specify a specific ethernet adapter using option -i as shown below.
# tshark -i eth1 -w -a duration:10 capture_dump

How to Monitor and Log Network Traffic on Linux Using vnStat

This article is part of our on-going performance monitoring series of articles.
In this article, let us focus on how to monitor network traffic and log them for later analysis using vnstat utility.
vnstat is a command line utility that displays and logs network traffic of the interfaces on your systems. This depends on the network statistics provided by the kernel. So, vnstat doesn’t add any additional load to your system for monitoring and logging the network traffic.

1. Install vnStat

Install vnStat on your system from the repository that is specific to your Linux distributions.
For example, on Ubuntu use apt-get to install it as shown below.
$ apt-get install vnstat
If you prefer to install it from source, download vnStat source from here , or use the wget as shown below to download and install it.
Please note that you can also use Mergecap and Tshark to analyze network traffic.
cd vnstat-1.11
make install
Please note that you don’t need to do “./configure” that you typically do for other source based installations.
Since vnstat depends on the information provided by kernel, execute the following command to verify whether kernel is providing all the information that vnStat is expecting.
# vnstat --testkernel
This test will take about 60 seconds.
Everything is ok.

2. Pick a Interface to Monitor using vnStat

vnStat doesn’t monitor any interfaces unless you specifically request it to do so.
To start monitoring eth0, do the following. This needs to be executed only once. As you see below, this creates a database file eth0 under /var/lib/vnstat directory that will contain all the network traffic log messages for this specific interface.
# vnstat -u -i eth0
Error: Unable to read database "/var/lib/vnstat/eth0".
Info: -> A new database has been created.
To view all the available interfaces on your system that vnStat can monitor, do the following.
# vnstat --iflist
Available interfaces: lo eth0 eth1 sit0
Start the vnstatd (vnstat daemon), which will monitor and log these information in the background.
# vnstatd -d
# ps -ef | grep vnst
root     14353     1  0 09:12 ?        00:00:00 vnstatd -d
root     14355   330  0 09:12 pts/1    00:00:00 grep vnst
Note: You can add “vnstatd -d” to your /etc/rc.local file, so that it starts automatically anytime you reboot your system.

3. vnStat Basic Usage

vnstat without any argument will give you a quick summary with the following info:
  • The last time when the vnStat datbase located under /var/lib/vnstat/ was updated
  • From when it started collecting the statistics for a specific interface
  • The network statistic data (bytes transmitted, bytes received) for the last two months, and last two days.
# vnstat
Database updated: Sat Oct 15 11:54:00 2011

   eth0 since 10/01/11

          rx:  12.89 MiB      tx:  6.94 MiB      total:  19.82 MiB

                     rx      |     tx      |    total    |   avg. rate
       Sep '11     12.90 MiB |    6.90 MiB |   19.81 MiB |    0.14 kbit/s
       Oct '11     12.89 MiB |    6.94 MiB |   19.82 MiB |    0.15 kbit/s
     estimated        29 MiB |      14 MiB |      43 MiB |

                     rx      |     tx      |    total    |   avg. rate
     yesterday      4.30 MiB |    2.42 MiB |    6.72 MiB |    0.64 kbit/s
         today      2.03 MiB |    1.07 MiB |    3.10 MiB |    0.59 kbit/s
     estimated         4 MiB |       2 MiB |       6 MiB |
Note: If you just installed the vnStat, it will give the following message “eth0: Not enough data available yet.”. Wait for some time and try the command again.

4. vnStat hours, days, months, weeks Network Data

Use “vnstat -h” (or) “vnstat –hours” for network statistic data breakdown by hour. This also displays a text based graph.
Use “vnstat -d” (or) “vnstat –days” for network statistic data breakdown by day.
# vnstat -d
 eth0  /  daily
         day         rx      |     tx      |    total    |   avg. rate
      10/10/11      2.48 MiB |    1.28 MiB |    3.76 MiB |    0.36 kbit/s
      10/11/11      4.07 MiB |    2.17 MiB |    6.24 MiB |    0.59 kbit/s
      10/12/11      4.30 MiB |    2.42 MiB |    6.72 MiB |    0.64 kbit/s
      10/13/11      2.06 MiB |    1.10 MiB |    3.16 MiB |    0.60 kbit/s
     estimated         3 MiB |       1 MiB |       4 MiB |
Use “vnstat -m” (or) “vnstat –months” for network statistic data breakdown by month.
# vnstat --m

 eth0  /  monthly

       month        rx      |     tx      |    total    |   avg. rate
      Sep '11     12.90 MiB |    6.90 MiB |   19.81 MiB |    0.14 kbit/s
      Oct '11     12.92 MiB |    6.96 MiB |   19.89 MiB |    0.15 kbit/s
    estimated        29 MiB |      14 MiB |      43 MiB |
Similar to days and months, use “vnstat -m” (or) “vnstat –months” for network statistic data breakdown by week.

5. Export the data to Excel or other DB

If you like to export the network monitoring data to an excel or other database, you can dump the data in a text format delimited with semi-colon, which you can import to Excel or other db.
The 1st few lines of the –dumpdb output contains some header information. After the header lines, it has 30 lines that starts with “d;” (d;0;1318316406;1;0;386;698;1). This lines has the following information separated by semi-colon.
  • d – stands for days
  • 0 – number of the day. 0 indicates today.
  • 1318316406 – data in Unix format
  • Followed by this, it contains the bytes transmitted and received
$ vnstat --dumpdb
You can also use “vnstat –oneline”, which displays the traffic summary in a single line where the values are delimited with semi-colon.
$ vnstat --oneline
1;eth0;10/11/11;1.45 MiB;801 KiB;2.23 MiB;0.59 kbit/s;Oct '11;3.93 MiB;2.06 MiB;6.00 MiB;0.05 kbit/s;3.93 MiB;2.06 MiB;6.00 MiB

6. Display Live Network Statistics

Use “vnstat -l” or “vnstat –live” to display the live network statistic information.
$ vnstat -l
Monitoring eth0...    (press CTRL-C to stop)

   rx:        2 kbit/s     5 p/s          tx:        2 kbit/s     4 p/s
After you press Ctrl-C to stop it, vnstat will display a summary for the time period the live monitor was running.

7. Change the default vnstat output format

Use “vnstat -s” or “vnstat –short” which will display a short summary of the network statistics. This includes statistics for today, yesterday and for the current month.
$ vnstat -s (--short)

                      rx      /      tx      /     total    /   estimated
       Oct '11      3.93 MiB  /    2.06 MiB  /    6.00 MiB  /   13.00 MiB
     yesterday      2.48 MiB  /    1.28 MiB  /    3.76 MiB
         today      1.45 MiB  /     801 KiB  /    2.23 MiB  /      --
You can also use “vnstat –style 0″, which will give a narrow column output, which is easier to read than the default wider column output.
$ vnstat --style 0
Following are the available style number:
  • 0 – Narrow output
  • 1 – Enable bar column
  • 2 – Enable bar column, and displays average traffic rate in summary
  • 3 – Display average traffic rate in all output
  • 4 – when combined with live mode (vnstat -l), disable the terminal control characters

8. Display Top 10 Traffic Days

Use “vnstat -t” or “vnstat –top10″ to display all time top 10 traffic days.
$ vnstat --top10

 eth0  /  top 10

    #      day          rx      |     tx      |    total    |   avg. rate
    1   10/12/11       4.30 MiB |    2.42 MiB |    6.72 MiB |    0.64 kbit/s
    2   10/11/11       4.07 MiB |    2.17 MiB |    6.24 MiB |    0.59 kbit/s
    3   10/10/11       2.48 MiB |    1.28 MiB |    3.76 MiB |    0.36 kbit/s

Top on Steroids – 15 Practical Linux HTOP Examples

htop is just like top, but on steroids.
Once you are used to htop, you’ll never go back to top again.
htop is a ncurses-based process viewer.
You can interact with the htop using mouse. You can scroll vertically to view the full process list, and scroll horizontally to view the full command line of the process.
This article explains 15 essential htop command examples.

Install Htop

top command is available on all Linux system by default.
To use htop, you need to install it first. Go to htop download page, and download the binaries that corresponds to your Linux distribution and install it.
If you prefer to install htop from souce, download the source code, and do the following. This will install htop under /usr/local/bin.
tar xvfz htop-0.9.tar.gz
cd htop-0.9
make install

1. htop Output

htop output consists of three sections 1) header 2) body and 3) footer.
Header displays the following three bars, and few vital system information. You can change any of these from the htop setup menu.
  • CPU Usage: Displays the %used in text at the end of the bar. The bar itself will show different colors. Low-priority in blue, normal in green, kernel in red.
  • Memory Usage
  • Swap Usage
Body displays the list of processes sorted by %CPU usage. Use arrow keys, page up, page down key to scoll the processes.
Footer displays htop menu commands.

2. Display CPU Averages in Header

htop by default displays a separate bar for every CPU (or core) on your system. On a multi core system, the list of individual CPU bar might be a distraction. Instead of multiple bars, you might want to display only one bar for CPU average usage.
Press F2 (or S) for setup menu -> Select “Meters” under setup -> Use right arrow to go to “Available Meters” column -> Use down arrow to select “CPU average” -> Press “F5″, which will add “CPU average” to the list under the “Left columns”.
Now, use left arrow to go to “Left columns” -> select “All CPUs” -> Press F9 to remove it from the “Left columns” -> Use down arrow, select “CPU” -> Press “F7″ to move it to the top -> Press to come out of the setup menu.
This will show only the CPU average in the bar as shown below, instead of showing all the individual cores.

3. Display Processes Sorted by any htop Output Column – Press F6 or >

By default htop command displays the processes sorted by CPU usage. Press F6 (or >), which will display the column names, select any one of the column, and press Enter.
Note: Instead sorting columns by going to sort menu, you can also press the sortcut key “P” to sort by CPU%, “M” to sort by MEM%, or “T” to sort by TIME%.

4. Kill a Process Without Exiting From htop – Press F9 or k

To kill a process, Select the process that needs to be killed from the list, and press F9 (or k), which will display the “Send signal” menu that lists all the available signals that you can send to the command.
If you want to do “kill -9″ on the selected process, select the “SIGKILL” from this menu, and press enter.

5. Renice a Unix Process Without Exiting From hTop – Press F7 or F8

Select the process that needs to be reniced from the list, and press F7 which will reduce thenice value of the process, press F8 which will increase the nice value of the process by one. If you press F8 three times, it sets the nice value of the process to 3.

6. Display Only Processes of a Single User – Press u

Press u, which will display “Show processes of” menu and list all the users who has some processes running. Select a user from this list, and press Enter, which will display only the processes that belongs to that particular user.
Note: You can also do “htop -u ramesh” from the command line, which will start htop and display only process that belong to the user ramesh.

7. Display Processes in Tree View – Press F5 or t

Tree view is expremely helpful to identify dependencies of a process. Press F5 (or t), which will display all the related processes in a tree view as shown below.
Note: In a tree view, you can selectively collapse (or expand) a specific process tree by pressing either “+” or “-” key.

8. Change the Output Refresh Interval

To change the refresh interval of the htop output, use the -d command line option. “htop -d x”. Where x is referred in thenths of seconds.
htop -d 10
9. Htop Setup Menu – Press F2 or S
The following are the 4 high-level menu items available in htop setup.
  • Meters – Change the behaviour of the meter bar in the header
  • Display Options – Customize some of the htop display options
  • Colors – Choose a color theme for the htop output
  • Columns – This is extermly helpful when you are debugging a problem. Choose what column needs to be displayed for the processes in the htop output. This has more than 60 columns that you can choose from.

10. Strace a Process from htop – Press s

You can attach the selected process to strace by pressing s. This will display live update of the strace output displaying the system calls invoked by the selected process as shown below.

11. Display lsof Output inside htop – Press l

Select a process from the htop output and press l, which will display all the open files by the selected process. This uses lsof.

12. Scroll Process list Hoziontally and Vertically

Unlike top, in htop you can scroll the process list both horizontally and vertically. Select a process and press down arrow key which will scroll the list vertically and show you more processes in the list. You can also press “Page Up” or “Page Down” key to scoll the process list.
Select a process and press right arrow key which will scroll vertically to display the full command line of the process.

13. Follow a Process in htop Output – Press F

Select a process, and press F, which will follow the process. Since the htop output is sorted based on a column (by default %CPU), a process will keep changing its position in the list.
Following a process in the htop output is helpful when you are monitoring a particular process.

14. Kill Multiple Processes using Tag – Press Space and F9

You can group multiple processes by tagging them. Select a process, and press “Space bar”, which will tag the selected process. Once a process is tagged, the color of that line will change indicating that it is tagged. Scoll the list and tag multiple processes, and press F9, which will display the kill menu, from where you can kill all the tagged process.
Note: To untag a specific process, highlight that process, and press space bar again. To untag all processes press U.

15. Use Shortcut Character Keys (Instead of Function Keys)

The htop footer displays only the function keys to invoke the menu items. I find it easier to remember the character keys (instead of function keys). Also, if you are using putty connection manager (or some other ssh client), where the function keys are mapped to other functionalies, you need to use the following character keys to interact with htop.
Shortcut KeyFunction KeyDescription
hF1Invoke htop Help
SF2Htop Setup Menu
/F3Search for a Process
IF4Invert Sort Order
tF5Tree View
>F6Sort by a column
[F7Nice - (change priority)
]F8Nice + (change priority)
kF9Kill a Process
qF10Quit htop