Tuesday, January 17, 2012

24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring


This article provides a total of 24 examples on iostat, vmstat, and mpstat commands.
  • iostat reports CPU, disk I/O, and NFS statistics.
  • vmstat reports virtual memory statistics.
  • mpstat reports processors statictics.
This article is part of our ongoing Linux performance monitoring series.
Please note that iostat and vmstat are part of the sar utility. You should install sysstat package as explained in our sar (sysstat) article to get iostat and vmstat working.

IOSTAT EXAMPLES

1. iostat – Basic example

Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system as shown below.
$ iostat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.72      1096.66      1598.70 2719068704 3963827344
sda1            178.20       773.45      1329.09 1917686794 3295354888
sda2             16.51       323.19       269.61  801326686  668472456
sdb             371.31       945.97      1073.33 2345452365 2661206408
sdb1            371.31       945.95      1073.33 2345396901 2661206408
sdc             408.03       207.05       972.42  513364213 2411023092
sdc1            408.03       207.03       972.42  513308749 2411023092

2. iostat – Display only cpu statistics

iostat option -c, displays only the CPU usage statistics as shown below.
$ iostat -c
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

3. iostat – Display only disk I/O statistics

iostat option -d, displays only the disk I/O statistics as shown below.
$ iostat -d
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.71      1096.61      1598.63 2719068720 3963827704
sda1            178.20       773.41      1329.03 1917686810 3295355248
sda2             16.51       323.18       269.60  801326686  668472456
sdb             371.29       945.93      1073.28 2345452365 2661209192
sdb1            371.29       945.91      1073.28 2345396901 2661209192
sdc             408.01       207.04       972.38  513364213 2411024484
sdc1            408.01       207.02       972.38  513308749 2411024484

4. iostat – Display only network statistics

iostat option -n, displays only the device and NFS statistics as shown below.
$ iostat -n
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)        07/09/2011

avg-cpu:  %user   %nice    %sys %iowait   %idle
           4.33    0.01    1.16    0.31   94.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.83         0.35         5.39   29817402  457360056
sda1              3.32        50.18         4.57 4259963994  387641400
sda2              0.20         0.76         0.82   64685128   69718576
sdb               6.59        15.53        42.98 1318931178 3649084113
sdb1             11.80        15.53        42.98 1318713382 3649012985

Device:                  rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
192.168.1.4:/home/data      90.67        0.00         0.00         0.00         5.33         0.00
192.168.1.4:/backup         8.74         0.00         0.00         0.00         8.74         0.00
192.168.1.8:/media          0.02         0.00         0.00         0.00         0.01         0.00

5. iostat – Display I/O data in MB/second

By default iostat, displays the device I/O statistics in Blocks. To change it to MB, use -m as shown below.
$ iostat -m
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             194.70         0.54         0.78    1327670    1935463
sda1            178.19         0.38         0.65     936370    1609060
sda2             16.51         0.16         0.13     391272     326402
sdb             371.27         0.46         0.52    1145240    1299425
sdb1            371.27         0.46         0.52    1145213    1299425
sdc             407.99         0.10         0.47     250666    1177259
sdc1            407.99         0.10         0.47     250639    1177259

6. iostat – Display I/O statistics only for a device

By default iostat displays I/O data for all the disks available in the system. To view statistics for a specific device (For example, /dev/sda), use the option -p as shown below.
$ iostat -p sda
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.51      1598.48 2719069928 3963829584
sda2            336.38        27.17        54.00   67365064  133905080
sda1            821.89         0.69       243.53    1720833  603892838

7. iostat – Display timestamp information

By default iostat displays only the current date. To display the current time, use the option -t as shown below.
$ iostat -t
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Time: 08:57:52 AM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.49      1598.45 2719070384 3963829704
sda1            178.18       773.32      1328.88 1917688474 3295357248
sda2             16.51       323.14       269.57  801326686  668472456
sdb             371.25       945.82      1073.16 2345452741 2661228872
sdb1            371.25       945.80      1073.16 2345397277 2661228872
sdc             407.97       207.02       972.27  513364233 2411030200
sdc1            407.97       207.00       972.27  513308769 2411030200

8. iostat – Display Extended status

Use option -x, which will displays extended disk I/O statistics information as shown below.
$ iostat -x
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              27.86    63.53 61.77 132.91  1096.46  1598.40    13.84     0.21    1.06   2.28  44.45
sda1              0.69    33.22 48.54 129.63   773.30  1328.84    11.80     1.39    7.82   2.28  40.57
sda2             27.16    30.32 13.23  3.28   323.13   269.56    35.90     0.55   32.96   3.44   5.68
sdb              39.15   215.16 202.20 169.04   945.80  1073.13     5.44     1.05    2.78   1.64  60.91
sdb1             39.15   215.16 202.20 169.04   945.77  1073.13     5.44     1.05    2.78   1.64  60.91
sdc               8.90     3.63 356.56 51.40   207.01   972.24     2.89     1.04    2.56   1.55  63.30
sdc1              8.90     3.63 356.55 51.40   206.99   972.24     2.89     1.04    2.56   1.55  63.30
To display extended information for a specific partition (For example, /dev/sda1), do the following.
$ iostat -x sda1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda1              0.69    33.21 48.54 129.62   773.23  1328.76    11.80     1.39    7.82   2.28  40.56

9. iostat – Execute Every x seconds (for y number of times)

To execute iostat every 2 seconds (until you press Ctl-C), do the following.
$ iostat 2
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.67      1096.39      1598.33 2719070584 3963891256
sda1            178.16       773.26      1328.79 1917688482 3295418672
sda2             16.51       323.11       269.54  801326878  668472584
sdb             371.22       945.74      1073.08 2345454041 2661251200
sdb1            371.22       945.72      1073.08 2345398577 2661251200
sdc             407.93       207.00       972.19  513366813 2411036564
sdc1            407.93       206.98       972.19  513311349 2411036564
..
To execute every 2 seconds for a total of 3 times, do the following.
$ iostat 2 3

10. iostat – Display LVM statistic (and version)

To display the LVM statistics use option -N as shown below.
$ iostat -N
To display the version of iostat, use -V. This will really display the version information of sysstat, as iostat is part of sysstat package.
$ iostat -V
sysstat version 7.0.2
(C) Sebastien Godard

VMSTAT EXAMPLES

11. vmstat – Basic example

vmstat by default will display the memory usage (including swap) as shown below.
$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 260688  29160 2356920    2    2     4     1    0    0  6  1 92  2  0
vmstat output contains the following fields:
  • Procs – r: Total number of processes waiting to run
  • Procs – b: Total number of busy processes
  • Memory – swpd: Used virtual memory
  • Memory – free: Free virtual memory
  • Memory – buff: Memory used as buffers
  • Memory – cache: Memory used as cache.
  • Swap – si: Memory swapped from disk (for every second)
  • Swap – so: Memory swapped to disk (for every second)
  • IO – bi: Blocks in. i.e blocks received from device (for every second)
  • IO – bo: Blocks out. i.e blocks sent to the device (for every second)
  • System – in: Interrupts per second
  • System – cs: Context switches
  • CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time

12. vmstat – Display active and inactive memory

By default vmstat doesn’t display this information. Use option -a, to display active and inactive memory information as shown below.
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 253820 1052680 2688928    2    2     4     1    0    0  6  1 92  2  0

13. vmstat – Display number of forks since last boot

This displays all the fork system calls made by the system since the last boot. This displays all fork, vfork, and clone system call counts.
$ vmstat -f
     81651975 forks

14. vmstat – Execute Every x seconds (for y number of times)

To execute every 2 seconds, do the following. You have to press Ctrl-C to stop this.
$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0
..
To execute every 2 seconds for 10 times, do the following. You don’t need to press Ctrl-C in this case. After executing 10 times, it will stop automatically.
$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0
..

15. vmstat – Display timestamp

When you use vmstat to monitor the memory usage repeately, it would be nice to see the timestap along with every line item. Use option -t to display the time stamp as shown below.
$ vmstat -t 1 100
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608728 148368 3898200    0    0     0     0    1    1  0  0 100  0  0     2011-07-09 21:16:28 PDT
 0  0      0 3608728 148368 3898200    0    0     0     0   60   15  0  0 100  0  0     2011-07-09 21:16:29 PDT
 0  0      0 3608712 148368 3898200    0    0     0     0   32   28  0  0 100  0  0     2011-07-09 21:16:30 PDT
For me, the timestamp option worked in the following version.
$ vmstat -V
procps version 3.2.8
Note: If you use a older version of vmstat, option -t might not be available. In that case, use the method we suggested earlier to display timestamp in vmstat output.

16. vmstat – Display slab info

Use option -m, to display the slab info as shown below.
$ vmstat -m
Cache                       Num  Total   Size  Pages
fib6_nodes                    5    113     32    113
ip6_dst_cache                 4     15    256     15
ndisc_cache                   1     15    256     15
RAWv6                         7     10    768      5
UDPv6                         0      0    640      6
tw_sock_TCPv6                 0      0    128     30
...

17. vmstat – Display statistics in a table format

Instead of displays the values in the record format, you can display the output of vmstat in table format using option -s as shown below.
$ vmstat -s
      4149928  total memory
      3864824  used memory
      2606664  active memory
      1098180  inactive memory
       285104  free memory
        19264  buffer memory
      2326692  swap cache
      4192956  total swap
       274872  used swap
      3918084  free swap
   1032454000 non-nice user cpu ticks
        14568 nice user cpu ticks
     89482270 system cpu ticks
  16674327143 idle cpu ticks
    368965706 IO-wait cpu ticks
      1180468 IRQ cpu ticks
..

18. vmstat – Display disk statistics

Use option -d to display the disk statistics as shown below. This displays the reads, writes, and I/O statistics of the disk.
$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
       total merged sectors      ms  total merged sectors      ms    cur    sec
sda   153189971 69093708 2719150864 737822879 329617713 157559204 3965687592 4068577985      0 1102243
sdb   501426305 97099356 2345472425 731613156 419220973 533565961 2661869460 1825174087      0 1510434
sdc   884213459 22078974 513390701 452540172 127474901 8993357 2411187300 2133226954      0 1569758

19. vmstat – Increase the width of the display

The default output without increasing the width is shown below.
$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608688 148368 3898204    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   72   30  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   60   27  0  0 100  0  0
Use option -w to increase the width of the output columns as shown below. This give better readability.
$ vmstat -w 1 3
procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
 r  b       swpd       free       buff      cache   si   so    bi    bo   in   cs  us sy  id wa st
 0  0          0    3608712     148368    3898204    0    0     0     0    1    1   0  0 100  0  0
 0  0          0    3608712     148368    3898204    0    0     0     0   93   23   0  0 100  0  0
 0  0          0    3608696     148368    3898204    0    0     0     0   35   34   0  0 100  0  0

20. vmstat – Display statistics for a partition

To display the disk I/O statistics of a specific disk partition use option -p as shown below.
$ vmstat -p sdb1
sdb1          reads   read sectors  writes    requested writes
           501423248 2345417917  419221612 2661885948

21. vmstat – Display in MB

By default vmstat displays the memory information in kb. To disply in MB, use the option “-S m” as shown below.
$ vmstat -S m
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    281    288     19   2386    0    0     4     1    0    0  6  1 92  2  0

MPSTAT EXAMPLES

22. mpstat – Display basic info

By default mpstat displays CPU statistics as shown below.
$ mpstat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

10:25:32 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
10:25:32 PM  all    5.68    0.00    0.49    2.03    0.01    0.02    0.00   91.77    146.55

23. mpstat – Display all information

Option -A, displays all the information that can be displayed by the mpstat command as shown below. This is really equalivalent to “mpstat -I ALL -u -P ALL” command.
$ mpstat -A
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:26:34 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:26:34 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:26:34 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

10:26:34 PM  CPU    intr/s
10:26:34 PM  all     36.51
10:26:34 PM    0      0.00
10:26:34 PM    1      0.00
10:26:34 PM    2      0.04
10:26:34 PM    3      0.00

10:26:34 PM  CPU     0/s     1/s     8/s     9/s    12/s    14/s    15/s    16/s    19/s    20/s    21/s    33/s   NMI/s   LOC/s   SPU/s   PMI/s   PND/s   RES/s   CAL/s   TLB/s   TRM/s   THR/s   MCE/s   MCP/s   ERR/s   MIS/s
10:26:34 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    7.47    0.00    0.00    0.00    0.00    0.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.90    0.00    0.00    0.00    0.00    0.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.04    0.00    0.00    0.00    0.00    0.00    3.32    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.17    0.00    0.00    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00

24. mpstat – Display CPU statistics of individual CPU (or) Core

Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.
$ mpstat -P ALL
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:28:04 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:04 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:28:04 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:28:04 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
To display statistics information of a particular CPU (or core), use option -P as shown below.
$ mpstat -P 0
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:53 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:53 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98

$ mpstat -P 1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:55 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:55 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
Finally, as we mentioned earlier mpstat is part of the sysstat package. When you do mpstat -V, it will really display the version number of the systat package as shown below.
$ mpstat -V
sysstat version 9.0.4
(C) Sebastien Godard (sysstat  orange.fr)