Sunday, December 11, 2011

BOOT PROCESS IN AIX



SUMMARY
=======




Loading the boot image of AIX

  •  After POST, the firmware (System Read Only Storage) detectsthe 1st bootable device stored in the bootlist. (here it is hdisk0)
  • then the bootstrap code (software ROS) i.e. 1st 512 bytes of the hard disk loads to RAM.
  • Bootstrap code locates the Boot Logical Volume (BLV = hd5) from the harddisk
  • BLV contains AIX kernel, rc.boot ScriptReduced ODM andBoot commands.
  • Then BLV in the RAM uncompresses and Kernel releases from it.
  • Then AIX Kernel gets control.
  • AIX Kernel creates a RAM File System (Rootvg not activated now).
  • kernel starts init process from the BLV.
  • init executes rc.boot script from the BLV in the RAM.
  • Init with rc.boot 1 configures base devices.
rc.boot 1 in detail

  • init process from RAMFS executes rc.boot 1 (if any error LED=c06)
  • restbase command copies ODM from BLV to RAMFS.(success LED=510, error LED=548)
  • cfgmgr -f calls Config_rules ( which are phase=1) andactivates all base devices.
  • run command bootinfo -b to check last boot device ( success LED=511).
Then 
  • rc.boot 2 activates rootvg from hard disk.

rc.boot 2 in detail

  • rc.boot 2 (LED 551)
  • ipl_varyon to activate rootvg ( success LED= 517, error LED=552,554,556).
  • run command fsck -f /dev/hd4 to check whether "/" unmounted uncleanely in the last shutdown ( error LED=555).
  • mount /dev/hd4 (/) to RAMFS (error LED=557 due to corrupted jfslog..)
  •  fsck -f /dev/hd2  i.e.  "/usr" ( error LED=518).
  • mount /dev/hd2 in RAMFS.
  • fsck -f /dev/hd9var  i.e. check "/var"
  • mount /var
  • copycore command checks whether dump occured. then copy dump from primary dump device paging space (/dev/hd6) to/var/adm/ras/.
  • unmount /var
  • swapon /dev/hd6  i.e. activate primary paging space.
Now the condition is /dev/hd4 is mounted on in the RAMFS;
cfgmgr -f configured all base devices . so configuration data has been written to ODM of RAMFS.
  • mergedev is called and copy /dev from RAMFS to disk.
  • copy customized ODM from RAMFS to hard disk(at this stage both ODM from hd5 and hd4 are sync now)
  • mount /var.
  • Boot messages copy to file on the hard disk (/var/adm/ras/bootlog)    alog -t boot -o to view bootlog
Now / , /usr and /var are mounted in rootvg on the hard disk. Then
  • Kernel removes RAMFS
  • init process start from in the rootvg
Here completes rc.boot 2, Now the condition is kernel removed RAMFSand accessing rootvg filesystems from hard disk. init from BLV replacedby init from hard disk
  • in rc.boot 3init process /etc/inittab file and remainingdevices are configured.
rc.boot 3 in detail
  •  /etc/init starts and reads /etc/inittab ( LED=553)
  •  runs /sbin/rc.boot 3 
  • fsck -f /dev/hd3 i.e. check /tmp.
  • mount /tmp
  • sysncvg rootvg &;    i.e. run syncvg in background and reportstale PPs.
  • cfgmgr -P2  i.e. run cfgmgr in phase 2 in normal startup. (cfgmgr -P3 in service mode)
  • All remaining devices are configured now.
  • cfgcon configures console ( LED= c31 select console, c32 lft, c33tty, c34 file on disk). If CDE mentioned in /etc/inittab we will getgraphical console.
  • savebase calls to sync ODM from BLV with / FS (i.e./etc/objrepos).
  • syncd daemon started. All data from cache memory to disksaves in every 60 seconds.
  • starts errdaemon for error logging.
  • LED display turned OFF.
  • rm /etc/nologin i.e. if the file is not removed, then login is not possible.
  • If any device are in missed state, (in Cudv chgstatus=3)display it.
  • Display "system initialization completed"
Then execute next line from /etc/inittab 

DETAILED
======= 

I. The boot process in AIX
As a system administrator you should have a general understanding of theboot process. This knowledge is useful to solve problems that can prevent asystem from booting properly. These problems can be both software or hardware.We also recommend that you be familiar with the hardware configuration of your system. 

Booting involves the following steps:

The initial step in booting a system is named Power On Self Test (POST). Its purpose is to verify that basic hardware is in functional state.The memory, keyboard, communication and audio devices are also initialized. You can see an image for each of these devices displayed on the screen. It is during this step that you can press a function key to choose a different boot list. TheLED values displayed during this phase are model specific. Both hardware and software problems can prevent the system from booting.

System Read Only Storage (ROS) is specific to each system type. It is necessary for AIX 5L Version 5.3 to boot, but it does not build the datastructures required for booting. It will locate and load bootstrap code.System ROS contains generic boot information and is operating system independentSoftware ROS (also named bootstrap) forms an IPL control block which is compatible with AIX 5L Version 5.3, takes control and builds AIX 5L
specific boot information. A special file system located in memory and named RAMFS file system is created. Software ROS then locates, loads, and turns control over to AIX 5L Boot Logical Volume (BLV). Software ROS is AIX 5L information created based on machine type and is responsible for completing machine preparation to enable it to start AIX 5L kernel. A complete list of files that are part of the BLV can be obtained from directory/usr/lib/boot.

The most important components are the following:

- The AIX 5L kernel
- Boot commands called during the boot process such as bootinfo, cfgmgr
- A reduced version of the ODM. Many devices need to be configured hd4 (/) made available, so their corresponding methods have to be stored in the BLV. These devices are marked as base in PdDv.
- The rc.boot script

Note: Old systems based on MCI architecture execute an additional stepbefore this, the so called Built In Self Test (BIST). This step is no longer required for systems based on PCI architecture.

The AIX 5L kernel is loaded and takes control. The system will display0299 on the LED panel. All previous codes are hardware-related. The kernelwill complete the boot process by configuring devices and starting the init process. LED codes displayed during this stage will be generic AIX 5Lcodes. So far, the system has tested the hardware, found a BLV, created theRAMFS, and started the init process from the BLV. The rootvg has not yet been activated. From now on the rc.boot script will be called three times,each timebeing passed a different parameter.

1.Boot phase 1

During this phase, the following steps are taken:

The init process started from RAMFS executes the boot script rc.boot 


If init process fails for some reason, code c06 is shown on LED display. At this stage, the restbase command is called to copy a partial image of ODM from the BLV into the RAMFS. If this operation is successful LED display shows 510, otherwise LED code 548 is shown.

After this, the cfgmgr -f command reads the Config_Rules class from the reduced ODM. In this class, devices with the attribute phase=1 are considered base devices. Base devices are all devices that are necessary to access rootvg.
For example, if the rootvg is located on a hard disk all devices starting from motherboard up to the disk will have to be initialized.The corresponding methods are called so that rootvg can be activated in the nextboot phase 2. At the end of boot phase 1, the bootinfo -b command is called to determine the last boot device. At this stage, the LED shows 511.
2.Boot phase 2
In this phase , the rc.boot script is passed to the parameter 2. During this phase the following steps are taken.

a) The rootvg volume group is varied on with the special version of the varyonvg command ipl_varyon. If this command is successful the system displays 517. otherwise one of the following LED will appear 552,554,556and the boot process is halted.

b) Root file system hd4 is checked using the fsck -f command. This will verify only whether the filesystem was unmounted cleanly before the last shutdown. If this command fails, the system will display code 555.

c) The root file system ( /dev/hd4 ) is mounted on a temporary mount point /mnt in RAMFS. If this fails 557 will appear in LED.

d) The /usr file system is verified using fsck -f command and then mounted. the copycore command checks if a dump occured. if it did, it is copied from default dump devices, /dev/hd6 to the default copy directory/var/adm/ras. After this /var is unmounted.

e) The primary pagingspace from rootvg, /dev/hd6 will be activated.

f) The mergedev process is called and /dev files from RAMFS are copiedto disk.

g) All customized ODM files from the RAMFS are copied to disk.Both ODM versions from hd4 and hd5 are synchronized.

h) Finaly, the root file system from rootvg (disk) is mounted over the root file system from the RAMFS. The mount points for the root filesystems become available. now the /var and /usr file systems from the rootvg aremounted again on their ordinary mount points. There is no consoleavailable at this stage; so all boot messages will be copied to alog. The alog command maintains and manages logs.


3.Boot Phase 3

After phase 2 is completed rootvg is activated and the following steps are taken,

a.
 /etc/init process is started. It reads /etc/inittab file and calls rc.bootwith argument 3

b. The /tmp filesystem is mounted.

c. The rootvg is synchronized by calling the synchvg command and launching it as background process. As a result all stale partitions from rootvg are updated.At this stage LED code 553 is shown.

d. At this stage the cfgmgr command is called.if the system is booted innormal mode the cfgmgr command is called with option -p2; in servicemode, option -p3. The cfgmgr command reads the Config_rules files fromODM and calls all methods corresponding to either phase 2 or 3. All otherdevices that are not base devices are configured at this time.

e. Next the console is configured by calling the cfgcon command. After the configuration of the console , boot messages are send to the console if no STDOUT redirection is made. However all missed messages can be found in/var/adm/ras/conslogLED codes that can be displayed at this time are :

c31 = console not yet configured.
c32 = console is an LFT terminal.
c33 = console is a TTY.
c34 = console is a file on the disk.

f. finally the synchronization of the ODM in the BLV with the ODM from the/ (root) filesyatem is done by the savebase command.

g. 
The syncd daemon and errdaemon are started.

h. LED display is turned off.

i. if the /etc/nologin exists, it will be removed.

j. If there are devices marked as missing in CuDv a message is displayed on the console.

i. the message system initialization completed is send to the console. the execution of the rc.boot has completedinit process will continueprocessing the next command from /etc/inittab.

II. system initialization
During system startup, after the root file system has been mounted in the pre-initialization process, the following sequence of events occurs:

1. The init command is run as the last step of the startup process.
2. The init command attempts to read the /etc/inittab file.
3. If the /etc/inittab file exists, the init command attempts to locate aninitdefauult entry in the /etc/inittab file.

a. If the initdefault entry exists, the init command uses the specifiedrunlevel as the initial system run level.
b. If the initdefault entry does not exist, the init command requests that the user enter a run level from the system console (/dev/console).
c. If the user enters an S, s, M, or m run level, the init command enters themaintenance run level. These are the only runlevels that do not requirea properly formatted /etc/inittab file.

4. If the /etc/inittab file does not exist, the init command places the system in the maintenance run level by default.
5. The init command rereads the /etc/inittab file every 60 seconds. If the /etc/inittab file has changed since the last time the init command read it, the new commands in the /etc/inittab file are executed.

III. The /etc/inittab file

The /etc/inittab file controls the initialization process.

The /etc/inittab file supplies the script to the init command's role as a general process dispatcher. The process that constitutes the majority of the init command's process dispatching activities is the /etc/getty line process, which initiates individual terminal lines. Other processes typically dispatched by the init command are daemons and the shell.

The /etc/inittab file is composed of entries that are position-dependentand have the following format,

/etc/inittab format = Identifier:RunLevel:Action:Command

Each entry is delimited by a newline character. A backslash (\) preceding a new line character indicated the continuation of an entry. There are no limits(other than maximum entry size) on the number of entries in the /etc/inittab file.

The maximum entry size is 1024 characters.

The entry fields are : 

Identifier
A one to fourteen character field that uniquely identifies an object.

RunLevel
The run level at which this entry can be processed. The run level has the following attributes: 

-Run levels effectively correspond to a configuration of processes in the system.

-Each process started by the init command is assigned one or more run levels in which it can exist.

-Run levels are represented by the numbers 0 through 9

Eg: if the system is in run level 1, only those entries with a 1 in the run-level field are started.

-When you request the init command to change run levels, all processes without a matching entry in the run-level field for the target run level receive a warning signal (SIGTERM). There is a 20-second grace period before processes are forcibly terminated by the kill signal (SIGKILL).

-The run-level field can define multiple run levels for a process by selecting more than one run level in any combination from 0 through 9. If no run levelis specified, the process is assumed to be valid at all run levels.

-There are four other values that appear in the run-level field, even though they are not true run levels: a, b, c and h. Entries that have these characters in the run level field are processed only when the telinit command requests them to be run (regardless of the current run level of the system). They differ from run levels in that the init command can never enter run level a, b, c or h. Also, a request for the execution of any of these processes does not change the current run level. Furthermore, a process started by an a, b,or c command is not killed when the init command changes levels. They are only killed if their line in the /etc/inittab file is marked off in the action field, their line is deleted entirely from /etc/inittab, or the init command goes into single-user mode.

Action
It tells the init command how to treat the process specified in the process field. The following actions are recognized by the init command:

respawn If the process does not existstart the process. Do not wait forits termination (continue scanning the /etc/inittab file). Restart the process when it dies. If the process existsdo nothing and continue scanning the /etc/inittab file.

wait When the init command enters the run level that matches the entry's run level, start the process and wait for its termination. All subsequent reads of the /etc/inittab file, while the init command is in the same run level, will cause the init command to ignore this entry.

once When the init command enters a run level that matches the entry's run level, start the process, and do not wait for termination. When it diesdo not restart the process. When the system enters a new run level, and the process is still running from a previous run level change, the program will not be restarted.

boot Process the entry only during system boot, which is when the init command reads the /etc/inittab file during system startupStart the process, do not wait for its termination, and when it diesdo not restart the process. In order for the instruction to be meaningful, the run level should be the default or it must match the init command's run level at boot time. This action is useful for an initialization function following a hardware reboot of the system.

bootwait Process the entry the first time that the init command goes fromsingle-user to multi-user state after the system is booted. Start the process, wait for its termination, and when it diesdo not restart the process. If the initdefault is 2, run the process right after boot.

powerfail Execute the process associated with this entry only when the init command receives a power fail signal ( SIGPWR).

powerwait Execute the process associated with this entry only when the init command receives a power fail signal (SIGPWR), and wait until itterminates before continuing to process the /etc/inittab file. 

off If the process associated with this entry is currently running, send thewarning signal (SIGTERM), and wait 20 seconds before terminating the process with the kill signal (SIGKILL). If the process is not running, ignore this entry.

ondemand Functionally identical to respawn, except this action applies to the a, b, or c values, not to run levels.

initdefault An entry with this action is only scanned when the init command is initially invoked. The init command uses this entry, if it exists, to determine which run level to enter initially. It does this by taking the highest run level specified in the run-level field and using that as its initial state. If the run level field is empty, this is interpreted as 0123456789. therefore, the init command enters run level 9. Additionally, if the init command does not find an initdefault entry in the /etc/inittab file, it requests an initial run level from the user at boot time. 

sysinit Entries of this type are executed before the init command tries toaccess the console before login. It is expected that this entry will only be used to initialize devices on which the init command might try to ask the run level question. These entries are executed and waited for before continuing. 

Command
A shell command to execute. The entire command field is prefixed with exec and passed to a forked sh as sh -c exec command. Any legal sh command syntax can appear in this field. Comments can be inserted with the # comment syntax. 

The getty command writes over the output of any commands that appear before it it in the /etc/inittab file. To record the output of these commands to the boot log, pipe their output to the alog -tboot command. The stdin,stdout, and stderr file descriptors may not be available while the init command is processing inittab entries. Any entries writing to stdout or stderr may not work predictably unless they redirect their output to a file or to /dev/console.
The following commands are the only supported methods for modifying the records in the /etc/inittab file. 

lsitab Lists records in the /etc/inittab file.
mkitab Adds records to the /etc/inittab file.
chitab Changes records in the /etc/inittab file.
rmitab Removes records from the /etc/inittab file.

Eg:

If you want to add a record on the /etc/inittab file to run the find command on the run level 2 and start it again once it has finished: 


1. Run the ps command and display only those processes that contain the word find: 
# ps -ef | grep find
root 19750 13964 0 10:47:23 pts/0 0:00 grep find
#
2. Add a record named xcmd on the /etc/inittab using the mkitab command:
mkitab "xcmd:2:respawn:find / -type f > /dev/null 2>&1"
3. Show the new record with the lsitab command:
# lsitab xcmd
xcmd:2:respawn:find / -type f > /dev/null 2>&1
#
4. Display the processes:
# ps -ef | grep find
root 25462 1 6 10:56:58 - 0:00 find / -type f
root 28002 13964 0 10:57:00 pts/0 0:00 grep find
#
5. Cancel the find command process:
# kill 25462
6. Display the processes:
# ps -ef | grep find
root 23538 13964 0 10:58:24 pts/0 0:00 grep find
root 28966 1 4 10:58:21 - 0:00 find / -type f
#

Since the action field is configured as respawn, a new process (28966 in this example) is started each time its predecessor finishes. The process will continue re-spawning, unless you change the action field,

Eg: 

1. Change the action field on the record xcmd from respawn to once:
chitab "xcmd:2:once:find / -type f > /dev/null 2>&1"
2. Display the processes:
# ps -ef | grep find
root 20378 13964 0 11:07:20 pts/0 0:00 grep find
root 28970 1 4 11:05:46 - 0:03 find / -type f
3. Cancel the find command process:
# kill 28970
4. Display the processes:
# ps -ef | grep find
root 28972 13964 0 11:07:33 pts/0 0:00 grep find
#

To delete this record from the /etc/inittab file, you use the rmitab command.

Eg:

rmitab xcmd
# lsitab xcmd
#

Order of the /etc/inittab entries

The base process entries in the /etc/inittab file is ordered as follows:

1. initdefault
2. sysinit
3. Powerfailure Detection (powerfail)
4. Multiuser check (rc)
5. /etc/firstboot (fbcheck)
6. System Resource Controller (srcmstr)
7. Start TCP/IP daemons (rctcpip)
8. Start NFS daemons (rcnfs)
9. cron
10.pb cleanup (piobe)
11.getty for the console (cons)

The System Resource Controller (SRC) has to be started near the begining of the etc/inittab file since the SRC daemon is needed to start other processes. Since NFS requires TCP/IP daemons to run correctly, TCP/IP daemons are started ahead of the NFS daemons. The entries in the /etc/inittab file are ordered according to dependencies, meaning that if a process (process2) requires that another process (process1) be present for it to operate normally, then an entry for process1 comes before an entry for process2 in the /etc/inittab file. 

rzip - great compression rates for logs and such


The rzip program is huge-scale data compression software designed around initial LZ77-style string matching on a 900 MB dictionary window, followed by Bzip2-based Burrows-Wheeler transform (BWT) and entropy coding (Huffman) on 900 kB output chunks. The first stage of rzip is similar to that of rsync, and, no wonder: they're both authored by Andrew Tridgell, author of Samba (and lots more).

Here are some "benchmarks" I did of rzip on a collection of binary logs and transaction logs. Compress ratio would be even better on plain text logs.

If you have to store huge amounts of log files, maybe archive them for undetermined periods of time, sometimes on write once media (WORM) such as DVD-R, BD-R or HP UDO to provide compliance with unalterable retention policies or being able to easily destroy media (just pop in the DVD in the microwave for 3 seconds on 700W) then you really should look at rzip. Just set your logs to rotate at 1-2 GB, maybe even trying to compensate for compression(to avoid filesystem limitations such as ISO vs. UDF, FAT32 and even some tools that can't handle big files) then compress them with rzip!



While on the system itself you can store them on a compressed filesystem (and I've gotten quite good compression ratios using ZFS compression) once you dump them to tape or write once media, it's a different story.

rzip -9 logs.tar 155.00s user 2.17s system 99% cpu 2:37.79 total

gzip -9 logs.tar 138.90s user 4.32s system 99% cpu 2:23.79 total

bzip2 -9 logs.tar 311.03s user 2.17s system 99% cpu 5:14.61 total

7za a logs.tar.7za logs.tar 805.75s user 11.16s system 165% cpu 8:12.84 total


-rw-r--r-- 1 cmihai sysadmin 899M Mar 6 14:33 logs.tar
-rw-r--r-- 1 cmihai sysadmin 528M Mar 6 14:43 logs.tar.7za
-rw-r--r-- 1 cmihai sysadmin 576M Mar 6 14:12 logs.tar.bz2
-rw-r--r-- 1 cmihai sysadmin 587M Mar 6 14:09 logs.tar.gz
-rw-r--r-- 1 cmihai sysadmin 223M Mar 6 14:30 logs.tar.rz


But this great compression ration does come at a price: memory usage. rzip uses a history buffer of 900MB compared to 32kb for gzip and 900k for bzip. And again, memory is cheap and plenty now, and it's not really that big of an issue. The other major issue is: you can't pipe.

Rzip uses a two stage process. The first stage of rzip is very similar to that of rsync. It finds and encodes large data segments using a 900MB history buffer. The second stage is basically bzip2.

Still, as you can see, it can be faster than bzip at times. It's speed is actually comparable to that of gzip.


Memory usage:
13960 cmihai 1520K 1048K cpu0 0 0 0:00:17 28% gzip/1
13965 cmihai 8848K 7584K cpu0 0 19 0:00:22 32% bzip2/1
13967 cmihai 643M 642M cpu0 0 0 0:00:14 24% rzip/1



Now for a larger quantity of log files:


~/rzip -9 biglogs.tar 1815.89s user 69.63s system 98% cpu 31:48.57 total

-rw-r--r-- 1 cmihai sysadmin 2.7G Mar 6 16:00 biglogs.tar.rz
-rw-r--r-- 1 cmihai sysadmin 9.5G Mar 6 15:27 biglogs.tar


As you can see, rzip proves to be quite the disk space saver :-). If you have to archive logs on a regular basis, consider giving rzip a spin. Though I can't stress enough that you should do a couple of benchmarks _yourself_, using the kind of data you're trying to archive. Just take a couple of samples and tar them up (1GB, 10GB sound like fair values), then do a simple time rzip / time gzip / time bzip and check the results. But, like I've said, expect memory usage for rzip to be around 500-900MB .

If it's logs you're archiving, decompression times shouldn't be an issue, but you should and least try to decompress the archive on a couple machines. Also, try testing with large files, see if there are any filesystem or application limitations you need to worry about.

Note: these tests have been done with an older version of rzip, on a Solaris machine (so yeah, it is quite portable). Newer versions of rzip are faster, and have an even better compression ratio.

There are also other implementations of rzip that use LZMA (Lempel-Ziv-Markov chain algorithm) as as the second stage, instead of bzip. Long Range ZIP or Lzma RZIP is such an implementation.

A few more tools to unerase files


A few free tools to restore deleted files:
  • Undelete Plus works under Win 95/98/Me/NT/2000/XP/2003/Vista operating systems. The program supports all Windows file systems for hard and floppy drives including FAT12/16/32,NTFS/NTFS5 and image recovery from CompactFlash, SmartMedia, MultiMedia and Secure Digital cards.
  • Recuva (pronounced "recover") is a freeware Windows utility to restore files that have been accidentally deleted from your computer.
  • Restoration - free undelete tool for Windows.
  • NTFS Undelete - free undelete tool for Windows.
  • R-LINUX is a free file recovery utility for the Ext2FS partitions used in the Linux OS and several Unix. Host OS: Win9x/ME/NT/2000/XP/2003. Recovered data can be written to any disk visible by the host OS. R-Linux also can create DISK IMAGES that can be later processed by more powerful R-Studio. more about Data Recovery for Linux
  • Ultimate Data Recovery - data recovery tool.

Disk monitoring and tuning with dd and S.M.A.R.T. - Reallocating bad sectors and predicting disk failure


What is S.M.A.R.T.?
Modern disk drives will automagically reallocate bad sectors on the fly, as soon as they encounter some kind of R/W/ECC error. But in order for this to happen, it must first access that sector. This is why you never see surface errors on modern disks.

Modern hard drives (ATA and SATA) have S.M.A.R.T. - Self-Monitoring, Analysis, and Reporting Technology. Once you have that enabled in BIOS (assuming you have a S.M.A.R.T. capable disk and controller) you can monitor a number of disk health and performance parameters.

What you should keep an eye on is the Reallocated Sectors Count (if the drive has a problem with a R/W/ECC error it will mark the sector "Reallocated" and transfer the data to a spare area on the disk). This will result in some performance decrease, and is a sign of imminent disk failure.


Monitoring S.MA.R.T.

ATA and SATA disks:

To monitor S.M.A.R.T. data you can use HDTune on Windows or SmartMonTools (smartd, smartctl) on Darwin (Mac OSX), Linux, FreeBSD, NetBSD, OpenBSD, Solaris, OS/2, or eComStation systems. If you're up to it, you can also use SmartMonTools on Windows.

USB Enclosures:While in most cases you should have no trouble using HDTune or SmartMonTools, some USB drive enclosures may be resilient to monitoring with S.M.A.R.T. programs and will require vendor software. In such cases, you can download vendor software to perform monitoring, like "Western Digital Data LifeGuard Diagnostics".

iPods:You can also get S.M.A.R.T. info on your iPod. You can either configure it to act as a pass through device (regular USB media) or boot your iPod in diagnostic mode. You can check S.M.A.R.T. disk data and perform more test on your iPod. To do so, you must reset your iPod and hold REW + Select (5G) at the Apple boot menu. For other iPod models, see here (or Google Apple Diagnostic Mode your iPod Model).

Forcing the disk to remap damanged sectors
Now you should know that if you see any problems with Reallocated Sector Count, Reallocated Event Count, Seek Error Rate, Offline Uncorrectable, UDMA CRC Error Count, Multizone Error Rate, Hardware ECC Recovered values, you should consider getting a new disk. These are all signs of a failing disk. Learn more about S.M.A.R.T. attributes and their meaning here. Note that depending on vendor, there may also be enhanced or propriotary S.M.A.R.T. attributes. Read your HDD vendor documentation.

But sometimes you just need to get a bit more life out of a disk, and force the disk to reallocated damaged sectors. You can do so easily by performing a full raw disk read and write operation. For this, you can use the UNIX "dd" tool. Make sure your target disks aren't mounted (Type "mount" to list mounted disks then use "umount disk").

You can perform a disk read operation (reading the whole disk) using a syntax similar to:
# dd if=/dev/disk of=/dev/null bs=2048
You can perform a disk write operation (zero out the disk, this WILL result in data loss) using syntax similar to:
# dd if=/dev/zero of=/dev/disk bs=2048
Now you may wish to perform both a read and write at the same time, and not wipe out your disk data (zero it out). You can perform such a "disk refresh" using syntax similar to:
# dd if=/dev/disk of=/dev/disk bs=1m
This will read and rewrite the data to disk in 1MB chunks to prevent presently recoverable read errors from progressing into unrecoverable read errors.

Of course, you should read the dd manpage for your OS (on Windows you could use a dd for Windows implementation or resort to some sort of Linux or BSD LiveCD). Replace /dev/disk with your disk (make sure you're using the right disk). On Linux you can find out what disk you need to use from "dmesg" or /proc/partitions:
# cat /proc/partitions
You can also use "fdisk -l" to list partitons on your disk, see if that's the right disk
# fdisk -l /dev/hda
Do note that you need root permissions for all of this activity, so on some Linux systems you may need to use "sudo -i" to get a root shell, or precede all operations with "sudo".

While you're doing this rewrite operation, you should monitor the kernel log (dmesg). You can monitor /var/log/messages for this:
# tail -f /var/log/messages
You usually watch out for "DriveReady SeekComplete Error status=0x51 DriveStatusError error=0x04" or some other error.

You should also keep an eye on the Reallocated Sectors and other Interesting Parameters in smartctl:
# smartctl -A /dev/hda
Do this every now and then, and note the values before you've started the operation.

Once you begin the "dd" operations you can send dd a SIGINFO signal (use pkill / kill / whatever) to make it print out I/O information (progress). Some shells / TERMS also respond to Ctrl-T by sending SIGINFO.
# pkill -SIGINFO dd
Once you're done with dd and S.M.A.R.T. tools you should also perform a filesystem check (fsck / chkdsk / whatever).

Conclusions:
  1. Monitor S.M.A.R.T. data with smartclt, keep an eye on Reallocs. Consider getting a new disk if you see reallocated sectors
  2. Perform a disk refresh with dd in order to prevent recoverable read errors from progressing into unrecoverable errors. You don't need fancy tools like SpinRite.
  3. You can use a simple Linux or BSD LiveCD to perform the disk refresh.
  4. This is NOT a data recovery procedure. If you're doing data recovery, use something like dd_recover to a separate media.
  5. This is NOT a step by step tutorial. Read your OS manpages to make sure you're not wiping out the wrong disk or something.
  6. Always monitor S.M.A.R.T. parameters in order to spot disk failure before it happens.
  7. Always keep backups.

Links and resources:

Mounting Linux NFS shares in Solaris 10


First, export a filesystem on your Linux box (add it to the exports file, and exportfs). Make sure the kernel-nfs server service is started.
# /etc/init.d/nfs-kernel-server start
Edit your exports file:
# ed /etc/exports
i
/home/cmihai 192.168.1.13/24(rw,no_root_squash,subtree_check,async)
w
q
Export the filesystem:
# exportfs
/home/cmihai 192.168.1.13/24
# showmount -e
Export list for loonix:
/home/cmihai 192.168.1.13/24

On your Solaris 10 box, run:
# showmount -e loonix
export list for loonix:
/home/cmihai 192.168.1.13/24
It should give you results consistent with the results you got on your Linux machine.
Trying to mount the share as NFSv4 will fail (Linux NFSv4 isn't compatible with Solaris NFSv4).
# mount loonix:/home/cmihai /storage
nfs mount: mount: /storage: Not owner
So we're going to mount the share as NFSv3:
# mount -o vers=3 loonix:/home/cmihai /storage
# mount
/storage on loonix:/home/cmihai remote/read/write/setuid/devices/vers=3/xattr/dev=4840010 on Tue Mar 18 14:59:06 2008


Windows PE tools, LiveCD, Deployment Tools


Tools that help with Windows Deployment (Imaging, mass deployment, PXE, multicast, LiveCDs, bare metal recovery, disk cloning, "JumpStart" technology, automation tools, etc).

Cool Windows PE Tools:
  • Bart's Preinstalled Environment (BartPE) bootable live windows CD/DVD - Bart's PE Builder helps you build a "BartPE" (Bart Preinstalled Environment) bootable Windows CD-Rom or DVD from the original Windows XP or Windows Server 2003 installation/setup CD, very suitable for PC maintenance tasks.
  • UBCD4Win is a bootable recovery CD that contains software used for repairing, restoring, or diagnosing almost any computer problem.
  • WinBuilder - Build customized Windows boot disks like LiveXP, VIstaPE, etc.
  • Windows PE 2.0 is the core deployment foundation for Windows Vista.
  • WAIK - Windows Automated Installation Kit - Customize and deploy Vista. Here's a guide.

Unattended tools, deployment tools:

  • nLite - Deployment Tool for the Bootable Unattended Windows
  • vLite - Deployment Tool for the Bootable Unattended Windows Vista. Remove Windows Vista components, integrate drivers, tweak, bootable ISO...
  • Windows PE is a tool based on Microsoft Windows XP Professional that allows IT staff to build custom solutions, and can run Windows setup, scripts, etc. Provides automation tools, deployment tools, etc.
  • WAIK - Microsoft Windows Automated Installation Kit (WAIK) lets you install, customize, and deploy the Microsoft Windows Vista. It lets you perform unattended installations of Windows, create Windows Images with ImageX and create Windows PE images.
  • Sysprep is a tool designed for corporate system administrators, OEMs, and others who need to deploy the Windows operating system on multiple computers.
  • NewSID from Sysinternals changes the computer's Security Identifier to help with disk image cloning to perform mass roullouts of Windows. Alternative to using Sysprep.
  • drbl-winRoll is a tool that enables MS-Windows client to accept commands from DRBL server, and it resolve windows hostname, SID duplication problem via using image clone tool at single local LAN.It can also change "workgroup" name via IP/netmask automatically.

Mass deployment, PXE:
  • Remote Installation Service (RIS) in Windows Server allows deployment of Windows images via PXE boot.
  • Windows Deployment Services (WDS). WDS is the next version of Microsoft Remote Installation Services (RIS). Windows Deployment Services is included in the Windows Automated Installation Kit (Windows AIK) and in Windows Server 2003 SP2. It supports advanced features such as customized images and has Vista support.
  • Unattended is an open source Windows 2000, XP and 2003 automation tool (similar to RIS) that also works on Linux and such.
  • Symantec Ghost Solution Suite provides a corporate imaging, deployment and system management solution.
  • Clonezilla is a free partition or disk clone software similar to Ghost.
  • Acronis Snap Deploy - Complete PC system deployment solution using disk imaging technology
  • RIS For Linux - deploy Windows from a *NIX system via PXE boot.

Other tools:
  • WUD - Windows Updates Downloader

A brief look at ClamAV security - Open Source Antivirus


A brief (over)look at ClamAV security and performance. Comparing Open Source Antivirus products with commercial products.

I've been looking into signature based open source security products lately, namely Antivirus software (for SMTP E-Mail gateways or File Servers or anything that would be used as a distribution point for example as well as for Desktop systems).

I've had a pretty good look at ClamAV and other ClamAV based products (ClamWin or Spyware Terminator which include the ClamAV Engine) and found them rather weak, both from a security point of view (vulnerabilities) and in terms of detection rates, as well as performance (speed) and usability(interface, features, etc).

Clam AntiVirus (ClamAV) is an open source (GPL) anti-virus toolkit for UNIX (also ported to Windows, and used by GUI products such as ClamWin) designed especially for e-mail scanning on mail gateways. It provides a number of utilities including a flexible and scalable multi-threaded daemon, a command line scanner and advanced tool for automatic database updates. The core of the package is an anti-virus engine available in a form of shared library.


I. Security track record - a look at common vulnerabilities

So let's take a look at ClamAV's security track record. A simple look on Secunia at the vulnerability summary reveals a stunning 25 security advisories (1 unpattched), 31 % of which lead to System Access and 91 % are exploitable from remote. 40 % of vulnerabilities are "Highly Critical". I realize that some of these are in 3rd party plugins and compression tools and such, but when an attacker just sends a specially crafted archive via E-mail or whatever means, and manages to buffer overflow and gain system access when ClamAV scans it, that's when you need to look at other products. There are ways to mitigate it, with permissions, limited users, chroots, jails and such, but still...




So let's compare that with another security product, Avira Antivir, basically multi-platform Antivirus software (also has a free version for non-commercial usage). We can see only 2 security vulnerabilities reported, both local and privilege escalation (Windows only too).



Vulnerabilities range from Denial of Service: "A NULL-pointer dereference error exists within the "cli_scanrtf()" function in libclamav/rtf.c. This can potentially be exploited to crash ClamAV via a specially crafted RTF file." to Buffer Overflow and System Access: "An integer overflow error in rebuildpe.c when rebuilding PE files after unpacking can be exploited to cause a heap-based buffer overflow via a specially crafted executable. Successful exploitation of this vulnerability may allow execution of arbitrary code."


II. Vulnerability assessment tools - Static Code Analsys


What about a source code audit of ClamAV? Well, I don't really have the time for that, but I did parse it through FlawfinderRATS and other Static Code Analysis tools looking for simple lexical "bad practices" and functions (strings functions for example) that may overflow buffers and so on.

So, I download the source code for the latest stable release: ClamAV 0.91.2 (signature) and stumble across a ton of bad programming practices. While most of the time, they mean nothing (as they aren't really vulnerabilities or even exploitable, they are usually where most errors occur, and, as such, should be avoided). We basically have hundreds of such occurrences (537 marked as High and 83 marked as Medium by RATS), so I'm just going to paste a few interesting examples here:

..\clamav-0.91.2/clamav-milter/clamav-milter.c:266: High: fixed size local buffer
Extra care should be taken to ensure that character arrays that are allocated on the stack are used safely. They are prime targets for buffer overflow attacks.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:859: High: getopt_long
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/shared/getopt.c:961: High: fprintf
Check to be sure that the non-constant format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/sigtool/vba.c:1127: High: sprintf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1205: High: popen
Argument 1 to this function call should be checked to ensure that it does not come from an untrusted source without first verifying that it contains nothing dangerous.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1404: High: getenv
Environment variables are highly untrustable input. They may be of any length, and contain any data. Do not make any assumptions regarding content or length. If at all possible avoid using them, and if it is necessary, sanitize them and truncate them to a reasonable length.

..\clamav-0.91.2/shared/output.c:159: High: umask
umask() can easily be used to create files with unsafe priviledges. It should be set to restrictive values.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1585: High: gethostbyname
DNS results can easily be forged by an attacker (or arbitrarily set to large values, etc), and should not be trusted.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:1703: High: strcpy
Check to be sure that argument 2 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.


..\clamav-0.91.2/shared/misc.c:132: High: printf
Check to be sure that the non-constant format string passed as argument 1 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/shared/output.c:235: High: syslog
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/freshclam/manager.c:1307: High: system
Argument 1 to this function call should be checked to ensure that it does not come from an untrusted source without first verifying that it contains nothing dangerous.

..\clamav-0.91.2/sigtool/sigtool.c:815: High: strcat
Check to be sure that argument 2 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.

..\clamav-0.91.2/libclamav/hashtab.c:408: High: sscanf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/shared/options.c:194: High: strncat
Consider using strlcat() instead.

..\clamav-0.91.2/shared/options.c:194: High: strncat
Check to be sure that argument 1 passed to this function call will not copy more data than can be handled, resulting in a buffer overflow.

..\clamav-0.91.2/shared/getopt.c:983: High: getopt
Truncate all input strings to a reasonable length before passing them to this function

..\clamav-0.91.2/shared/output.c:212: High: vfprintf
Check to be sure that the non-constant format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle.

..\clamav-0.91.2/sigtool/sigtool.c:609: High: scanf
Check to be sure that the format string passed as argument 2 to this function call does not come from an untrusted source that could have added formatting characters that the code is not prepared to handle. Additionally, the format string could contain `%s' without precision that could result in a buffer overflow.

..\clamav-0.91.2/test/pe/debugpe.c:165: Medium: signal
When setting signal handlers, do not use the same function to handle multiple signals. There exists the possibility a race condition will result if 2 or more different signals are sent to the process at nearly the same time. Also, when writing signal handlers, it is best to do as little as possible in them. The best strategy is to use the signal handler to set a flag, that another part of the program tests and performs the appropriate action(s) when it is set.
See also: http://razor.bindview.com/publish/papers/signals.txt

..\clamav-0.91.2/libclamav/mbox.c:4659: Medium: getc
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/sigtool/sigtool.c:172: Medium: read
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/clamav-milter/clamav-milter.c:4260: Medium: stat
A potential TOCTOU (Time Of Check, Time Of Use) vulnerability exists. This is the first line where a check has occured. The following line(s) contain uses that may match up with this check: 4269 (open)

..\clamav-0.91.2/sigtool/vba.c:1063: Medium: realloc
Don't use on memory intended to be secure, because the old structure will not be zeroed out.

..\clamav-0.91.2/libclamav/lockdb.c:246: Medium: SetSecurityDescriptorDacl
If the third argument, pDacl, is NULL there is no protection from attack. As an example, an attacker could set a Deny All to Everyone ACE on such an object.

..\clamav-0.91.2/libclamav/msexpand.c:130: Medium: fgetc
Check buffer boundaries if calling this function in a loop and make sure you are not in danger of writing past the allocated space.

..\clamav-0.91.2/libclamav/others.c:433: Medium: srand
Standard random number generators should not be used to generate randomness used for security reasons. For security sensitive randomness a crytographic randomness generator that provides sufficient entropy should be used.

..\clamav-0.91.2/libclamav/others.c:697: Medium: lstat
A potential TOCTOU (Time Of Check, Time Of Use) vulnerability exists. This is
the first line where a check has occured.
The following line(s) contain uses that may match up with this check:
699 (rmdir), 715 (unlink)

..\clamav-0.91.2/contrib/Windows/Projects/clamAV/libclamav/regex.c:70: Medium: realloc
Don't use on memory intended to be secure, because the old structure will not be zeroed out.


You should keep in mind that ClamAV also relies on 3rd party libraries and tools, and it's security also depends on those. Again, I remind you that these aren't actual vulnerabilities, just bad practices that MAY lead to such vulnerabilities. You would need to look at the code and employ various testing tools to find them.

The authors of ClamAV should really solve these problems like OpenBSD developers do, even if it is something as simple as replacing strncat() withstrlcat() - functions designed to be safer, more consistent, and less error-prone (this would be an issue on other platforms though, so it's not as simple as that).


III. Detection rates 
So, here are some tests made by various research projects:
AV-Test is an anti-virus research project at the Institute of Technical and Business Information Systems at the Otto-von-Guericke University Magdeburg (Germany).
They measured the detection times for six of the malware programs released last week utilizing the MS05-039 Plug and Play vulnerability under 36 different anti-virus products. Eleven of the products were able to detect one or more of the attacks proactively, without any special pattern update to identify it specifically. Here are the numbers for each of the eleven:
Product Score
BitDefender 6 of 6 Fortinet 6 of 6 Nod32 5 of 6 eSafe 3 of 6 F-Prot 3 of 6 Panda 3 of 6 QuickHeal 3 of 6 McAfee 2 of 6 Norman 2 of 6 AntiVir 1 of 6ClamAV 1 of 6

AV Comparatives did a test on various other Antivirus products not in their current testing process, including the ClamAV based ClamWin. Here are the results:
http://www.av-comparatives.org/seiten/ergebnisse/2ndgrouptest.pdf

This test also places ClamAV among the last in terms of detection rates:
http://www.sunbelt-software.com/ihs/alex/malwarereportjun3007.pdf

ClamAV also scores a 17 % (sig: 99% / heur: 1%) in retrospective Antivirus Performance Statistics, which also placed it among the last.

At VirusPool Tested Products ClamAV scores:

Number of descriptions in the database: 31928 out of 45159 live samples ( 70.7 %)
Number of 'in the wild' descriptions in the database: 25 out of 30 live samples ( 83.3 %)
(not very good, but not all that bad)

IV. Signature Updates:


How often is the clamav-virusdb updates? According to the FAQ, multiple times a week, and the response should be rather prompt. Anyway, to get an idea on that, take a look at http://lurker.clamav.net/list/clamav-virusdb.html

Conclusion:

I definitely need to investigate this further, but so far I find ClamAV to be highly overrated, simply based on the fact it is "part of the open source movement". It lacks a real time scanner (that's fair seeing how it was designed for mail gateways though), it has a horrible security track recordpoor detection rates and dreadful performance. I somewhat doubt that it would last long as a commercial product. Still, it is available on multiple platforms, and the cost is just about right :-).

Things aren't all bad though (and I may have been to rash and only brushed the surface here). ClamAV is a free product (open source even), and it does leave a lot of room an opportunity to evolve. I just don't see this happening without powerful commercial backing. If you plan on using it though, make sure you've got at least another product scanning your emails :-).

Other links:
http://www.av-comparatives.org/
http://www.virustotal.com/

Memory Scanner and Editor for Linux


If you're familiar with ArtMoney, MemEdit or Cheat Engine on Windows, then you know what it's used for :-).

Basically, it lets you search for variables in the memory of a running process, then refine the search further (eg: give higher, lower or fixed values). You can use this basically to cheat at games. Look for the health, money or mana values, refine the search, then freeze or set the value to whatever you want.

ScanMem is a simple commandline tool on Linux that lets you do the same. There are other uses, but this one is the most common...