Move LVM Volume Group to another computer

To move a volume group form one system to another, perform the following steps:


Make sure that no users are accessing files on the active volumes in the volume group, then unmount the logical volumes.

Use the -a n argument of the vgchange command to mark the volume group as inactive, which prevents any further activity on the volume group.

Use the vgexport command to export the volume group. This prevents it from being accessed by the system from which you are removing it.

After you export the volume group, the physical volume will show up as being in an exported volume group when you execute the pvscan command.

When the disks are plugged into the new system, use the vgimport command to import the volume group, making it accessible to the new system.

Activate the volume group with the -a y argument of the vgchange command.

Mount the file system to make it available for use.

Team multiple network interface into single interface

Linux allows binding multiple network interfaces into a single channel/NIC using special kernel module called bonding.


After trying few hours i became successful in implementing bonding on RHEL 5.2. The steps that implemented bonding is given below.

Step #1: Create a bond0 configuration file

First, create bond0 config file in /etc/sysconfig/network-scripts/

# vi /etc/sysconfig/network-scripts/ifcfg-bond0Append following lines to it:DEVICE=bond0

IPADDR=192.168.1.20

NETWORK=192.168.1.0

NETMASK=255.255.255.0

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

Step #2: Modify eth0 and eth1 config files:

Open both configuration using vi text editor and make sure file read as follows for eth0 interface# vi /etc/sysconfig/network-scripts/ifcfg-eth0 Modify/append directive as follows:DEVICE=eth0

USERCTL=no

ONBOOT=yes

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

And do same for /etc/sysconfig/network-scripts/ifcfg-eth1

Step # 3: Load bond driver/module

Make sure bonding module is loaded when the channel-bonding interface (bond0) is brought up. One need to modify kernel modules configuration file:# vi /etc/modprobe.conf Append following two lines:alias bond0 bonding

options bond0 mode=balance-alb miimon=100

Save file and exit to shell prompt.

Step # 4: Test configuration

First, load the bonding module:

# modprobe bonding

Restart networking service in order to bring up bond0 interface

# service network restart

Verify everything is working:# less /proc/net/bonding/bond0Output:

Bonding Mode: load balancing (round-robin)

MII Status: up

MII Polling Interval (ms): 0

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth0

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:0c:29:c6:be:59

Slave Interface: eth1

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:0c:29:c6:be:63

List all interfaces:# ifconfig

I want separate 'document root' in Apache .

To set separate document root in httpd , we need to make certain changes in httpd.conf. Add following line in httpd.conf


Alias /personal/vishesh /var/personal_work_area/vishesh



DirectoryIndex index.php



After that create .htaccess file inside /var/personal_work_area/vishesh and put following entry in that

Options +FollowSymLinks

RewriteEngine On

RewriteBase /personal/vishesh/test

RewriteRule ^alice.html$ bob.html

alice.html and bob.html are html files

Using SSL Certificate in your web site

1) Generate a key:


$ openssl genrsa -out www.example.com-key 2048

Generating RSA private key, 2048 bit long modulus

2) Generate a Certificate Sigining Request (CSR):

$ openssl req -new -key www.example.com-key -out

www.example.com-csr

You are about to be asked to enter information that will be

incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished

Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [GB]:

State or Province Name (full name) [Berkshire]:Greater London

Locality Name (eg, city) [Newbury]:London

Organization Name (eg, company) [My Company Ltd]:Acme Websites

Ltd.

Organizational Unit Name (eg, section) []:

Common Name (eg, your name or your server's hostname)

[]:www.example.com

Email Address []:

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

3) Buy a certificate:

You can buy certificate from verisign, or thwate or such CA. What you need to do is goto website of these Certificate Authority and submit your csr file.

4) Setup an SSL Vhost:



ServerName "www.example.com"

SSLEngine on

SSLCertificateFile "/etc/httpd/conf/ssl/www.example.com-cert"

SSLCertificateKeyFile "/etc/httpd/conf/ssl/www.example.com-key"

...

Complete system backup using dump

Today i n installed a new linux server , and after setting my servers i take system backup on my dat by writing a shell script in follwoing way


root#vi sysbackup.sh

mt -f /dev/st0 rewind

dump -0uf /dev/nst0 /

dump -0uf /dev/nst0 /boot

dump -0uf /dev/nst0 /home

dump -0uf /dev/nst0 /opt

dump -0uf /dev/nst0 /tmp

dump -0uf /dev/nst0 /usr

dump -0uf /dev/nst0 /var

echo "Backup Completed!"
:wq

root# Now executing my script resulted in completed file system backup on tape.

Must configure ldap.conf

I setup a ldap server to authenticating linux clients. My setup was ok , and clients was authenticating from server properly. Today due to some reason my ldap server become down and i try to logon linux using local account. But there was long time taken by clients for login screen, i become fade up with that, i edited my /etc/nsswitch.conf file in single use mode, removed ldap from passwd and group section. But then i concluded that this occur occur because my ldap client configuration is not proper. I edited my /etc/ldap.conf and entered following entries in it.


base dc=abc,dc=del

uri ldaps:// s1.abc.del ldaps://s2.abc.del

ldap_version 3

timelimit 10

bind_timelimit 10

nss_initgroups_ignoreusers root,ldap,named,avahi,

haldaemon,dbus,radvd,tomcat,radiusd,news,mailman

ssl yes

pam_password md5

nss_base_passwd ou=People,dc=abc,dc=del

nss_base_group dc=abc,dc=del

use_sasl off

tls_checkpeer yes

TLS_CACERTFILE /etc/pki/tls/certs/ca-bundle.crt

bind_policy hard_open

idle_timelimit 3550

Now everything is fine.

How du and df is diffrent ?

df and du measure two different things....


du reports the space used by files and folders--even this is more than the file size. A few quick experiments on my system show that 4K is a minimum file size in terms of disk space.

df reports the space used by the file system. This includes the overhead for journals and inode tables and such.

Finding the deleted node.

Today i found that some of the files are missing from my server, then i tried debugfs

# debugfs /dev/hda2

debugfs: lsdel

OR

echo lsdel
debugfs /dev/hda1 > lsdel.out

debugfs has a stat command which prints details about an inode. Issue the command for each inode in your recovery list. For example, if you're interested in inode number 148003, try this:

debugfs: stat <148003>

If you have a lot of files to recover, you'll want to automate this. Assuming that your lsdel list of inodes to recover in is in lsdel.out, try this:

# cut -c1-6 lsdel.out
grep "[0-9]"
tr -d " " > inodes

This new file inodes contains just the numbers of the inodes to recover, one per line. We save it because it will very likely come in handy later on. Then you just say:

# sed 's/^.*$/stat <\0>/' inodes
debugfs /dev/hda1 > stats

and stats contains the output of all the stat commands.

If the file was no more than 12 blocks long, then the block numbers of all its data are stored in the inode: you can read them directly out of the stat output for the inode. Moreover, debugfs has a command which performs this task automatically. To take the example we had before, repeated here:

debugfs: stat <148003>

Inode: 148003 Type: regular Mode: 0644 Flags: 0x0 Version: 1

User: 503 Group: 100 Size: 6065

File ACL: 0 Directory ACL: 0

Links: 0 Blockcount: 12

Fragment: Address: 0 Number: 0 Size: 0

ctime: 0x31a9a574 -- Mon May 27 13:52:04 1996

atime: 0x31a21dd1 -- Tue May 21 20:47:29 1996

mtime: 0x313bf4d7 -- Tue Mar 5 08:01:27 1996

dtime: 0x31a9a574 -- Mon May 27 13:52:04 1996

BLOCKS:

594810 594811 594814 594815 594816 594817

TOTAL: 6

This file has six blocks. Since this is less than the limit of 12, we get debugfs to write the file into a new location, such as /mnt/recovered.000:

debugfs: dump <148003> /mnt/recovered.000

Of course, this can also be done with fsgrab; I'll present it here as an example of using it:

# fsgrab -c 2 -s 594810 /dev/hda1 > /mnt/recovered.000

# fsgrab -c 4 -s 594814 /dev/hda1 >> /mnt/recovered.000

With either debugfs or fsgrab, there will be some garbage at the end of /mnt/recovered.000, but that's fairly unimportant. If you want to get rid of it, the simplest method is to take the Size field from the inode, and plug it into the bs option in a dd command line:

# dd count=1 if=/mnt/recovered.000 of=/mnt/resized.000 bs=6065

Why ssh sometime become slow ?

Sometime you found that that ssh take more than 20 seconds before it ask for login.

The reason for the slow connections is that SSH is trying to use Kerberos, hangs

for about 10 seconds, then tries public key authentication, hangs for about

10 seconds, and then finally prompts for password. By setting the "GSSAPIAuthentication" option to false, either in /etc/ssh/ssh_config, or on the command line, everything works perfectly.

How to configure multiple vhosts with SSL ?

The problem is that in order to know from which virtual host to serve content a


webserver must inspect the "host" header. This is part of the http

request. However the SSL handshake takes place before any http request

is initiated. In order to complete the handshake the webserver needs

to know which SSL certificate to use. Since the websever can't yet

know which virtual host content is being requested from it uses the

certificate of the first host. It's really a limit of the protocol, not the server.

However the latest version of the HTTPS protocol includes SNI, which

permits a client to transmit to the host the name of the virtualhost

it wants to contact during the SSL handshake. So what you need to do

is make sure you have the very latest apache, compiled with the latest

openssl libraries, and use a recent webbrowser.

How to list active Apache modules ?

Sometime we stuck with some basic things in Apache. httpd command is very useful in this way.


To list active modules in Apache

root# httpd -t -D DUMP_MODULES

or

root# httpd -M

To list compiled in modules

root#httpd -l

To get virtual host lists

root#httpd -S

To test httpd configuration

root#httpd -t

Understanding vmstat output

As we know vmstat is very useful command to monitor system performance in linux. I issued this command on my RHEL Server, and output is as follows


root# vmstat

procs -------- --------memory------ ---swap--- --io-- ----system- ---cpu--------

r b swpd free buff cache si so bi bo in cs us sy id wa st

0 0 128 1217472 302800 2383476 0 0 17 14 10 15 0 0 100 0 0


What we can conclude from this output ?

Let us first understand, meaning of this result.

procs refer process. r for running and b for blocking.

swpd refer swapping free refer free memory, buff memory used in buffering and so for cache

swap section include swap in(si) and swap out (so)

io section include block in (bi) and block out(bo)

system section include interrupt(in) and context switch (cs)

cpu in important section, it include cpu usage by us(user) and sy(system) process,id (idle) percentage of cpu. wa refer time waiting for IO and st refer time stolen from virtual machine.

Linux Boot Process

As soon as we boot out linux system, after bios execution , Stage1 boot loading take place. In stage1 boot loading the code inside Master Boot Record of booting disk execute. MBR consist 512 bytes which include 446 bye booting instruction, 64 byte partition table and 2 byte magic number.


The 446 bye of mbr, load 2nd stage of boot loader. For example if you are using GRUB as boot loader, the GRUB is so big that it can't fit in 446 byte. So mbr load 2nd stage of boot loader.

What is 1.5 stage of boot loader ?

Unlike LILO GRUB can load a Linux kernel from an ext2 or ext3 file system. It does this by making the two-stage boot loader into a three-stage boot loader. Suppose your kernel is on partition formatted by ext2 or ext3, then GRUB first load e2fs_stage1_5 then proceed to stage 2.

With stage 2 loaded, GRUB can, upon request, display a list of available kernels, /boot/grub/grub.conf get diaplayed. You can select a kernel and insert kernel parameters if required.Here, you can use a command-line shell for greater manual control over the boot process.

After stage2 specify kernel and root filesystem, kernel uncompress and loaded after that root filesystem mounted. If your kernel executes and root file system (/) get mounted properly. Next stage will be of system initialization.

Remember that kernel panic error may panic you if your root filesystem not mounted properly.
init is the first user space process run after kernel loading. Setting for init process present in /etc/inittab, wrong configuration of /etc/inittab may also disturb you.

TCP Wrapper

TCP Wrapper add additional layer of protection for linux system. TCP Wrappers can be used to GRANT or DENY access to various network services on your machine to the outside network or other machines on the same network. It does this by using simple access list rules which are included in the two files /etc/hosts.allow and /etc/hosts.deny .


One must remember that hosts.allow takes precedence over hosts.deny. So for example if host A is allowed to ssh access your system using hosts.allow then hosts.deny entry doesn't affect any way. Also remember that by default all sort of incoming and outgoing is allowed if respective entries missing in both hosts.allow and hosts.deny.

Example of using TCP Wrapper

Suppose you want to allow SSH access to hosts in a particular domain say abc.com and deny access to all the others. Then edit hosts.allow and hosts.deny files in following ways

/etc/hosts.allow
sshd : .abc.com

/etc/hosts.deny
sshd : ALL
I will also discuss some complex examples of using tcp wrapper in coming days.

Linux Network Installation

As we know we can install linux from network, the steps for network installation are very simple. One need to specify 'linux askmethod' at installation prompt. By keeping installation tree on NFS, HTTP or FTP server on can install linux on network client computers.


But booting into linux require linux bootable dvd or cd. By PXE network installation there is no need to keep linux bootable cd or dvd. System get bootable through network card pxe boot loader.

For PXE Network installation, one need to create following setup

Configure the network (NFS, FTP, HTTP) server to export the installation tree.

Configure the files on the tftp server necessary for PXE booting.

Configure which hosts are allowed to boot from the PXE configuration.

Start the tftp service.

Configure DHCP.

After given steps boot the client, and start the installation

PXE Boot Configuration

system-config-netboot , command present graphical screen to setup PXE boot configuration.

Command that can be used on text terminal is pxeos

By setting up DHCP and installtion tree on NFS,FTP or HTTP one can install linux using PXE.

Following link can be used to get exact steps

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/sysadmin-guide/ch-pxe.html

Logging telnet sessions

We can use TCP Wrapper in logging telnet access of system. spawn is used to launches shell command as child process, this concept can be used to log telnet session. To achieve this just do following entries in /etc/hosts.allow on system hosting telnet server


in.telnetd : .abc.com : spawn /bin/echo `/bin/date` from %h>>/var/log/telnet.log : allow

Secure deletion of files

Suppose you want to delete a file in a way that it become impossible to recover that. Normally when we delete a file the data is not actually destroyed only the index listing where the file is stored is destroyed. There a number of recovery tools that try to rebuild the index and recover the data.


On a busy system freed space may get used reused in second in that case data recovery become impossible, but suppose you want to be sure that after deletion none of method get data back. The best way is destroy the media which holding data, but in case of large capacity this is not practical solution. shred utility try to achieve similar affect. So shred command help in securely deleting files.

Understanding subshell

To understand concept of subshell, let us take an example


suppose we execute following command

root#pwd

/root

root#cd /opt/backup;pwd

output will be

/opt/backup

root# pwd

/opt/backup

Now try to execute following command

root# (cd /opt/backup;pwd)

/opt/backup

But

root#pwd

/root

We can notice the diffrence , in () causes sub-shell to be invoked, in

which change of directory in executed, after executing pwd in subshell

the sub shell ceases to exist. back in current shell , so we are still

in /root

Migration old style slapd.conf to new slapd-config

Today i decided to migrate my openldap configuration from traditional slapd.conf file to new slapd-config structure.In new slapd-config structure it is possible to apply changes in running ldap server, i mean there is no need to restart ldap server for configuration changes. As traditional slapd.conf have following format


# global configuration directives



# backend definition

backend



# first database definition & config directives

database



# second database definition & config directives

database



New slapd-config structure create .ldif files that store configuration in ldap format.

My existing slapd.conf have following domain definition

----------------------------

access to * by none

Include ...

pidfile ...

argsfile ...

database bdb

suffix dc=vk,dc=com

rootdn cn=Manager,dc=vk,dc=com

rootpw secret

database config

rootpw config

index objectClass eq

-------------------------------

Note: config database added for migration purpose.

I migrated this configuration into new slapd.d (slapd-config). For this i taken following steps

Created slapd.d folder

root#mkdir /usr/local/etc/openldap/slapd.d

Apply slaptest command

root# slaptest -f /usr/local/etc/openldap/slapd.conf -F /usr/local/etc/openldap/slapd.d

After successful execution of command i noticed that one file and one folder created inside directory /usr/local/etc/openldap/slapd.d

folder name : cn=config and filename: cn=config.ldif

Inside cn=config folder a number of other files created.

Using syslog

We can configure our syslog configuration file for organized logging. system logger collects messages from programs and even from the kernel. These messages are tagged with a facility that identifies the broad category of the source, e.g., mail, kern (for kernel messages), or authpriv (for security and authorization messages). In addition, a priority specifies the importance (or severity) of each message. The lowest priorities are (in ascending order) debug, info, and notice; the highest priority is emerg, which is used when your disk drive is on fire. The complete set of facilities and priorities are described in syslog.conf(5) and syslog(3).


Messages can be directed to different log files, based on their facility and priority; this is controlled by the configuration file /etc/syslog.conf. The system logger conveniently records a timestamp and the machine name for each message.

Priority names in the configuration file normally mean the specified priority and all higher priorities. Therefore, info means all priorities except debug. To specify only a single priority (but not all higher priorities), add "=" before the priority name. The special priority none excludes facilities, as we show for /var/log/messages and /var/log/debug. The "*" character is used as a wildcard to select all facilities or priorities. See the syslog.conf(5) manpage for more details about this syntax.

local[0-7] facilities, reserved for arbitrary local uses, are sent to separate files. This provides a convenient mechanism for categorizing your own logging messages.

Logging remotely

Configure /etc/syslog.conf for remote logging, using the "@" syntax:

/etc/syslog.conf:

# Send all messages to remote system "loghost"

*.* @loghostOn loghost, tell syslogd to accept messages from the network by adding the -r option:

# syslogd -r ...

Disable console program access

The /etc/security/console.apps/ directory should contain one file per


application that wishes to allow access to console users. The filename

should be the same as the servicename. To disable console equivalent

access to programs like shutdown, reboot, and halt for regular users

on server.

[root@vishesh] /# rm -f /etc/security/console.apps/halt

[root@vishesh] /# rm -f /etc/security/console.apps/poweroff

[root@vishesh] /# rm -f /etc/security/console.apps/reboot

[root@vishesh] /# rm -f /etc/security/console.apps/shutdown

samba+ldap setup

After working around 2 years on samba+ldap setup, i can say it is stable and most useful , where we need linux based authentication server to authenticate windows users. Users, computers and group account get stored in ldap(openldap) format in samba+ldap setup, same as in windows Active Directory . My ideal setup that is functioning properly from last 2 years without any issues is as follows .


My smb.conf files content is as follows
--------------------------------------------
[global]

workgroup = test server string = test1 netbios name = test1

ldap passwd sync = yes security = user passdb backend = ldapsam:ldap://127.0.0.1 ldap suffix = dc=test,dc=com

ldap machine suffix = ou=Computers ldap user suffix = ou=People ldap group suffix = ou=Group ldap admin dn= "uid=root,ou=People,dc=test,dc=com"

domain master = yes domain logons = yes

logon path = add user script = /usr/sbin/smbldap-useradd "%u" add group script = /usr/sbin/smbldap-groupadd "%g" add machine script = /usr/sbin/smbldap-useradd -w "%u" delete user script = /usr/sbin/smbldap-userdel "%u" delete group script = /usr/sbin/smbldap-groupdel "%g"

local master = yes os level = 254 preferred master = yes wins support = yes [netlogon] comment = Network Logon Service path = /var/lib/samba/netlogon guest ok = yes writable = no share modes = no

------------------------------------------------------

My ldap server configuration is as follows

(content of slapd.conf file)

------------------------------------------------------

include /etc/openldap/schema/core.schema

include /etc/openldap/schema/cosine.schema

include /etc/openldap/schema/samba.schema

include /etc/openldap/schema/inetorgperson.schema

allow bind_v2

pidfile /var/run/openldap/slapd.pid

argsfile /var/run/openldap/slapd.args

access to *

by self write

by users read

by anonymous read

database bdb

suffix "dc=test,dc=com"

rootdn="cn=Manager,dc=test,dc=com"

rootpw {SSHA}oifg.ytugjhkk

directory /var/lib/ldap/test.com

index uidNumber,gidNumber

------------------------------------------------------

Note: Ensure that samba.schema file

present is /etc/openldap/schema directory.

If not present, search samba.schema file on system

and copy that file in /etc/openldap/schema

Download & install smbldap tool from following link.

http://tinyurl.com/344ypzg

Apache server-status

Today i decided to monitor my apache web server by server-status. For detail analysis i turned 'ExtendedStatus' flag on. And uncomment following lines




SetHandler server-status

Order deny,allow

Allow from all

Deny from none



After that i restarted apache. Accessing http://localhost/server-status show server status in detail. But Request Section attracted my attention. In request section other that GET method PROFOUND and OPTIONS are also mentioned. After investigation i conclude that PROFOUND in related with WebDAV and OPTIONS method represents a request for information about the communication options available on the request/response chain identified by the Request-URI.

While doing investigation through google i also noticed some concern that apache server-status shows thousand of ::1 OPTIONS * HTTP/1.0. But the some important thing i noticed that even though none of client accessing Apache serve, in server-status a number of PROFOUND and OPTIONS request are showing from random client.

super block recovery and fsck stages

As we know fsck is a great command to check and repair error on file system. Many times i found filesystem in panic and fsck make it operation-able. I used fsck many times but every time i ensured that filesystem on which i applying fsck be in unmount state.


But the most important thing i was interested is fsck stages. Whenever i issued fsck command it output as

Phase 1: Checking Inodes,blocks and sizes

Phase 2: Checking directory structure

Phase 3: Checking directory connectivity

Phase 4: Checking Reference count

Phase 5: Checking group summary information

fsck checks the integrity of several different features of the file system. Most important checking that fsck do is of super block. As we know super block is most important aspect of file system which stores summary information for the volume. super block is also most modified item in file system so chances of corruption of super block is always high.

Checks on the superblock include:

A check of the file system size, which obviously must be greater than the size computed from the number of blocks identified in the superblock

The total number of inodes, which must be less than the maximum number of inodes

A tally of reported free blocks and inodes

On number of occasion I found super block of my file system get corruption. Although its a very difficult for me to dictates reasons of super block corruption. But the better part is that a backup super block is always present in our file system. To know that where is alternate super block we can use dumpe2fs command as follwing root# dumpe2fs /dev/sda1

more Generally block number 32768 is backup super block, so to recover filesystem using backup super block fsck command can be used in following ways root# fsck -b 32768 /dev/sda1 Other than super block corruption other error can be easily fixed by using fsck in straight ways as following root# fsck -y /dev/sda1 (Here -y option save us from pressing y while asking for yes during recovery ) When inodes are examined by fsck, the process is sequential in nature and aims to identify inconsistencies in format and type, link count, duplicate blocks, bad block numbers, and inode size. Inodes should always be in one of three states: allocated (being used by a file), unallocated (not being used by a file), and partially allocated, meaning that during an allocation or unallocation procedure, data has been left behind that should have been deleted or completed. Alternatively, partial allocation could result from a physical hardware failure. In both of these cases, fsck will attempt to clear the inode. The link count is the number of directory entries that are linked to a particular inode. fsck always checks that the number of directory entries listed is correct, by examining the entire directory structure beginning with the root directory, and tallying the number of links for every inode. Clearly, the stored link count and the actual link count should agree, but the stored link count can occasionally be different than the actual link count. This could result from a disk not being synchronized before a shutdown, for example, and while changes to the file system have been saved, the link count has not been correctly updated. If the stored count is not zero, but the actual count is zero, then disconnected files are placed in the lost+found directory found in the top level of the file system concerned. In other cases, the actual count replaces the stored count

for finding alternate/backup superblocks the following command can also be used,

#mke2fs -n /dev/device
use -j switch also if it is an ext3 file system (mke2fs -n -j /dev/device)

Files with no inode, no owner and no group.

Yesterday i encountered a file system error on my server. Some of files in one of partition formatted with ext3 was showing ? in place of inode, owner and group position.


root# cd /data/Trash

root# ls -li

? ? ? ? ? abc.txt

I become confused. find command also report a number of files which has no owner

root# find /data/Trash -nouser

Investing through problem result in conclusion that there is file system error in that partition. I advised apply fsck command immediately. Applying fsck solved the problem but i am searching in what condition file exist without inode number.

root# fsck -y /dev/vg1/lv1

I also ensured that that file system get applied fsck on reboot

root# shutdown -Fr now

One interesting point i learned in between is that maximum 16 consequent times a file system can get mounted without applying fsck after that a warning come to fsck although this setting can be override with tune2fs command.

The best option to reduce file system error is to apply fsck at booting time. This can be easily done by making entry in /etc/fstab for example in my case the entry is

/dev/vg1/lv1 /data ext3 defaults 1 2

Here 1 says apply fsck and 2 says after applying fsck on root.