Tuesday, July 6, 2010

RHCE Notes 3

This is the place where i am keeping my study notes for my RedHat Enterprise Linux Certified Engineer

I will add more to this page as I go along....

RHEL 5 requires at least 192MB of RAM

scenarios - /etc/inittab missing, try passing init=/bin/sh as a kernel parameter to boot to a shell then you can recreate or restore the inittab file

you can pass kernel parameters at boot to enter various runlevels,
- for runlevel 1 pass the word single or emergency

.---------------- minute (0 - 59)
| .------------- hour (0 - 23)
| | .---------- day of month (1 - 31)
| | | .------- month (1 - 12) OR jan,feb,mar,apr ...
| | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
| | | | |
* * * * * command to be executed
*/5 * * * * /home/user/test.pl
run the script /home/user/test.pl every 5 minutes.

Managing daemons
ntsysv --level 35 - starts a GUI service config for runlevels 3 and 5

chkconfig --list all daemons
chkconfig --level sendmail on
chkconfig sendmail off

LABEL=/ / ext3 defaults 1 1
server1:/store /store nfs rw 0 0

fstab column meanings
- device name: can be the disks LABEL from e2label or UUID, device /dev/sdb1
- mount point: where to mount
- fs-type:
- options: defaults means ??? need to find out
- dump-freq: en/disable dump when dump is called
- pass-num: indicates the order in which the fsck utility will scan the partitions for errors when the computer powers on. 0 = none, 1 = first, 2 = next

Automounter autofs - mounts on demand
/etc/auto.net servername, lists nfs shares on host

lsattr , list file attributes
chattr -i , changed a file to immutable so even root cannot delete it
chattr -a , change a file to append only ie allowing users to append to a log file.

remount partition with option "acl"
getfacl prints file access control lists
gstfacl /home/john

setfacl prints file acl's similar to chown only you can have multiple users access, use -m to modify
setfacl -m user:john:r-x /home/john/file1

mask, ??? need to research this further

ls -Z, lists security context of files

output is Identity, role, domain

Identity: user_u (Generic user), root (root), system_u (system users)
Role: all files seem to be associated with the "object_r" role
Context domain ???


..print [free | Num | all]
rm 10 , remove partition 10

- If harddrive is new you need to make a disk label
mklabel ....msdos

Creating partitions
...start.. 0
...end .. 100MB or 50%

partprobe - make sure linux reads the new partition table

Creating a swap disk
mkswap /dev/sda2
swapon /dev/sda2, don't forget to add an entry to fstab


rpm -i install
-U upgrade
-F upgrades only existing packages
-e erase/remove package
-v verbose
-h hash on progress

You can use urls in rpm eg:
rpm -ivh ftp://ftp.blah/pub/blah.rpm

rpm --verify --file /bin/vi , Verify file has changed since install

yum whatprovides evince , can user wildcards
yum whatprovides /etc/fstab

need to insall rpm-build package before you can build
SRPMS Install source to /usr/src/redhat
..SOURCES contains orig source
..SPECS contains spec files
..BUILD source is unpacked here
..RPMS output of rpm after rpmbuild
..SRPMS output of srpm

rpmbuild -ba vsftp.spec , builds source and binaries
.. -bb builds just binaries

pass kernel parameters to start kickstart, if there are multiple network interfaces then add the command ksdevice=eth0 so that it doesn't prompt for input
boot: linux ks=cdrom:/ks.cfg
boot: linux ks=hd:sdb1:/ks.cfg
boot: linux ks=nfs:
boot: linux ks=http:

utmpdump /var/log/wtmp, checks recent logins


john:x:500:500:John Bencic:/home/john:/bin/bash

Username: username, . Usernames can include hyphens (-) or underscores (_). However, they should not start with a number or include uppercase letters.
Password:an x, (*) or password. An x points to /etc/shadow for the actual password. An asterisk means the account is disabled.
User ID: The unique numeric user ID (UID) for that user. By default, Red Hat starts user IDs at 500.
Group ID: The numeric group ID (GID) associated with that user.
User info: eg Full Name
Home Directory: By default, RHEL places new home directories in /home/username.
Login Shell: By default, RHEL assigns users to the bash shell.

/etc/shadow file

Username: username
Password: Encrypted password; requires an x in the second column of /etc/passwd
Password history: Date of the last password change, in number of days after January 1, 1970
mindays: Minimum number of days that you must keep a password (-m)
maxdays: Maximum number of days after which a password must be changed (-M)
warndays: Number of days before password expiration when a warning is given (-W)
inactive: Number of days after password expiration when an account is made inactive (-I)
disabled: Number of days after password expiration when an account is disabled (-E)

Adding users
useradd , Adds user
usermod , modifys user

-g: set the main login group
-G : supplementary groups -a appends to list
-e 2009-12-30 john : sets account expiration
-U: unlocks an account effectively removing the ! from the password

chage command changes user password expiration
userdel: removes user, use -r to delete the users home directory as well


to enable quotas you need to mount the disk with group and user quota's as follows
/dev/sdb1 /home ext3 defaults,usrquota,grpquota 1 2

quotacheck -cugvm /home
-c Performs a new scan.
-v Performs a verbose scan.
-u Scans for user quotas.
-g Scans for group quotas.
-m dont try Remount the scanned filesystem readonly.

quotacheck will create and aquota.user and aquota.group file to keep track of the quotas

edquota -u john , edit quota for user john

edquota -t sets the grace periods, default is 7 days for inodes and files sizes

repquota -a / , -a all users. report on quota usage

Creating a share area for users
chown nobody.staff /home/staff
chmod 2770 /home/staff , or g+s to set group id

Pluggable Authentication Modules (PAM)

Format or PAM file
module_type control_flag module_path [arguments]

Module Types:auth, account, password, session

- auth: Authentication management Establishes the identity of a user. decides whether to prompt for a username and/or a password.
- account: Account management, Allows or denies access according to the account policies. time, password expiration or a list of restricted users.
- password: Password management, Manages other password policies eg limit number of times a user can try to log.
- session: Session management, Applies settings for an application eg set default settings for a login console.

Control Flags: determines what PAM does if module succeeds or fails
- required: If the module works, the command proceeds. If it fails, PAM proceeds to the next command in the configuration file-but the command controlled by PAM will still fail.
- Requisite: Stops the process if the module fails.
- Sufficient: If the module works, the login or other authentication proceeds. No other commands need be processed.
- Optional: PAM ignores module success or failure.
- Include: Includes all module_type directives from the noted configuration file; for example, if the directive is password include system-auth, this includes all password directives from the PAM system-auth file.

to connect a client to an nis server
edit /etc/yp.conf
add line: domain nisdomain server

start the service on boot
chkconfig ybbind on

manage password with the yppasswd command

to connect a client to an ldap server

modify the /etc/ldap.conf file

host IP of the LDAP server.
base dc=example,dc=com: Sets the default base distinguished name
ssl start_tls: Required if you want Transport Layer Security (TLS) support
pam_password: Supports encryption schemes for passwords; options include crypt, nds (Novell Directory Services), and ad (Active Directory).

modify the following lines to the /etc/nsswitch.conf to use for authenticating to ldap and ins
hosts: files nisplus nis dns
passwd: files nis ldap
shadow: files nis ldap
group: files nis ldap


netstat -r
Kernel routing table
Destination Gateway Genmask Flags MSS Window irtt Iface * U 40 0 0 eth0 * UH 40 0 0 lo
Flag Description
G: The route uses a gateway.
U: The network adapter (Iface) is up.
H: Only a single host can be reached via this route.
D: This entry was created by an ICMP redirect message.
M: This entry was modified by an ICMP redirect message.

# arp
Address HWtype HWaddress Flags Mask Iface ether 52:A5:CB:54:52:A2 C eth0

arp -s bugsy 00:00:c0:cf:a1:33 , add a static arp entry for host bugsy
arp -d bugsy , remove entry from rap table


install the printing group
yum groupinstall printing
use the system-config-printing to configure

printer class's - group of similar printers, printing to a class prints to any printer in the class which is not busy at the time.

using the older printing commands
lpr: used to add print requests
lpq: list queued requests
lprm: Remove requests from queue
lpc: Line Print Control administer one or more print queues.

lpc status
lpr -Pprintname filename , prints to printer named printername

Areas which I need work are


Quiick SElinux notes for the impatient, read full document at

Selinux has 2 levels access control:
1) File context, Daemon can only access file with particular file context
2) Boolean Value: enable/disalbe a feature
for example: By default SElinux does not allow users to login and read their home directories, turn it on by "setsebool -P ftp_home_dir 1"

#==Confined and Unconfined Process
Confined process enter paritcular domain after started, only particular domain has access to particular TYPE files
SElinux has no effect for Unconfined Processes (apps doen's support SElinux)

$ ls -Z /usr/sbin/httpd
-rwxr-xr-x root root system_u:object_r:httpd_exec_t /usr/sbin/httpd #httpd is confined by default
$chcon -Rt unconfined_exec_t /usr/sbin/httpd #change httpd to unconfied_exec_t, it will enter unconfied domain, so it can access any file as long as OS level file permission allowed
$ restorecon -Rv /usr/sbin/httpd #restore default type

#== SELinux: File context
for example: system_u:object_r:httpd_sys_content_t :s0:c0
Not all systems will display s0:c0

# ls -aZ /var/www/html/
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t .
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t ..
# ls -aZd /home
drwxr-xr-x root root system_u:object_r:home_root_t /home
httpd_exec_t can access httpd_sys_content_t not home_root_t

#==SElinux managment
SELINUX=permissive #in /etc/selinux/config. if it changed from disabled . it needs reboot to lable files
getenforce or sestatus #get current status
setenforce 0 # set to permissive mode
setenforce 1 #set to enforce mode
getsebool -a #list booleans and its value , no desc
setsebool httpd_can_network_connect_db on #change current boolean
setsebool -P httpd_can_network_connect_db on #change permanent boolean with -P

- Temparary change context
chcon -R -t httpd_sys_content_t /web/ #change context type dir/file
# it will survive reboot, but not relabel. To relabel, touch /.autorelabel reboot

- Persistent Changes: semanage fcontext
/etc/selinux/targeted/contexts/files/file_contexts #saved to orginal context
/etc/selinux/targeted/contexts/files/ file_contexts.local #saved to new user context
semanage fcontext -a -t samba_share_t /etc/file1 #-a add new context, the file doesn't need to exist.
restorecon -Rv /etc/file1 #read the new customized context and apply it

- Restore default context
semanage fcontext -d /etc/file1 #remove context,the file doesn't need to exist
restorecon -RFv /etc/file1 #apply the change, -F is needed you to restore from customized to default.

/var/log/audit/audit.log #enable auditd daemon first
chkconfig --levels 345 setroubleshoot on #enable troubleshoot daemon
sealert -a /var/log/messages #analyse log
sealert -l \* #show all alert
grep "SELinux is preventing" /var/log/messages
grep "denied" /var/log/audit/audit.log
Port Numbers # services are allowed to run on some defined ports
/usr/sbin/semanage port -l grep http_port_t
ttp_port_t tcp 80, 443, 488, 8008, 8009, 8443
semanage port -a -t http_port_t -p tcp 9876 #add the new port to allowed range

#==== document
selinux-policy-2.4.6-137.el5#man pages for ftpd_selinux, samba_selinux ...etc

RHCE Notes - Troubleshooting booting issue

booting issue is optional question in section I,The proctor will re-image your PC to introduce booting issue, You will be given rescue CD to fix it.

It is easy to troubleshoot Linux boot issue, if you break it intentionally at each step, observe the symptom and find the fix.

#==Linux boot order
The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

#==Linux rescue env
boot first linux cd then type linux rescue
linux rescue will try to mount all partions, however if there is error only some partions are mounted, run choot /mnt/sysimage now will lost /dev /proc mounts, here is how to transfer these mounts.
mount -o bind /dev /mnt/sysimage/dev
mount -o bind /proc /mnt/sysimage/proc

Linux rescue env supports both software RAID and LVM. normal LVM commands e.g vgdisplay are not availiable,but it can be accssed by LVM "master" command e.g "lvm vgdisplay"

#== Grub boot manager
= go to grub cmd prompt by pressing c at boot menu
=find root partition, 2 methords
grub> root
(hd0,0) Filesystem type is ext2fs, partition type 0x83
grub> find /grub/stage1
=list files/dirs in current drive
cat / #type cat SPACE / TAB, it will list all fies/dir just like ls
= display contents of the file
cat /grub/grub.conf
= now you can boot interactively by type kernel and initrd commands from grub.conf

#==Restore missed file from RPM
#cd /tmp
#rpm2cpio initscripts-7.93.11.EL-1.i386.rpm cpio -icumvd ./etc/inittab
#rpm2cpio initscripts-7.93.11.EL-1.i386.rpm >init.cpio /* file is ./etc/inittab not /etc/initab

List contents: cpio -tv

install file to alternative location the copy the file
rpm --root-directory /var/tmp/a X.rpm

#== MBR corrupted.
MBR has 512 byte in total
446 Executable code section
4 Optional Disk signature
2 Usually nulls
64 Partition table #if this is overwritten, no way to recover unless you backuped the partion table or re-partion using #exact same layout
2 MBR signature

Corrupt MBR intentionaly:dd if=/dev/zero of=/dev/hda bs=446 count=1 #MBR should be at the start whole disk(not partition hda1), it has 512, the first 446 byte is exec code. DON'T overwrite whole 512 byte because it has partion table data.
ERR: no bootable media found,Missing operating system" or "Operating System Not Found
boot from cd run "linux rescue", let it mount linux partions automaticlly.
chroot /mnt/sysimage then grub-install /dev/hda
boot from cd run "linux rescue", if linux partions failed to mout
mount mannually. sfdisk -l; e2label find the boot partition
mkdir /a; mount /dev/hda1 /a; ln -s /usr/sbin/grub /sbin/grub; grub-install --root-direcotry=/a /dev/hda #it is hda not hda1

#= root (/)was not mounted
mount couldn't find file system /dev/root
switchroot mount faild...
Error 2 mounting none;exec of init ((null)) failed!!!
kernel /vmlinuz-test ro root=LABEL=/
/* root=LABEL=/ mout using label, or root=/dev/sda3 mount with direct dev-name */

#= not loading initrd image
VFS: Cannot open root device "Label=/1" or unknow-block(0,0)
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount roof fs on unknow-block(0,0)
1) Kernel doesn't Support for the file system .compile kernel with FS support NOT as a module
2) initrd was not loaded. Add initrd=... in grub.conf
linux rescue, then chroot /mnt/syimage and create initrd file
mkinitrd /boot/initrd-filename `uname -r` #make initrd file mannually

#==/sbin/init problem.
Switching to new root
kernel panic -not syncing :Attepmted to kill init
switching to new root
/bin/sh: ro : no such file or directory
/* boot to rescue, check /sbin/init. restore from rpm package*/

#== /etc/inittab not found
"enter run level" prompt enter s. or at grub menu append s or init=/bin/sh or emergency, then restore initab from source RPM.

RHCE TIPS - Preparation
Reference book:

RHCE Red Hat Certified Engineer Linux Study Guide (Exam RH302) 5th edition by Michael Jang.
if something is not clear in the book, read official
Red Hat Enterprise Linux Documentation

Lab Setup:
CentOS on Virutalbox

Virtualbox is free opensource virtualization software alternative to Vmware.You need 2 CentOS instances to prepare for RHCE lab, The networking in Virtualbox is very different to Vmware.

-Virtualbox Networking Type:
--NAT: your guest OS can access outside network through NAT provided by virtualbox, but your host OS can’t access guest OS
--Host interface networking: Host and guest can communicate each other, but guest can’t access outside network unless you setup NAT manually on Host OS
--Internal network: Guest OS can communicate with each other within the SAME network name (something ike VLAN ID), but not Host OS.

-Centos ServerA network setup
1*NAT adapter for internet access to do yum.
1*Host network adapter for your host to ssh to ServerA
1*Internal Network adapter to communicate with ServerB

-Centos ServerB network setup
1* Internal Network adapter to communicate with ServerA (join the SAME network name of ServerA )

How can ServerB access outside network? Point the default GW to serverA, and turn on ip forwarding on ServerA.
How can my Host OS access ServerB?
1. ssh to serverA first then jump from serverA to ServerB
2. -setup porforwarding or 1 to 1 static mapping in ServerA
--Forwarding port 200 to ssh of ServerB
iptables -t nat -A PREROUTING -p tcp -d ServerA-Host-NIC-IP --dport 200 -j DNAT --to-destination ServerB-IP:22 
--Static 1 to 1 mapping
Assign secondary ip to serverA’s host Inc then
iptables -t nat -A PREROUTING -p tcp -d ServerA-SEC-NIC-IP -j DNAT --to-destination ServerB-IP
Last but least, read through each chapter and practice it LAB, you never know if it works until you really do it! RHCE exam is all about security, hence I suggest jumping to security chapter before reading networking services. Then apply your security knowledge (pam/tcp-wrapper/iptables/selinux) to each network services read later.