Troubleshooting and System Maintenance
RHCTs should be able to:
1) boot systems into different run levels for troubleshooting and system maintenance
boot single to set root password, or fix mounting, or init problem
Q: Machine won't boot
A: modify /boot/grub/grub.conf
root (hd0,0)
A: Check files
/etc/rc.local
/etc/inittab
A: Fix initrd
mkinitrd initrd-`uname -r`.img `uname -r`
Q: root password not known
A: 1) boot single (at grub screen, "a", "space", "1", enter),
2) passwd root
Items to study: grub, initrd,vmlinuz, inittab, rc.sysinit...
2) diagnose and correct misconfigured networking
system-config-network, or
vi /etc/sysconfig/network-scripts/ifcfg-eth0
check IPADDR, NETMASK, GATEWAY
ONBOOT=yes
PEERDNS=no
vi /etc/sysconfig/network
3) diagnose and correct hostname resolution problems
/etc/resolv.conf
/etc/hosts
4) configure the X Window System and a desktop environment
Remember X troubleshooting should be done from the command prompt, and
not within X.
Files:
/etc/X11/xinit/xinitrc.d
/etc/X11/xorg.conf
Q: Boot to X
A: Set runlevel 5 in /etc/inittab
vi /etc/inittab
change
id:3:initdefault:
to
id:5:initdefault:
Q: X won't load
A: Check config files
A: system-config-display --reconfig
A: /tmp or /home is full
5) add new partitions, filesystems, and swap to existing systems
This one is a little hard to explain. It just takes some practice to understand.
Tools available:
fdisk /dev/hda
t / fd (raid)
partprobe
mdadm -C /dev/md0 -l 5 -n 3 /dev/hda6 /dev/hda7 /dev/hda8
mke2fs -j /dev/md0
Q: mount something, such as: /dev/hda7 on /data
A: make directory, mount, modify /etc/fstab
1) mkdir /data
2) mount -t ext3 /dev/hda7 /data
3) vi /etc/fstab
/data /data ext3 defaults 1 2
view or change label:
e2label /dev/hda2
e2label /dev/vg0/data0 /data
6) use standard command-line tools to analyze problems and configure system
ls, cp, mv, rm, tail, cat, etc
system-config-
Installation and Configuration
RHCTs must be able to:
1) perform network OS installation
Enter the nfs, or ftp server address and path
2) implement a custom partitioning scheme
Probably easiest during GUI install
3) configure printing
probably the easiest thing to do is use:
system-config-printer
/etc/cups/cupsd.conf
lpadmin
lpstat, lpq
reject disable
accept /usr/bin/enable
4) configure the scheduling of tasks using cron and at
Understand how cron fields work:
min hr month_day month weekday
copy script to /etc/cron.daily, /etc.cron.monthly, etc. or
Modify /etc/crontab: crontab -e
List cron jobs: crontab -l
Add an at job: at [-f file] TIME
View at queue: atq
Remove at jog: atrm job
5) attach system to a network directory service, such as NIS or LDAP
1) Install ypbind, portmap
2) system-config-authentication
check: Enable NIS Support, or Enable LDAP Support
click: Configure NIS..., or Configure LDAP...
3) configure firewall, if required
iptables -A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 835:837 -j ACCEPT
service iptables save
test:
ypwhich
nisdomainname -y
genent services
getent passwd smith
rpcinfo -p
files:
/etc/yp.conf
/etc/sysconfig/network
/etc/pam.d/system-auth
/etc/nsswitch.conf
6) configure autofs
/etc/auto.master
/home/guests /etc/auto.guests --timeout=60
/etc/auto.guests
* -rw,soft,intr server1:/home/guests/&
service autofs restart
chkconfig portmap on
chkconfig autofs on
7) add and manage users, groups, quotas, and File Access Control Lists
Q: Create group, add users, set primary group
A: Use groupadd, useradd, usermod
groupadd -g 1000 groupname
useradd username
usermod -G groupname username
(-G for primary group)
quotas
Q: Setup a 70kb hard block user quota for user tim on /home/tim:
A: Use a Hard Block user quota (or Soft depending on question).
1) modify /etc/fstab
change defaults to usrquota for /home
2) mount -o remount /home
3) quotacheck -cuf /home
4) quotaon /home
5) setquota -u tim 0 60 0 0 /home
or
5a) edquota tim
Disk quotas for user tim (uid 502):
Filesystem blocks soft hard inodes soft hard
/dev/hda5 16 0 0 8 0 0
6) repquota /home
or
quota tim
7) test
a) quota tim
b) su - tim
c) dd if=/dev/zero of=/home/tim/somefile bs=1024 count=70
d) ls -l
file should be less than 70 kilobytes
Quota Command to remember:
quotacheck -cuf /home
quotaon /home
edquota tim
quota tim
repquota /home
*** Lab: Set a user soft block quota of 1MB for user diskhog on /blackhole,
(or hard block, hard / soft inode). Know when to use each type of quota ***
Enable quotas:
quotacheck -cuf /blackhole
quotaon /blackhole
Set Quota:
block inode
h s h s
setquota -u diskhog 0 1024 0 0 /blackhole
or
edquota -u diskhog
Report on Quota for user or directory:
quota -u diskhog
repquota /blackhole
Password
defaults:
/etc/login.defs
chage [options] username
password files:
/etc/passwd
/etc/shadow
/etc/group
8) configure filesystem permissions for collaboration
Q: Make user alex a member of sales with write permissions to /depts/sales
Q: Make user sales, hr, and web groups in /depts/
A:
mkdir -p /depts/{sales,hr,web}
for GROUP in sales hr web;do chgrp $GROUP /depts/$GROUP;done
chmod 770 /depts/*
chmod g+s /depts/*
Test permissions as alex:
su - alex
9) install and update packages using rpm
You may have to mount a directory over nfs, or install directly from http://
rpm -Uvh filenamex.i386.rpm
rpm --import /usr/share/rhn/RPM-GPG-KEY
rpm -K /tmp/rpmversion.i386.rpm
10) properly update the kernel package
Just remember to use rpm -i, so the old kernel will still be there.
1) mkdir /server1
2) mount server1:/var/ftp/pub /server1
3) cd /server1/Redhat/RPMS
4) rpm -ivh kernel-*
5) vi /boot/grub/grub.conf
default=0
11) configure the system to update/install packages from remote repositories using yum or pup
/etc/yum.conf
12) modify the system bootloader
/boot/grub/grub.conf
13) implement software RAID at install-time and run-time
fdisk /dev/hda
fd (raid)
partprobe
mdadm -C /dev/md0 -l 5 -n 3 /dev/hda6 /dev/hda7 /dev/hda8
mke2fs -j /dev/md0
mount
/etc/fstab
mdadm --detail /dev/md0
recover
mdadm /dev/md0 -a /dev/sda1
14) use /proc/sys and sysctl to modify and set kernel run-time parameters
Q: turn on ip forwardarding
A: 1) vi /etc/sysctl.conf
net.ipv4.ip_forward=1
2) sysctl -p
or
echo 1 > /proc/sys/net/ipv4/ip_forward
to view:
sysctl -a | grep ipv4
to find available options for ipv4:
ls /proc/sys/net/ipv4
15) use scripting to automate system maintenance tasks
Maybe put a script in /etc/cron.daily?
Login Shell Scripts
/etc/profile
/etc/profile.d/*.sh
~/.bash_profile
~/.bashrc
/etc/bashrc
RHCE skills
Troubleshooting and System Maintenance
RHCEs must demonstrate the RHCT skills listed above, and should be able to:
1) use the rescue environment provided by first installation CD
linux rescue
2) diagnose and correct boot failures arising from bootloader, module, and filesystem errors
init
/etc/rc.d/rc.sysinit
/etc/rc.d/rc adn /etc/rc.d/rc?.d
/etc/rc.d/rc.local
grub
/boot/grub/grub.conf
filesystem
/etc/fstab
fsck /dev/hda
3) diagnose and correct problems with network services (see Installation and Configuration below for a list of these services)
4) add, remove, and resize logical volumes
LVM -
Hint: use apropos quota, or just lvm to find commands.
Q: expand or shrink logical volume (RHCE)
A: Make sure there is enough space, and volume is online, use vgconvert if necessary
ext2online /dev/vg0/data0 120M
lvextend -L +100M /dev/vg0/data0
lvreduce -L 120M /dev/vg0/data0
vgdisplay -v vg0
pvdisplay
lvdisplay
Q: Create a logical volume
A: First create physical volume, then volume group, then logical volume
PV -> VG -> LV
pvcreate /dev/hda6 /dev/hda7 /dev/hda8
vgcreate vg0 /dev/hda6 /dev/hda7
lvcreate -L 50M -n data0 vg0
ext2online /dev/vg0/data0
lvextend -L +6M /dev/vg0/data0
resize2fs -L /dev/vg0/data0 40M
lvreduce -L 40M /dev/vg0/data0
vgconvert
5) diagnose and correct networking services problems where SELinux contexts are interfering with proper operation.
SELinux
getenforce
setenforce 1
check context with ls -Z
Q: Set up directory to use context of another directory
A: Use the other directory as a reference
chcon -R --reference /var/www/html /var/www/html/www1
Installation and Configuration
RHCEs must demonstrate the RHCT-level skills listed above, and they must be capable of configuring the following network services:
1) HTTP/HTTPS
install httpd, check context with ls -Z
Q: Create a virtual host www1.example.com w/ subdirectory /var/www/html/www1
A:
1) install httpd, modify /etc/httpd/conf/httpd.conf file
ServerName www1.example.com
DocumentRoot /var/www/html/www1
2) chcon -R --reference /var/www/html /var/www/html/www1
3) service httpd restart
4) chkconfig httpd on
Testing
service httpd configtest
2) SMB
Q: Configure Samba share /home/depts/legal
1) install samba
rpm -Uvh samba-* system-config-samba*(optional)
2) vi /etc/samba/smb.conf
[sharename]
path = /home/depts/legal
browseable = no
writeable = no
3) configure firewall
port 445, 137-139
3) NFS
Q: Export /data directory with nfs
The nfs server is an RPC service, and thus requires portmap
A: Modify /etc/exports, start nfs & portmap (make sure /data directory exists)
1) modify /etc/exports
/data *(sync,rw)
2) start services
service nfs start
service portmap start
chkconfig nfs on
chkconfig portmap on
3) verify config, & check mounts
exportfs -v
showmount -e localhost
rpcinfo -p localhost
4) restart
exportfs -r, or
service nfs reload
Q: Mount server1:/var/ftp/pub with nfs on /server1
1) mkdir /server1
2) vi /etc/fstab
192.168.2.254:/var/ftp/pub /server1 nfs soft,defaults 0 0
3) mount -a
* configure autofs
Automount
Q: Configure automount for nfs mount from nis domain on server1
A:
1) mkdir /net
2) vi /etc/auto.master
# uncomment /net line
/net /etc/auto.net
3) start services
service portmap start
service autofs start
chkconfig autofs on
chkconfig portmap on
Q: Configure automount for nfs mount of /rhome directories from nis domain on server1
A: Create base directory
1) mkdir /rhome
2) vi /etc/auto.master
# copy and modify /misc line
/rhome /etc/auto.rhome
3) vi /etc/auto.rhome
* -rw,soft.intr server1:/home/guests/&
4) start services
service portmap start
service autofs start
chkconfig autofs on
chkconfig portmap on
4) FTP
Q: Configure ftp with /var/ftp/incoming directory
A:
install vsftpd
1) rpm -Uvh vsftpd*
2) service vsftpd start
3) chkconfig vsftpd on
4) create incoming directory
cd /var/ftp
mkdir incoming
chown root:ftp fincoming
chmod 730 incoming
5) vi /etc/vsftpd/vsftpd.conf
uncomment the lines --
#anon_upload_enable=YES
#chown_uploads=YES
#chown_username=whoever
add
anon_umask=077
6) service vsftpd restart
7) configure firewall
iptables -A INPUT -s 192.168.0.0/24 -p tcp --dport 21 -j ACCEPT
service iptables save
5) Web proxy
Q: Install a web proxy and allow a certain network to access it
A: Install & configure squid
1) install squid, and start the service
rpm -ivh --aid squid*
service squid start
chkconfig squid on
2) vi /etc/squid
acl example src 192.168.0.0/24
http_access allow example
3) service squid reload
4) Configure browser to test:
Edit / Preferences /General / Connection Settings
Manual proxy configuration / HTTP Proxy: localhost Port: 3128
Enable port in firewall, if required
iptables -A INPUT -s 192.168.0.0/24 -p tcp --dport 3128 -j ACCEPT
service iptables save
Useful parameters:
http_port 3128
cache_mem 8 MB
cache_dir ufs /var/spool/squid 100 16 256
acl all src 0.0.0.0/0.0.0.0
http_access allow localhost
http_access deny all
hint: find out that port number is 3128
grep squid /etc/services
6) SMTP
install sendmail, sendmail-cf, sendmail-doc (optional)
Q: Configure mail server to accept internet email
A: modify /etc/mail/sendmail.mc
1) cd /etc/mail
2) vi /etc/mail/sendmail.mc
search for 127.0, put dnl at the front of the line
3) make
or m4 sendmail.mc > sendmail.cf
service sendmail restart
Q: Mail alias
A: modify /etc/aliases, run newaliases
Q: Receive mail for DomainX.example.com
A: modify sendmail mc as above, and add domain to /etc/mail/local-host-names
domainx.example.com
Debugging:
mail -v root
mailq, mailq -Ac
sendmail -q
tail -f /var/log/maillog
7) IMAP, IMAPS, and POP3
Q: Configure for pop3 (or imap)
A: 1) install dovecot
2) vi /etc/dovcot.conf
protocols = pop3
3) service dovecot restart
4) chkconfig dovecot on
Testing:
note: root is not permitted to login
echo "pop" | mail -s test student
telnet localhost 110
user student
pass student
stat
list
retr 1
quit
8) SSH
/etc/ssh/
~/.ssh/
9) DNS (caching name server, slave name server)
Q: Setup a slave name server
A:
1) install bind, bind-utils, and caching-nameserver
2) when configuring a slave name server, start with caching, and modify
3) vi /etc/named.conf
comment out dump-file section
add:
zone "example.com" {
type slave;
masters { 192.168.0.254 };
file "slaves/slave-example.com.zone";
};
4) vi 0.168.192.in-addr.arpa
add:
zone "0.168.192.in-addr.arpa" {
type slave;
masters { 192.168.0.254 };
file "slaves/0.168.192.zone";
};
To verify:
named-checkconf
named-checkconf -t /var/named
named-checkzone example.com example.com.zone
5) start named, and make it start at boot
service named restart; tail -f /var/log/messages | grep named
chkconfig named on
Remember to check /etc/resolv.conf on all client machines.
10) NTP
/etc/ntp.conf
For each of these services, RHCEs must be able to:
1) install the packages needed to provide the service
rpm -Uvh packagename.rpm, or
yum install packagename
2) configure SELinux to support the service
getenforce
setenforce 1
check context with ls -Z
chcon -R --reference /var/www/html /var/www/html/www1
3) configure the service to start when the system is booted
chkconfig servicename on
4) configure the service for basic operation
different for each service
5) Configure host-based and user-based security for the service
setuid?
/etc/xinet.d/
RHCEs must also be able to:
1) configure hands-free installation using Kickstart
2) implement logical volumes at install-time
3) use iptables to implement packet filtering and/or NAT
iptables is usually configured to be as restrictive as possible, but this
may be difficult to implement in an exam environment. Read the instructions
carefully, and try to find the best way to implement it.
--- commands ---
iptables -L
iptables -F
service iptables save
service iptables restart
chkconfig iptables on
--- starting config ---
iptables -P INPUT DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED, RELATED -j ACCEPT
--- sample file ---
cat /etc/sysconfig/iptables
*filter
:INPUT DROP [67:11217]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [58:6450]
-A INPUT -s 192.168.0.10 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.0.1 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 192.168.0.254 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.0.254 -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 445 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 21 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 110 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -p tcp -m tcp --dport 835:837 -j ACCEPT
COMMIT
4) use PAM to implement user-level restrictions
/lib/security/
/etc/pam.d/
