Friday, January 27, 2012

HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples


Managing log files effectively is an essential task for Linux sysadmin.
In this article, let us discuss how to perform following log file operations using UNIXlogrotate utility.
  • Rotate the log file when file size reaches a specific size
  • Continue to write the log information to the newly created file after rotating the old log file
  • Compress the rotated log files
  • Specify compression option for the rotated log files
  • Rotate the old log files with the date in the filename
  • Execute custom shell scripts immediately after log rotation
  • Remove older rotated log files

1. Logrotate Configuration files

Following are the key files that you should be aware of for logrotate to work properly.
/usr/sbin/logrotate – The logrotate command itself.
/etc/cron.daily/logrotate – This shell script executes the logrotate command everyday.
$ cat /etc/cron.daily/logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
/etc/logrotate.conf – Log rotation configuration for all the log files are specified in this file.
$ cat /etc/logrotate.conf
weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}
/etc/logrotate.d – When individual packages are installed on the system, they drop the log rotation configuration information in this directory. For example, yum log rotate configuration information is shown below.
$ cat /etc/logrotate.d/yum
/var/log/yum.log {
    missingok
    notifempty
    size 30k
    yearly
    create 0600 root root
}

2. Logrotate size option: Rotate the log file when file size reaches a specific limit

If you want to rotate a log file (for example, /tmp/output.log) for every 1KB, create the logrotate.conf as shown below.
$ cat logrotate.conf
/tmp/output.log {
        size 1k
        create 700 bala bala
        rotate 4
}
This logrotate configuration has following three options:
  • size 1k – logrotate runs only if the filesize is equal to (or greater than) this size.
  • create – rotate the original file and create the new file with specified permission, user and group.
  • rotate – limits the number of log file rotation. So, this would keep only the recent 4 rotated log files.
Before the logrotation, following is the size of the output.log:
$ ls -l /tmp/output.log
-rw-r--r-- 1 bala bala 25868 2010-06-09 21:19 /tmp/output.log
Now, run the logrotate command as shown below. Option -s specifies the filename to write the logrotate status.
$ logrotate -s /var/log/logstatus logrotate.conf
Note : whenever you need of log rotation for some files, prepare the logrotate configuration and run the logroate command manually.
After the logrotation, following is the size of the output.log:
$ ls -l /tmp/output*
-rw-r--r--  1 bala bala 25868 2010-06-09 21:20 output.log.1
-rwx------ 1 bala bala        0 2010-06-09 21:20 output.log
Eventually this will keep following setup of rotated log files.
  • output.log.4.
  • output.log.3
  • output.log.2
  • output.log.1
  • output.log
Please remember that after the log rotation, the log file corresponds to the service would still point to rotated file (output.log.1) and keeps on writing in it. You can use the above method, if you want to rotate the apache access_log or error_log every 5 MB.
Ideally, you should modify the /etc/logrotate.conf to specify the logrotate information for a specific log file.

3. Logrotate copytruncate option: Continue to write the log information in the newly created file after rotating the old log file.

$ cat logrotate.conf
/tmp/output.log {
         size 1k
         copytruncate
         rotate 4
}
copytruncate instruct logrotate to creates the copy of the original file (i.e rotate the original log file) and truncates the original file to zero byte size. This helps the respective service that belongs to that log file can write to the proper file.

4. Logrotate compress option: Compress the rotated log files

If you use the compress option as shown below, the rotated files will be compressed with gzip utility.
$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create 700 bala bala
        rotate 4
        compress
}
Output of compressed log file:
$ ls /tmp/output*
output.log.1.gz output.log

5. Logrotate dateext option: Rotate the old log file with date in the log filename

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create 700 bala bala
        dateext
        rotate 4
        compress
}
After the above configuration, you’ll notice the date in the rotated log file as shown below.
$ ls -lrt /tmp/output*
-rw-r--r--  1 bala bala 8980 2010-06-09 22:10 output.log-20100609.gz
-rwxrwxrwx 1 bala bala     0 2010-06-09 22:11 output.log
This would work only once in a day. Because when it tries to rotate next time on the same day, earlier rotated file will be having the same filename. So, the logrotate wont be successful after the first run on the same day.

6. Logrotate monthly, daily, weekly option: Rotate the log file weekly/daily/monthly

For doing the rotation monthly once,
$ cat logrotate.conf
/tmp/output.log {
        monthly
        copytruncate
        rotate 4
        compress
}
Add the weekly keyword as shown below for weekly log rotation.
$ cat logrotate.conf
/tmp/output.log {
        weekly
        copytruncate
        rotate 4
        compress
}
Add the daily keyword as shown below for every day log rotation. You can also rotate logs hourly.
$ cat logrotate.conf
/tmp/output.log {
        daily
        copytruncate
        rotate 4
        compress
}

7. Logrotate postrotate endscript option: Run custom shell scripts immediately after log rotation

Logrotate allows you to run your own custom shell scripts after it completes the log file rotation. The following configuration indicates that it will execute myscript.sh after the logrotation.
$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        postrotate
               /home/bala/myscript.sh
        endscript
}

8. Logrotate maxage option: Remove older rotated log files

Logrotate automatically removes the rotated files after a specific number of days.  The following example indicates that the rotated log files would be removed after 100 days.
$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        maxage 100
}

9. Logrotate missingok option: Dont return error if the log file is missing

You can ignore the error message when the actual file is not available by using this option as shown below.
$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        rotate 4
        compress
        missingok
}

10. Logrotate compresscmd and compressext option: Sspecify compression command for the log file rotation

$ cat logrotate.conf
/tmp/output.log {
        size 1k
        copytruncate
        create
        compress
        compresscmd /bin/bzip2
        compressext .bz2
        rotate 4
}
Following compression options are specified above:
  • compress – Indicates that compression should be done.
  • compresscmd – Specify what type of compression command should be used. For example: /bin/bzip2
  • compressext – Specify the extension on the rotated log file. Without this option, the rotated file would have the default extension as .gz. So, if you use bzip2 compressioncmd, specify the extension as .bz2 as shown in the above example.

How to Rotate Apache Log Files in Linux


Add the following file to /etc/logrotate.d directory.
# vi /etc/logrotate.d/apache
/usr/local/apache2/logs/access_log /usr/local/apache2/logs/error_log {
    size 100M
    compress
    dateext
    maxage 30
    postrotate
      /usr/bin/killall -HUP httpd
      ls -ltr /usr/local/apache2/logs | mail -s "$HOSTNAME: Apache restarted and log files rotated" suresh@sureshkumarpakalapati.in
    endscript
}
Note: Refer to our logrotate tutorial (with 15 examples) that explains more details about how to use logrotate options.
In the above /etc/logrotate.d/apache example:
  • size 100M – Once the access_log, and error_log reaches 100M, it will be rotated. You can also use 100k (for Kb), 100G (for GB). Instead of size, you can also rotate apache logs using frequency (daily, weekly, monthly).
  • compress – Indicates that the rotated log file will be compressed. By default this uses gzip. So, the rotated file will have .gz extension.
  • dateext - Appends the date in YYYYMMDD format to the rotated log files. i.e Instead of access_log.1.gz, it creates access_log-20110616.gz
  • maxage - Indicates how long the rotated log files should be kept. In this example, it will be kept for 30 days.
  • postrotate and endscript – Any commands enclosed between these two parameter will be executed after the log is rotated.
Important: Once you rotate the log files, you want apache to write the new log messages to the newly created access_log and error_log. So, you need to send the HUP signal to the apache as shown here. Make sure to do /usr/bin/killall -HUP httpd, which will restart the apache after rotating the log files (Read more about kill).
Also, you might want to send an email to yourself indicating that the log file is rotated, along with the output of ls -ltr command as the body of the email. i.e Add the following between “postrotate” and “endscript” option (after the killall command).
ls -ltr /usr/local/apache2/logs | mail -s "$HOSTNAME: Apache restarted and log files rotated" suresh@sureshkumarpakalapati.in
The /etc/cron.daily/logrotate script runs everyday that will perform log rotate of all the files as specified in the /etc/logrotate.conf and all the file under /etc/logrotate.d directory.
After adding the above /etc/logrotate.d/apache file, for testing purpose, you can manually call the logrotate script as shown below.
# /etc/cron.daily/logrotate
Once the log files are rotated, do a ls to verify them. As we explained above, the rotated log files will be kept for 30 days.
# ls /usr/local/apache2/logs
access_log
error_log
access_log-20110716.gz
error_log-20110716.gz

7 Linux chkconfig Command Examples – Add, Remove, View, Change Services


Chkconfig command is used to setup, view, or change services that are configured to start automatically during the system startup.
This article contains 7 practical examples that explains how to use the chkconfig command.

1. Check Service Startup status from Shell Script

When you execute chkconfig command only with the service name, it returns true if the service is configured for startup. The following code snippet shows how to check whether a service is configured for startup or not from a shell script.
# vi check.sh
chkconfig network && echo "Network service is configured"
chkconfig junk && echo "Junk service is configured"

# ./check.sh
Network service is configured
You can also specifically check whether it is configured for a particular run level or not.
# vi check1.sh
chkconfig network --level 3 && echo "Network service is configured for level 3"
chkconfig network --level 1 && echo "Network service is configured for level 1"

# ./check1.sh
Network service is configured for level 3

2. View Current Status of Startup Services

The –list option displays all the services with the current startup configuration status.
# chkconfig --list
abrtd   0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid   0:off   1:off   2:off   3:off   4:off   5:off   6:off
atd     0:off   1:off   2:off   3:on    4:on    5:on    6:off
...
To view only the services that are configured to be started during system startup, do the following. Please note that this assumes that your system startup level is 3.
chkconfig --list | grep 3:on
To view the startup configuration of a particular service, grep the output of ‘chkconfig –list’ for that service.
chkconfig --list | grep network

3. Add a new Service to the Startup

Use –add option to add a specific service to the list of services that will be started during system reboot.
The following example shows how to add a new service (for example, iptables) to the list of services that needs to be started. The ‘chkconfig –add’ command will also turn on level 2, 3, 4 and 5 automatically as shown below.
# chkconfig --list | grep iptables

# chkconfig --add iptables

# chkconfig --list | grep iptables
iptables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
Note: “chkconfig –add” only adds an existing service to the list of startup. If the service doesn’t exist, you should first install it before adding it to the system startup list. While this is pretty obvious, it is worth to mention it, as a newbie might make this mistake.

4. Remove a Service From Startup List

The following example shows that ip6tables services is configured for startup.
# chkconfig --list | grep ip6tables
ip6tables       0:off   1:off   2:off   3:on   4:off   5:off   6:off
To remove it from the startup list, use the –del option as shown below.
# chkconfig --del ip6tables

# chkconfig --list | grep ip6tables

5. Turn-on or Turn-off a Service for a Selected Run Level

Sometimes you might not want to delete the whole service. Instead, you might just want to turn the flag on or off for a particular run level (for a particular service).
The following example will turn off nfserver service for level 5
# chkconfig --level 5 nfsserver off
You can also combine multiple levels. The following example will turn off nfsserver for both level 3 and 5.
# chkconfig --level 35 nfsserver off

6. Script Files under rc.d Subdirectories

Whenever you add or remove a service from chkconfig control, it does the following under the /etc/rc.d sub-directories.
  • When chkconfig –add command is executed, it creates a symbolic link file to start and stop the service under corresponding rc directory.
  • When chkconfig –del command is executed, it removes the symbolic link file from the corresponding rc directory.
The following example shows that xinetd is enabled for both run level 3 and 5.
So, xinetd will have two files under rc3.d directory, and two files under rc5.d directory. The file that starts with K is used during shutdown (K stands for kill). The file that starts with S is used during startup (S stands for start).
# chkconfig --list | grep xinetd
xinetd                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
xinetd based services:

# cd /etc/rc.d/rc3.d
# ls | grep xinetd
K08xinetd
S14xinetd

# cd /etc/rc.d/rc5.d

# ls | grep xinetd
K08xinetd
S14xinetd

7. rcx.d Directory Changes for Add Operation

When you add a new service to chkconfig control, the default run levels for that service will be turned on automatically, and files will be created under the corresponding rcx directories.
For example, if the nfsserver service doesn’t exist in the chkconfig control, no nfsserver service startup files would be present under /etc/rc.d/rc*.d directories as shown below.
# chkconfig  --list | grep nfsserver
nfsserver                 0:off  1:off  2:off  3:off  4:off  5:off  6:off

# ls /etc/rc.d/rc3.d | grep nfsserver

# ls /etc/rc.d/rc5.d | grep nfsserver
After you add the nfsserver service, you’ll see the symbolic links under these directories.
# chkconfig --add nfsserver
nfsserver                 0:off  1:off  2:off  3:on   4:off  5:on   6:off

# cd  /etc/rc.d/rc3.d
# ls -l | grep nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 K08nfsserver -> ../nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 S14nfsserver -> ../nfsserver

# cd /etc/rc.d/rc5.d
# ls -l | grep nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 K08nfsserver -> ../nfsserver
lrwxrwxrwx 1 root root 12 2011-06-18 00:52 S14nfsserver -> ../nfsserver
When you turn off the service either using –del option or –level option, the corresponding symbolic link file under rcx.d directory will be deleted as shown below.
# chkconfig --level 5 nfsserver off

# ls /etc/rc.d/rc5.d  | grep nfsserver

7 Default OpenSSH Security Options You Should Change in /etc/ssh/sshd_config


OpenSSH options are controlled through the /etc/ssh/sshd_config file. This article explains the 7 default options in sshd_config file that you should change.
In sshd_config, the lines that start with # are comments. For those options that uses the default values, the sshd_config file contains a commented line with the option and its default value.
This makes it easier for us, as we can see the OpenSSH option name and the default value without having to lookup somewhere else.

For example, sshd_config file contains the following commented line. This indicates that the PubkeyAuthentication option contains “yes” as the default value.
$ grep -i pubkey /etc/ssh/sshd_config
#PubkeyAuthentication yes
If you like to change this, you should remove the comment and change the value (from yes to no) as shown below.
$ vi /etc/ssh/sshd_config
PubkeyAuthentication no
I showed the above only as an example. You don’t need to change the default value of PubkeyAuthentication option, as allowing public key authentication is good.
You don’t need to modify any of the default values in the sshd_config file except the 7 options mentioned in this article.

1. Disable Root Login (PermitRootLogin)

By default you can ssh to the server as root. It is best not to allow root to login directly to the server. Instead, you should login to the system as your account and then do ‘su -’ to login as root.

If you have multiple sysadmins in your organization, and if they all login to the server directly as root, you might not know which sysadmin logged in as root. Instead, if you disable login as root, sysadmins are forced to login as their account first, before they can do ‘su -’, this makes the auditing easier.
Add the following entry to sshd_config to disable root to login to the server directly.
$ vi /etc/ssh/sshd_config
PermitRootLogin no

2. Allow Only Specific Users or Groups (AllowUsers AllowGroups)

By default anybody who is authenticated successfully are allowed to login. Instead you can restrict which users (or groups) you allow to login to the system.
This is helpful when you have created several user accounts on the system, but want only few of them to login.
This is also helpful when you are using NIS, openLDAP (or some other external system) for authentication. Every user in your company might have account on NIS, OpenLDAP etc. But, on a specific server you want only few of them to login. For example, on production system you want only sysadmins to login.
Add the following entry to the sshd_config file to allow only specific users to login to the system. In the example below only ramesh, john and jason can login to this system. Usernames should be separated by space.
$ vi /etc/ssh/sshd_config
AllowUsers ramesh john jason
Add the following entry to the sshd_config file to allow only the users who belong to a specific group to login. In the exampe below only users who belong to sysadmin and dba group can login to the system.
$ vi /etc/ssh/sshd_config
AllowGroups sysadmin dba

3. Deny Specific Users or Groups (DenyUsers DenyGroups)

Instead of allowing specific users (or groups), you can also deny specific users or groups.
Add the following entry to the sshd_config file to deny specific users to login to the system. In the example below cvs, apache, jane cannot login to this system. Usernames should be separated by space.
$ vi /etc/ssh/sshd_config
DenyUsers cvs apache jane
Add the following entry to the sshd_config file to deny users who belong to a specific group to login. In the exampe below users who belong to developers and qa group cannot login to the system.
$ vi /etc/ssh/sshd_config
DenyGroups developers qa
Note: You can use combination of all the Allow and Deny directivies. It is processed in this order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups

4. Change SSHD Port Number (Port)

By default ssh runs on port 22. Most of the attackers will check if a server is open on port 22, and will randomly use brute force to login to the server using several username and password combination.
If you change the port # to something different, others need to know exactly what port to use to login to the server using ssh. The exampe below uses port 222 for ssh.
$ vi /etc/ssh/sshd_config
Port 222
From your logs (/var/log/secure), if you see lot of invalid logins using ssh for accounts that don’t exist on your system, from the
ip-address that you don’t recognize, it migth be some brute-force attack. Those kind of ssh invalid login will stop, if you change the port number.
Please note that this causes little inconvenience to your team who login to the system, as they need to know both the ip-address and the port number.

5. Change Login Grace Time (LoginGraceTime)

When you ssh to a server, you have 2 minutes to login. If you don’t successfully login within 2 minutes, ssh will disconnect.
2 minutes time to login successfully is too much. You should consider changing it to 30 seconds, or may be 1 minute.
Add the following entry to the sshd_config file to change the login grace time from 2 minutes to 1 minute.
$ vi /etc/ssh/sshd_config
LoginGraceTime 1m

6. Restrict the Interface (IP Address) to Login (ListenAddress)

If you have multiple interfaces on the server that are configured to different ip-address, you might not want everybody to login to the server using all those ip-address.
Let us assume that you have the following 4 interfaces on the server:
  • eth0 – 192.168.10.200
  • eth1 – 192.168.10.201
  • eth2 – 192.168.10.202
  • eth3 – 192.168.10.203
By default ssh will listen on all of the above ip-addresses. If you want users to login only using ip-address 200 and 202, do the following in your sshd_config
$ vi /etc/ssh/sshd_config
ListenAddress 192.168.10.200
ListenAddress 192.168.10.202

7. Disconnect SSH when no activity (ClientAliveInterval)

Once you’ve successfully logged in to the system, you might want to get disconnected when there are no activities after x number of minutes. This is basically idle timeout.
In Bash, you can achieve this using TMOUT variable.
In OpenSSH, this can be achieved by combining ClientAliveCountMax and ClientAliveInterval options in sshd_config file.
  • ClientAliveCountMax – This indicates the total number of checkalive message sent by the ssh server without getting any response from the ssh client. Default is 3.
  • ClientAliveInterval – This indicates the timeout in seconds. After x number of seconds, ssh server will send a message to the client asking for response. Deafult is 0 (server will not send message to client to check.).
If you want ssh client to exit (timeout) automatically after 10 minutes (600 seconds), modify the sshd_config file and set the following two parameters as shown below.
$ vi /etc/ssh/sshd_config
ClientAliveInterval 600
ClientAliveCountMax 0