Suresh Kumar Pakalapati's Linux Administration

Tuesday, August 23, 2011

Troubleshooting Incidents related to Server Unavailability

Troubleshooting Chart for Server Unavailbility

List of Vmware Links

20 Linux Log Files that are Located under /var/log Directory

If you spend lot of time in Linux environment, it is essential that you know where the log files are located, and what is contained in each and every log file.

When your systems are running smoothly, take some time to learn and understand the content of various log files, which will help you when there is a crisis and you have to look though the log files to identify the issue.

/etc/rsyslog.conf controls what goes inside some of the log files. For example, following is the entry in rsyslog.conf for /var/log/messages.

$ grep "/var/log/messages" /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

In the above output,

*.info indicates that all logs with type INFO will be logged.
mail.none,authpriv.none,cron.none indicates that those error messages should not be logged into the /var/log/messages file.
You can also specify *.none, which indicates that none of the log messages will be logged.

The following are the 20 different log files that are located under /var/log/ directory. Some of these log files are distribution specific. For example, you’ll see dpkg.log on Debian based systems (for example, on Ubuntu).

/var/log/messages – Contains global system messages, including the messages that are logged during system startup. There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
/var/log/dmesg – Contains kernel ring buffer information. When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process. These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten. You can also view the content of this file using the dmesg command.
/var/log/auth.log – Contains system authorization information, including user logins and authentication machinsm that were used.
/var/log/boot.log – Contains information that are logged when the system boots
/var/log/daemon.log – Contains information logged by the various background daemons that runs on the system
/var/log/dpkg.log – Contains information that are logged when a package is installed or removed using dpkg command
/var/log/kern.log – Contains information logged by the kernel. Helpful for you to troubleshoot a custom-built kernel.
/var/log/lastlog – Displays the recent login information for all the users. This is not an ascii file. You should use lastlog command to view the content of this file.
/var/log/maillog /var/log/mail.log – Contains the log information from the mail server that is running on the system. For example, sendmail logs information about all the sent items to this file
/var/log/user.log – Contains information about all user level logs
/var/log/Xorg.x.log – Log messages from the X
/var/log/alternatives.log – Information by the update-alternatives are logged into this log file. On Ubuntu, update-alternatives maintains symbolic links determining default commands.
/var/log/btmp – This file contains information about failed login attemps. Use the last command to view the btmp file. For example, “last -f /var/log/btmp | more”
/var/log/cups – All printer and printing related log messages
/var/log/anaconda.log – When you install Linux, all installation related messages are stored in this log file
/var/log/yum.log – Contains information that are logged when a package is installed using yum
/var/log/cron – Whenever cron daemon (or anacron) starts a cron job, it logs the information about the cron job in this file
/var/log/secure – Contains information related to authentication and authorization privileges. For example, sshd logs all the messages here, including unsuccessful login.
/var/log/wtmp or /var/log/utmp – Contains login records. Using wtmp you can find out who is logged into the system. who command uses this file to display the information.
/var/log/faillog – Contains user failed login attemps. Use faillog command to display the content of this file.

Apart from the above log files, /var/log directory may also contain the following sub-directories depending on the application that is running on your system.

/var/log/httpd/ (or) /var/log/apache2 – Contains the apache web server access_log and error_log
/var/log/lighttpd/ – Contains light HTTPD access_log and error_log
/var/log/conman/ – Log files for ConMan client. conman connects remote consoles that are managed by conmand daemon.
/var/log/mail/ – This subdirectory contains additional logs from your mail server. For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file
/var/log/prelink/ – prelink program modifies shared libraries and linked binaries to speed up the startup process. /var/log/prelink/prelink.log contains the information about the .so file that was modified by the prelink.
/var/log/audit/ – Contains logs information stored by the Linux audit daemon (auditd).
/var/log/setroubleshoot/ – SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files, and logs those information in this log file.
/var/log/samba/ – Contains log information stored by samba, which is used to connect Windows to Linux.
/var/log/sa/ – Contains the daily sar files that are collected by the sysstat package.
/var/log/sssd/ – Use by system security services daemon that manage access to remote directories and authentication mechanisms.

Instead of manually trying to archive the log files, by cleaning it up after x number of days, or by deleting the logs after it reaches certain size, you can do this automatically using logrotate as we discussed earlier.

To view the log files use any one of the following methods. But, please don’t do ‘cat | more’.

vi – If you are comfortable with the vi commands, use vi editor for quick log file browsing.
tail – If you want to view the content of the log files real time, as the application is writting to it, use “tail -f”. You can also view multiple log files at the same time (using “tail -f”).
grep – If you know exactly what you are looking for in a log file, you can quickly use grep command to grep a pattern. The 15 practical grep examples will take out all your excuses of not using grep.
less – Less command is very powerful to browse log files. Use these 10 less command tipsto master the less command.

Introduction to GIT on Linux – Install, Create Project, Commit Files

The way GIT tracks and handles file changes is very efficient and different than how other version control software tracks the changes (including CVS and Subversion).

GIT is the most versatile distributed version control system.

This article is for those who are new to GIT. This is a jump-start guide that will show you how to install GIT from source, create a new project, commit changes to the GIT repository.

If you are from CVS/SVN background, you are used to the client-server model, where the repository is installed on a server, and you’ll use a client to download the project from the repository, make changes, and commit it to the repository in the server.

GIT doesn’t use the client-server model. When you download a project from a remote GIT repository, you download everything, including the version history and changes of the individual files, and your local GIT acts as a server, where you can do check-in, check-out, and all other typical version control activities. Later when you are ready, you can merge your changes to the remote GIT repository.

So, the installation and configuration steps are exactly the same whether you using GIT on your local machine, to manage your own project, or you are installing GIT on a server, from where other developers will download the project to their local GIT repositories.

If you are a developer, you might want to install GIT on your local machine for two reasons: 1) You like to manage your own project locally using a version control tool 2) You want to modify a code that is located in a remote central GIT repository.

If you are a sysadmin, you might want to install GIT on a server so that it acts as a central repository to hold all the source code for your company. From here, you can allow developers to download the projects to their local GIT repositories, make changes and they can check-in back to your central repository when they are done.

Irrespective of how you are planning to use GIT, the installation steps, and the basic commands mentioned below are exactly the same.

1. Download and Install GIT

First, download the GIT from here. Or, download it directly using wget as shown below.

cd
wget http://kernel.org/pub/software/scm/git/git-1.7.6.tar.bz2

Next, extract the downloaded file.

tar xvfj git-1.7.6.tar.bz2
cd git-1.7.6

Finally, install GIT as shown below using the default configure option. If you want to customize the installation, do “./configure –help” to view all available configuration options.

./configure

make

make install

2. Initial Configuration

Git is installed by default under /usr/local/bin. Once you’ve installed GIT, verify it as shown below.

$ whereis git
git: /usr/local/bin/git

$ git --version
git version 1.7.6

$ git --help.

The first step is to specify your username and email address to your GIT repository using “git config” as shown below.

git config --global user.name "GIT Admin"

git config --global user.email ramesh@thegeekstuff.com

Verify the git configuration information as shown below.

$ git config --list
user.name=GIT Admin
user.email=ramesh@thegeekstuff.com
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true

This information is stored in the .gitconfig file under your home directory.

$ cat ~/.gitconfig
[user]
        name = GIT Admin
        email = ramesh@thegeekstuff.com

3. Create a Project

You can make any of your local directory as a GIT project (i.e repository). For example, if your project is located under /home/ramesh/projects/passworddragon, you can make that as your GIT project. First, cd to that directory, and execute git init as shown below.

$ cd /home/ramesh/projects/passworddragon

$ git init
Initialized empty Git repository in /home/ramesh/projects/passworddragon/.git/

This will create a .git directory under your project folder. Following is the content of the .git directory. GIT uses this directory to store information on how it is tracking the changes.

$ ls -altr .git
total 40
drwxrwxr-x  4 git git 4096 Aug 13 22:39 refs
drwxrwxr-x  4 git git 4096 Aug 13 22:39 objects
drwxrwxr-x  2 git git 4096 Aug 13 22:39 info
drwxrwxr-x  2 git git 4096 Aug 13 22:39 hooks
-rw-rw-r--  1 git git   23 Aug 13 22:39 HEAD
-rw-rw-r--  1 git git   73 Aug 13 22:39 description
-rw-rw-r--  1 git git   92 Aug 13 22:39 config
drwxrwxr-x  2 git git 4096 Aug 13 22:39 branches
drwxrwxr-x 36 git git 4096 Aug 13 22:39 ..
drwxrwxr-x  7 git git 4096 Aug 13 22:39 .

Note: If you are sysadmin, who is trying to create a GIT central repository for your company, from where developers can download the projects, you may want to create a username called ‘git’ and organize all your projects under this account. For example: /home/git/project1, /home/git/project2, etc. Once you have the project organized, cd to the project directory, and do ‘git init’ from there as git user.

4. Add and Commit files to the Project

Once you’ve initialized the project using “git init”, add the files located under this project directory, using “git add”.

If there are different types of files under your project directory, and you want GIT to manage only certain types of files, add only those to the GIT as shown below. This example adds only the *.java and *.c files.

git add *.java
git add *.c

Typically you would like to add all the files under the project directory to the GIT project. Just do “git add .”, which will add all the files in the current directory and all the sub-directories to the GIT project.

git add .

Once you’ve added the files to the repository, you should commit those files, as shown below.

$ git commit -m 'Initial upload of the project'
 create mode 100755 PasswordDragon.java
 create mode 100755 pwm/ui/DataManager.java
 create mode 100755 pwm/ui/PasswordFrame.java
 create mode 100755 pwm/tools/StrongEncryption.java
 create mode 100755 pwm/tools/PasswordStrength.java
 ..

If you didn’t specify your username and email address using “git config” as explained above, you’ll get the following error message.

$ git commit -m 'Initial upload of the project'

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident   not allowed

5. Make Changes and Commit the File

You’ve installed GIT, created a project repository, committed all the files to the GIT project.

Now it is time to start making some changes to a file and commit it to the repository.

vi PasswordDragon.java

Once you’ve modified a file locally, you can view the changes. i.e The difference between your local copy, and the copy already committed in the GIT project using “git diff” as shown below.

$ git diff
diff --git a/PasswordDragon.java b/PasswordDragon.java
index 6166ed1..fd82d32 100644
--- a/PasswordDragon.java
+++ b/PasswordDragon.java
@@ -2,7 +2,7 @@
-    public counter=10
+    public counter=55

Once you’ve made modifications to it, reviewed the changes, and happy with it, you can commit the file to GIT repository. This is a two step process. First, you should add the file to the staging area, and commit to the GIT project as shown below.

git add PasswordDragon.java

When you perform commit, it will open your default editor, where you can enter the comment. Once you save your comment and exit the editor, it will commit the file to the GIT project and display the following message.

$ git commit
[master 80f10a9] Added password strength meter functionality
 1 files changed, 56 insertions(+), 7 deletions(-)

Note: You can also do “git commit -a”, which will do the add and commit at the same time.

6. View Status and Commit Logs

From your local repository, when you perform “git status”, it will display the current status. When the local copy is not changed (or when all the files are committed), you’ll see the following message.

$ git status
# On branch master
nothing to commit (working directory clean)

If you’ve made changes to a file, and not committed yet, you’ll see the following message.

$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#       modified:   PasswordDragon.java
#
no changes added to commit (use "git add" and/or "git commit -a")

You can also view the history of a file as shown below.

$ git log PasswordDragon.java
commit c919ced7f42f4bc06d563c1a1eaa107f2b2420d5
Author: GIT Admin
Date:   Sat Aug 13 22:54:57 2011 -0700

    Added password strength meter functionality

commit c141b7bdbff429de35e36bafb2e43edc655e9957
Author: GIT Admin
Date:   Sat Aug 13 20:08:02 2011 -0700

    Initial upload of the project