Thursday, August 19, 2010

Overview of OpenNebula Step by Step Guide

( Ref : OpenNubula.org  )


OpenNebula can be primarily used as a virtualization tool to manage your virtual infrastructure in the data-center or cluster.

  • OpenNebula is a toolkit for building Clouds ( public cloud,private cloud )
  •  It Supports KVM , Xen , Vmware Server , VirtualBox (planned from 1.4.2)
  •  It consists of  Front End , Node
    Front End installation

    $ sudo apt-get install opennebula
          Reading package lists... Done 
          Building dependency tree 
          Reading state information... Done 
          The following extra packages will be installed: 
            opennebula-common 
          The following NEW packages will be installed: 
            opennebula opennebula-common 
          0 upgraded, 2 newly installed, 0 to remove and 2 not upgraded. 
          Need to get 280kB of archives. 
          After this operation, 1352kB of additional disk space will be used. 
          Do you want to continue [Y/n]? 
          … 
          Setting up opennebula-common (1.2-0ubuntu1~intrepid1) … 
          … 
          Adding system user `oneadmin’ (UID 107) … 
          Adding new user `oneadmin’ (UID 107) with group `nogroup’ … 
          Generating public/private rsa key pair. 
          … 
          Setting up opennebula (1.2-0ubuntu1~intrepid1) …
          oned and scheduler started 


    Node Installation

    $ sudo apt-get install opennebula-node 
        Reading package lists... Done 
        Building dependency tree 
        Reading state information... Done 
        The following extra packages will be installed: 
          opennebula-common 
        The following NEW packages will be installed: 
          opennebula-common opennebula-node 
        ... 
        Setting up opennebula-node (1.2-0ubuntu1~intrepid1) ...
        Adding user `oneadmin' to group `libvirtd' ... 
        Adding user oneadmin to group libvirtd 
        Done. 

     OpenNebula Core
    ----------------------
        *      Request Manager, to handle client requests
        *      Virtual Machine Manager, to manage and monitor of VMs
        *      Transfer Manager, to manage VM images
        *      Virtual Network Manager, to manage virtual networks
        *      Host Manager, to manage and monitor physical resources
        *      Database, persistent storage for ONE data structures

    OpenNebula does the following:
    ------------------------------------
        *      Manages Virtual Networks. Virtual networks interconnect VMs. Each Virtual Networks includes a description.
        *      Creates VMs. The VM description is added to the database.
        *      Deploys VMs. According to the allocation policy, the scheduler decides where to execute the VMs.
        *      Manages VM Images. Before execution, VM images are transferred to the host and swap disk images are created. After execution, VM images may be copied back to the repository.
        *      Manages Running VMs. VM are started, periodically polled to get their consumption and state, and can be shutdown, suspended, stopped or migrated.

    The main functional components of an OpenNebula Private Cloud are the following:
    -------------------------------------------------------------------------------------
        *      Hypervisor: Virtualization manager installed in the resources of the cluster that OpenNebula leverages for the management of the VMs within each host.
        *      Virtual Infrastructure Manager: Centralized manager of VMs and resources, providing virtual network management, VM life-cycle management, VM image management and fault tolerance.
        *      Scheduler: VM placement policies for balance of workload, server consolidation, placement constraints, affinity, advance reservation of capacity and SLA commitment.



    Daemon Configuration Attributes
    -------------------------------------

        *      HOST_MONITORING_INTERVAL : Time in seconds between host monitorization
        *      VM_POLLING_INTERVAL : Time in seconds between virtual machine monitorization
        *      VM_DIR : Remote path to store the VM images, it should be shared between all the cluster nodes to perform live migrations. This path will be used for all the cluster nodes.
        *      MAC_PREFIX: Default MAC prefix to generate virtual network MAC addresses
        *      NETWORK_SIZE: Default size for virtual networks
        *      PORT : Port where oned will listen for xml-rpc calls
        *      DEBUG_LEVEL : Sets the level of verbosity of $ONE_LOCATION/var/oned.log log file. Possible values are:

    DEBUG_LEVEL    Meaning
    ---------------------------------
    0                               ERROR
    1                               WARNING
    2                               INFO
    3                               DEBUG



    Command line interface
    --------------------------------

        *      onevm: to submit, control and monitor virtual machines
        *      onehost: to add, delete and monitor hosts


    $ onehost list
     HID NAME                      RVM   TCPU   FCPU   ACPU    TMEM        FMEM       STAT
       0 host01                      0         800     800     800      8194468    7867604   on
       1 host02                      0         800      797    800      8387584    1438720   on

    $ onevm list
     ID     USER     NAME         STAT  CPU     MEM            HOSTNAME        TIME
     1001 oneadmin  fc11AOS   dele      0          512000           node2                02 22:10:56

    OpenNebula

    What is OpenNebula ?
    •It is an open source toolkit for building clouds.
    •The latest release has built-in support for KVM, Xen Kernel and VmWare Server.

    How OpenNebula works ?
    OpenNebula is a distributed application containing two components 

    The first component referred to as OpenNebula front-end runs on VX64 server. 

    sudo apt-get install opennebula 

    The second component referred to as OpenNebula Node should be installed on all hosts part of the compute cluster. This package prepares the machine for being a node in an OpenNebula cloud. It in turn configures the following dependent packages: 

    1) KVM 
    2) libvirt 
    3) oneadmin user creation 
    4) ruby 

    sudo apt-get install opennebula-node


    What are the components of opennebula front-end ?


    OpenNebula front-end spawns the following processes when it starts: ( Note that front-end should always be started as oneadmin user using the command sudo -u oneadmin one start ) 

    1) OpenNebula Daemon ( oned ) - Responsible for handling all incoming requests ( either from CLI or from API ). Talks to other processes whenever required. 

    2) Scheduler ( mm_sched ) - It does match making to find a suitable host ( amongst the hosts part of compute cluster ) for bringing up virtual machines. 

    3) Information Manager ( one_im_ssh.rb ) - Collects resource availability/utilization information for hosts/VMs respectively. ( Resources include CPU and Memory) 

    4) Transfer Manager ( one_tm.rb ) - Responsbile for Image Managemnt ( Clone Image, Delete Image, etc. ) 

    5) Virtual Machine Manager ( one_vmm_kvm.rb )- Acts as interface to the underlying Hypervisor. ( All operations to be performed on Virtual Machines go through this interface ) 

    6) Hook Manager ( one_hm.rb ) - Responsible for executing Virtual Machine Hooks. ( Hooks are programs which are automatically triggered on VM state changes. They must be configured prior to starting front-end ) 

    OpenNebula front-end can be configured to a great extent by modifying the contents of file /etc/one/oned.conf. Consider a sample configuration below: 

    ################################################################################
    HOST_MONITORING_INTERVAL=10 # Used by Information manager to decide the frequency at which resource availability details have to be collected for hosts 

    VM_POLLING_INTERVAL=10 # Used by Information manager to decide the frequency at which resource utilization details have to be collected for VMs 

    VM_DIR=/mnt/onenfs/ # Should be shared across all hosts in compute cluster. Contains Disk Images required for booting VMs 

    PORT=2633 # All supported API calls are converted to XML-RPC calls. front-end runs an XML-RPC Server on this port to handle these calls 

    DEBUG_LEVEL=3 # DEBUG_LEVEL: 0 = ERROR, 1 = WARNING, 2 = INFO, 3 = DEBUG 

    NETWORK_SIZE = 254 # Default size for Virtual Networks ( applicable while using onevnet ) 

    MAC_PREFIX = "00:03" # Default MAC prefix to use while generating MAC Address from IP Address ( applicable while using onevnet ) 

    # The following configuration supports KVM hypervisor. Note that the executables one_im_ssh one_vmm_kvm one_tm and one_hm can be found in /usr/lib/one/mads/ 

    IM_MAD = [ 
    name = "im_kvm", 
    executable = "one_im_ssh", 
    arguments = "im_kvm/im_kvm.conf" ] 

    VM_MAD = [ 
    name = "vmm_kvm", 
    executable = "one_vmm_kvm", 
    default = "vmm_kvm/vmm_kvm.conf", 
    type = "kvm" ] 

    TM_MAD = [ 
    name = "tm_nfs", 
    executable = "one_tm", 
    arguments = "tm_nfs/tm_nfs.conf" ] 

    HM_MAD = [ 
    executable = "one_hm" ]



    Working with OpenNebula CLI ( OpenNebula CLI is available only on front-end ) ?

    1) Adding a new Host to compute cluster 

    onehost create

    Note that im_mad, vmm_mad, tm_mad in our case should be im_kvm, vmm_kvm and tm_nfs respectively as we have configured them in /etc/one/oned.conf

    Also note that Information manager needs to collect resource availability ( CPU and Memory ) information for the host we have added. This requires:

    • oneadmin user on front-end should be able to ssh to host without entering password ( test this using sudo -u oneadmin ssh oneadmin@ on front-end )
    • Inorder for this to work copy the contents of /var/lib/one/.ssh/id_rsa.pub on front-end to /var/lib/one/.ssh/authorized_keys on host

    Type onehost list to check the status:

    Notice the value of STAT attribute. If the attribute has a value of 'on' then the host has been successfully added to compute cluster:

    ID NAME RVM TCPU FCPU ACPU TMEM FMEM STAT
    0 192.168.155.127 0 200 198 198 1800340 1341620 on


    Look into /var/log/one/oned.log on front-end for debugging
    Once a host has been successfully added use onehost disable/enable to toggle its status.


    2) Submitting a new Virtual Machine job 

    In order to provision a Virtual Machine in the compute cluster we need to construct a template and submit it using

    onevm create

    The following is a sample template:
    ################################################################################
    NAME = Ubuntu
    CPU = 0.5
    MEMORY = 150
    DISK = [clone=no, type="disk",source="/mnt/onenfs/ubuntu.img", target="hda"]
    GRAPHICS = [
    type = "vnc",
    listen = "0.0.0.0",
    port = "10"]

    ################################################################################

    Note that all Virtual Machines will use the default properties present in /etc/one/vmm_kvm/vmm_kvm.conf unless they are overriden in the template.

    The following is a sample /etc/one/vmm_kvm/vmm_kvm.conf which enables ACPI for all VMs and configures boot from hard disk.

    ###############################################################################
    OS = [ boot = "hd" ]

    FEATURES = [
    PAE=no,
    ACPI=yes
    ]
    ################################################################################

    Issues with VM coming up will be captured either in /var/log/one/oned.log in front-end or /var/log/libvirt/qemu/one-.log on host where VM job has been scheduled.


    4) Creating Virtual Networks 

    ( Note that onevnet does not create bridge or vlan. We should use brctl and vconfig respectively )

    Create a VLAN ( creating VLAN 12 here )

    sudo vconfig add eth0 12

    Create a bridge and encapsulate the above VLAN

    sudo brctl addbr br12

    sudo brctl addif br12 eth0.12

    Bring up the VLAN

    ip link set eth0.12 up

    Create a file named "vlan12.one" containing the following content:

    BRIDGE=br12
    NAME=VLAN12
    NETWORK_ADDRESS=192.168.12.0
    NETWORK_SIZE=B
    TYPE=RANGED

    Create a Virtual Network using the following command:
    onevnet create vlan12.one

    View the list of created Virtual Networks

    # COMMAND #
    onevnet list

    # OUTPUT #
    ID USER NAME TYPE BRIDGE #LEASES
    0 shashank VLAN12 Ranged br12 0

    View the configuration of a specific Virtual Network

    # COMMAND #
    onevnet show 0

    # OUTPUT #
    VIRTUAL NETWORK 0 INFORMATION
    ID: : 0
    UID: : 0

    VIRTUAL NETWORK TEMPLATE
    BRIDGE=br12
    NAME=VLAN12
    NETWORK_ADDRESS=192.168.12.0
    NETWORK_SIZE=B
    TYPE=RANGED

    To Bring up a Virtual Machine in above created Virtual Network, add the following to VM template:

    NIC=[NETWORK="VLAN12",IP=192.168.12.4]

    ( Note that MAC address for this VM is dynamically generated based on the IP specified )

    To view the list of IP addresses currently allocated in the above created virtual network, issue the following command

    # COMMAND #
    onevnet show 0

    VIRTUAL NETWORK 0 INFORMATION
    ID: : 0
    UID: : 0

    VIRTUAL NETWORK TEMPLATE
    BRIDGE=br12
    NAME=VLAN12
    NETWORK_ADDRESS=192.168.12.0
    NETWORK_SIZE=B
    TYPE=RANGED

    LEASES INFORMATION
    LEASE=[ IP=192.168.12.4, MAC=00:03:c0:a8:0c:04, USED=1, VID=12 ]


    5) Disadvantages of using onevnet 

    1) An additional script is required inside the VM which does MAC -> IP conversion and configures the network device on boot.

    2) onevnet does not create bridge or vlan.

    3) Virtual Machine job will not be submitted if the specified IP is not available.

    4) There seems to be no way to specify the IP lease time. DHCP server is highly customizable.

    5) If we plan to give hostnames to Virtual Machines in future we might retain DHCP server and configure DDNS updates which may not be possible while using onevnet leases.


    3) Enabling additional debugging 

    Edit /etc/one/defaultrc file to add the following:

    ONE_MAD_DEBUG=1

    The above setting will create the following log files in /var/log/one/

    im_kvm.log
    one_vmm_kvm.log
    tm_nfs.log
    one_hm.log


    6) Viewing and Modifying OpenNebula backend database 

    If requried we can use sqlitebrowser to open and edit /var/lib/one/one.db ( sudo apt-get install sqlitebrowser )


    How to Configure Local Apt-Cacher-Server for Intranet U

    Server Installation:

    1. Install apt-cacher and apache2 webserver

    #apt-get install apt-cacher apache2

    2.Enable apt-cacher

    Edit /etc/default/apt-cacher and change autostart to 1. (This will automatically start
    cache server).

    #vi /etc/default/apt-cacher

    autostart=1

    3.Restart the apache server

    #/etc/init.d/apache2 restart

    4.Restart the apt-cacher server

    #/etc/init.d/apt-cacher restart

    Now test your apt-cacher server.Go to remote machine in your network.open your
    browser type the hostname or IP address of your apt-cacher
    server.

    #firefox http:///apt-cacher  [ hostname / ip ]

     
    sometimes it won't work. Then try

    #firefox http://hostname/ip :3142";

    save and quit

    [OR]

    Create a small script.

    #vi check-apt-cacher.sh
    ---------------------------------------------------------------------------------------------------

    . /lib/lsb/init-functions
    log_daemon_msg "Configuring APT cache proxy" "(based on SERVER_NAME_HERE's
    presence...)"
    ping -c 1 SERVER_NAME_HERE &> /dev/null
    if [ $? = "0" ]; then
    echo "Acquire::http::Proxy \"http://SERVER_NAME_HERE:3142\";" > /etc/apt/
    apt.conf.d/01SERVER_NAME_HEREproxy
    else
    rm /etc/apt/apt.conf.d/01SERVER_NAME_HEREproxy &> /dev/null
    fi
    log_end_msg 0

    ---------------------------------------------------------------------------------------------------

    (Note: Replace SERVER_NAME_HERE with the ipaddress of apt-cacher server).
    save and quit
    #chmod +x /root/check-apt-cacher
    #crontab -e
    */1 * * * * /bin/bash /root/check-apt-cacher
    save and quit
    This script will run each minute (*/1) you can change it as you see fit.
    You're done.
    ---------------------------------------------------------------------------------------------------
    ---------------------------------------------------------------------------------------------
    How It Works?
     
    In each and every client machine in our network, we are going to mention the apt-
    cacher server as a proxy server in the /etc/apt/apt.conf.d/ directory. So in our network,
    If anyone Downloads packages using apt-get command. First, it checks in the cache
    folder of apt-cacher server i.e.., /var/cache/apt-cacher/packages. If it doesn't find any
    packages, it downloads from the internet and keeps in its cache folder and as well as
    installs on the client machines(regardless of what version of ubuntu they are using). So
    that if any other people tries to install the same package,they can directly install it from
    the cache instead of hitting internet all the time,which saves bandwidth.
    ---------------------------------------------------------------------------------------------------
    (Note : I have installed apt-cacher server on ubuntu 9.04 and i tried to access from
    client side which is of ubuntu 9.04 and ubuntu 10.04. i installed few packages on the
    client side, which client and as well as server doesn't have. The new package is fetched
    form the internet and is kept in the apt-cacher cache directory.)

    Centralized Log Management with RSyslog

    Why Rsyslog?

    For Centralized logging.Having a centralized logging is a prerequisite if you want to have your logs intacts. But having the events recorded in plain files is virtually impossible for queries.Because of this, we can configure rsyslog to write events in MySQL DB and a web interface for queries with filters to facilitate viewing of the logs without have to access the console for such task.

    Rsyslog Installation

    Installing rsyslog on Ubuntu is easy:

     #aptitude install rsyslog
     #aptitude install rsyslog-doc
    All configuration is placed in the /etc/rsyslog.conf file or in files found under the /etc/rsyslog.d directory.
    • Configuration structure :Configuration files are structured in the following manner:



      • Modules
      • Global directives
      • Filter rules
    • Rsyslog Server configuration: 
     #vim /etc/rsyslog.conf
    Local logging functionality is provided by the imuxsock plug-in (also 
    enabled by default).

    $ModLoad imuxsock
     $ModLoad imklog
    On the server, assuming you are running rsyslog, you do this by enabling the appropriate input module, as well as specifying the port to be used:
    UDP:
     
    $ModLoad imudp
    $UDPServerRun 514
    TCP:

    $ModLoad imtcp 
    $InputTCPServerRun 514 

    Rsyslog provides a flexible system to specify the output formats. It is template-based. A template with the traditional syslog format looks as follows:



    $template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"


     Now we have the right template - but how to write it to a file? You probably have a line like this in your rsyslog.conf:

    *.* -/var/log/messages.log;TraditionalFormat

    -------------------------------------------------------------------------

    Writing syslog messages to MySQL

    --------------------------------------------------------------------------

     rsyslog-mysql mysql-server php-mysql

    while installing you will be prompted for mysql rsyslog user password.Enter the details and remember them.Next, the server should load the output module ommysql and be configured to connect to the database. Its configuration should be similar to the following:

    #vim /etc/rsyslog.conf
    # service mysqld start
    # mysql -uroot -predhat
    mysql> GRANT SELECT, UPDATE, INSERT ON Syslog.* TO rsyslog@localhost IDENTIFIED BY 'password'; 
    mysql> \q 
    #vim /etc/rsyslog.conf 
    $ModLoad ommysql 
    *.* :ommysql:127.0.0.1,Syslog,rsyslog,rsyslog 
    [*.*  :ommysql:database-server,database-name,database-userid,database-password]
    #restart mysql 
    #restart rsyslog 

    Now check your database:

    #mysql -uroot -predhat 
    #show databases; 
    #USE Syslog;
    #show tables;
    #DESCRIBE SystemEvents; 
    #select * from SystemEvents; 

    you can see all your log messages here,if you have properly configured.Generate few log messages with logger command, then open three terminals.

    #tail -f /var/log/messages
    #logger system rebooted 
    #mysql 


    check the timestamps of the logs and whether they are being logged into database or not.
    • Clients Setup:
    #apt-get install rsyslog


    First, you need to create a working directory for rsyslog. This is where it stores its queue files (should need arise). You may use any location on your local system. There is nothing else to do. With the following simple config file, you forward anything you receive to a remote server and have buffering applied automatically when it goes down. This must be done on the client machine.

    #vim /etc/syslog.conf
    $ModLoad imuxsock    # local message reception 
    $WorkDirectory /rsyslog/work       # default location for work (spool) files 
    $ActionQueueType LinkedList       # use asynchronous processing
    $ActionQueueFileName srvrfwd    # set file name, also enables disk mode 
    $ActionResumeRetryCount -1       # infinite retries on insert failure 
    $ActionQueueSaveOnShutdown on  # save in-memory data if rsyslog shuts down. 

    *.* @@server:port 
    *.* @172.168.155.75:514      #forward all its logs via UDP 
    *.* @@172.168.155.75:514  #forward all its logs via TCP. 

    -----------------------------------------------------------

    LogAnalyzer - WebUI

    ------------------------------------------------------------

    The LogAnalyzer project provides an easy to use but powerful frontend for searching, reviewing and analyzing syslog, event log and many other event sources.
    • HOWTO install LogAnalyzer
    To install LogAnalyzer, you need:

    * Apache
    * PHP5

    #apt-get install apache2
    #apt-get install php5
    #apt-get install libapache2-mod-php5
    #/etc/init.d/apache2 restart 

    Now install Loganalyzer. It's pretty easy.
    #wget http://download.adiscon.com/loganalyzer/loganalyzer-3.0.1.tar.gz 
    #tar -xvzf loganalyzer-3.0.1.tar.gz 
    #mkdir -p /var/www/loganalyzer 

    Upload all files from the loganalyzer/src/ folder to you webserver. The other files are not needed on the webserver.

    #cp -R loganalyser-v3.0.1/src/* /var/www/loganalyzer 


    Upload the scripts configure.sh and secure.sh from the contrib folder to your webserver, into the same folder where you uploaded the other LogAnalyzer files into. Then set the execution flag to them (chmod +x configure.sh secure.sh).Now run ./configure.sh, this will create a blank config.php, and will also set write access to everyone to it.You can of course do this manually if you want.

    #chmod +x configure.sh secure.sh 
    #./configure.sh 

    Now open your LogAnalyzer installation in your favourite webbrowser, you will see an error, and you will be pointed to the installation script. The install script will guide you through the LogAnalyzer installation, just follow the instructions.



     

     

     

     

     

     

     

     Some Related Links for more information

    http://www.ubuntu.com/system/files/CentralLogging-v4-20090901-03.pdf

    ShellScripts

    Bash Script to find CPU type or model?

    #!/bin/sh

    if [ -f /proc/cpuinfo ]; then

        echo -n "MODELNAME=\""
        grep -m 1 "model name" /proc/cpuinfo | cut -d: -f2 | sed -e 's/^ *//' | sed -e 's/$/"/'
       
    fi



    OUTPUT:
    MODELNAME="Intel(R) Core(TM)2 Duo CPU     E8200  @ 2.66GHz"




    Useful Tips & Tricks

    View Config Files Without Comments
    grep ^[^#] /etc/apache2/apache2.conf
    grep ^[^#] /etc/httpd/conf/httpd.conf

    Restore data from amanda backup server

    Continution to my post install and configure amanda backup server i am posting step by step process of
    recovering data from the backup server.

    Our Current Scenario is : 

    1. My daily backup profile is DailySet1

    2. The data backup is in /data/amanda/vtape/DailySet1 on amanda server.

    3. We will recover data on client system to /tmp/amanda  directory

    -----------------------------------------------------------------------------------------------
    Do this on Amanda Client System
    -----------------------------------------------------------------------------------------------

    Switch user to root

    # sudo su
     Then as a root user do amrecover

    # amrecover DailySet1

     then use  listdisk command to see the list of  backups

    # listdisk 

      it shows the client names and the directories we are taking backup on that client
    Then use setdisk command to set the directory which we want to recover data from


    # setdisk /data/backup


    then add the files you want to recover for example if you want to recover file1
    use

    # add file1


    In my case i have give only one file to take backup so i will use '*' to  recover all files in that
    direcory.


    # add *


    Then finally use extract command to recover all files.


    The output of all these commands can be seen here :




    After giving extract command it will gives the list of data extracting location , tapes needed and the location of backup directory and it will ask for confirmation


    Continue [ ?/Y/n/s/d]?


    i am showing the rest of the flow in screen shots have a look at them



    we can see from the above screenshot that amanda-backup-client debian package has been recovered .


    we can see that file below in the temp directory.


    Install and configure Amanda Backup Server

    Amanda is one of the complex installation as it has got many features and one can be easily be confused when
    installing it for first time. So i have thought of  sharing minimal installation steps to jump start your backup

    First download Server and Client packages of  amanda depending on operating system you use from below
    location

    http://www.zmanda.com/downloads/community/Amanda/

    I have taken 2 ubuntu machines 9.04

    Server -- 192.168.155.108  - amanda-backup-server_3.1.0-1Ubuntu904_i386.deb
    Client  -- 192.168.155.109 - amanda-backup-client_3.1.0-1Ubuntu904_i386.deb

    after installation just update your repository
    ------------------------------------------------------------------------------------------------
    On Server do this 
    -----------------------------------------------------------------------------------------------


    #apt-get update

    then install all build essentials required for a package to install

    #apt-get install build-essential

    afther that install debian package

    #dpkg -i  amanda-backup-server_3.1.0-1Ubuntu904_i386.deb

    I've noticed that if I try to install this package it doesn't seem to work because of missing dependent librararies typing


    #apt-get -f install

    will solve dependency problem

    After installation of that package

    User amandabackup
    group disk
     will be created.

    add password for the user amandabackup

    # passwd amandabackup

    after adding user we need to create directory where our backup has to be stored.

    # mkdir -p /data/amanda/vtape/DailySet1 
    # chown amandabackup:disk /data/amanda/vtape/DailySet1
    # chmod -R 750 /data/amanda/vtape/DailySet1



    ------------------------------------------------------------------------------------------------
    On Client do this 
    -----------------------------------------------------------------------------------------------
     Install client package same as you installed on server

    here also user amandabackup will get created

    change password for user amandabackup 

    #passwd amandabackup

    The client configuration file of amanda is

    /etc/amanda/amanda-client.conf

    Change this line:

    tapedev    "tape:/dev/YOUR-TAPE-DEVICE-HERE" # your tape device
    to this:

    tapedev    "file://data/amanda/vtape/DailySet1" # your tape device

    this is the location where we are taking backup.

    after configuring  go to server again.

    ------------------------------------------------------------------------------------------------
    On Server do this 
    -----------------------------------------------------------------------------------------------

    We have created a backup directory and created user amandabackup and set its password on Server
    now switch user to amandabackup

    # su – amandabackup

    then enter this command to configure server configuration file for DailySet1

    # amserverconfig DailySet1 --template harddisk --tapedev /data/amanda/vtape/DailySet1 --mailto root@localhost --dumpcycle 1week --runspercycle 5 --tapecycle 12 --runtapes 1 

    the output will be:



















    Now adding Client 192.168.155.109 on server give second command

    Explaining:
    amserverconfig : command that creates the initial Amanda configuration
    daily : the name of this backup set
    template disk : this backup is written to disk (or “virtual” tape -vtape-)
    tapedev /var/lib/amanda/vtapes/daily : this is the path of the vtapes
    mailto me@mydomain.com : Amanda emails the backup report to this address
    dumpcycle 5 : This backup cycle has 5 days (lets say ... Mon to Fri)
    runspercycle 1 : This backup runs only once per day
    tapecycle 20 : 20 vtapes (4 weeks) on the cycle
    runtapes 1 : The maximum number of tapes (in this case “vtapes”) used in a single run


    # amaddclient --config DailySet1 --client 192.168.155.109 --diskdev /data/backupme --dumptype  comp-user-tar 


    /data/backupme  is the directory on the client you want to take backup

    output will be :

























    after adding client now everything is ready we should test its working or not
    we can test it using amcheck command

    # amcheck DailySet1


    the output will be:













    Now checking is successful if you see 0 problems found in the output as shown above.
    now we are ready to take first backup with the command amdump

    # amdump DailySet1

    This command will take backup and the status is emailed to root@localhost of server as we have configured above.

    the mail will be like this :


    From amandabackup@amandaserver  Tue Jul 20 15:19:10 2010
    Return-Path:
    X-Original-To: root@localhost
    Delivered-To: root@localhost
    Received: by amandaserver (Postfix, from userid 1001)
        id 98122602ED; Tue, 20 Jul 2010 15:19:10 -0400 (EDT)
    To: root@localhost
    Subject: DailySet1 AMANDA MAIL REPORT FOR July 20, 2010
    Message-Id: <20100720191910.98122602ED@amandaserver>
    Date: Tue, 20 Jul 2010 15:19:10 -0400 (EDT)
    From: amandabackup@amandaserver (Amanda)

    Hostname: amandaserver
    Org     : DailySet1
    Config  : DailySet1
    Date    : July 20, 2010

    These dumps were to tape DailySet1-1.
    The next tape Amanda expects to use is: 1 new tape.
    The next new tape already labelled is: DailySet1-2.


    STATISTICS:
                                            Total       Full      Incr.
                                            --------   --------   --------
    Estimate Time (hrs:min)  0:00
    Run Time (hrs:min)         0:00
    Dump Time (hrs:min)      0:00       0:00       0:00
    Output Size (meg)            1.6        1.6        0.0
    Original Size (meg)          1.6        1.6        0.0
    Avg Compressed Size (%) 98.8       98.8        --
    Filesystems Dumped         1          1          0
    Avg Dump Rate (k/s)        163.1      163.1        --

    Tape Time (hrs:min)          0:00       0:00       0:00
    Tape Size (meg)                1.6        1.6        0.0
    Tape Used (%)                  0.0        0.0        0.0
    Filesystems Taped              1          1          0
    Parts Taped                        1          1          0
    Avg Tp Write Rate (k/s)    160.1      160.1        --

    USAGE BY TAPE:
      Label               Time         Size      %    Nb    Nc
      DailySet1-1         0:00           0G    0.0     1     1

    NOTES:
      planner: Adding new disk 192.168.155.109:/home/vxadmin.
      taper: tape DailySet1-1 kb 1601 fm 1 [OK]


    DUMP SUMMARY:
                                                 DUMPER STATS               TAPER STATS
    HOSTNAME     DISK        L ORIG-GB  OUT-GB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
    -------------------------- ------------------------------------- -------------
    192.168.155. -me/vxadmin 0       0       0   98.8    0:10  163.2   0:10  160.1

    (brought to you by Amanda version 3.1.0)


    So you have succesfully taken backup of your first client

    Is Virtualization the Same as Cloud Computing?




    Turning to that modern oracle, Wikipedia, is helpful. Wikipedia doesn't provide a single definition for virtualization; instead, it defines (if you can call it that) various types of virtualization, e.g. hardware virtualization, virtual memory, storage virtualization, operating system-level virtualization, etc. Clicking through to Wiktionary for "virtualization" doesn't shed any additional light. But a careful reading of Wikipedia's coverage turns up two concepts repeatedly: simulation and abstraction.

    Wikipedia does provide a definition for cloud computing: "Cloud computing is Internet-based computing, whereby shared resources, software, and information are provided to computers and other devices on demand, like the electricity grid." Not entirely adequate, but not bad.

    The fact is, virtualization and cloud computing are related but entirely different concepts. Virtualization is simulation; the request fulfillment is in terms of a simulation -- make-believe that behaves like real. Several "make-believes" can be produced from one "real" -- often without significant impact in terms of performance, and at far lesser cost, hence the attraction of virtualization. Loosely speaking, anything can be virtualized -- memory, storage, operating system services, etc. The primary objective (and hence benefit) of virtualization is to "do more with less" and save on costs.

    Cloud computing is something else entirely -- it is, literally, computing in the (Internet) cloud. Components that make up the application -- for example, application components, databases, external/real-time data feeds, etc. -- are typically dispersed across the Internet and made available in the form of services. A "cloud app" is then in effect an assembly of such cloud-based services. In turn these services may use virtualization, but that's only incidental. Internal clouds -- which are seeing in increasing prevalence -- are just that; mostly internal services, hosted internally within the organization. In that sense, internal clouds are an extension of classic SOA architecture. By "merely" dispersing these services across the internet, we begin to truly realize the benefits of SOA. Cloud computing allows us to not just "do more with less" -- it enables us to "do more," period.
    Virtualization and cloud computing make a good team, but they complement rather than supplant each other. And they are not identical.

    Important Linux Port Numbers

    21 => FTP
    22 => SSH
    23 => Telnet
    25 => SMTP Mail Transfer
    43 => WHOIS service
    53 => name server (DNS)
    80 => HTTP (Web server)
    110 => POP protocol (for email)
    995 => POP over SSL
    3306 => MysQL Server
    111 => rpcbind
    953 => rndc
    143 => IMAP Protocol (for email)
    993 => IMAP Secure
    443 => HTTP Secure 
    3128 => Squid Proxy
    3306 => MysQL Server
    3636 => Piranha
    4643 => Virtuosso Power Panel
    10000 => Webmin


    For more port numbers Click Here 
    and Click Here 

    Tuesday, August 17, 2010

    LAMP Web Server with out SSL


    This document will walk you through the installation of what is known as a “LAMP” system: Linux,Apache, MySQL and PHP. Depending on who you talk to, the P also stands for Perl or Python, but in general, it is assumed to be PHP. I run CentOS on my servers; these directions were written for CentOS/Red Hat/Fedora. I have had requests for SuSE (another RPM-based distribution) as well as Debian-based systems, so I will work on variants of these directions for those distributions in the future (donations might help speed that process up!). The main difference between the distributions is in the paths to the startup scripts. Red Hat systems used /etc/rc.d/init.d and SuSE uses /etc/init.d.
    Initial Steps
    PLEASE BE AWARE THAT A SOURCE-BASED INSTALLATION LIKE THIS ONE IS NOT NEEDED FOR A BASIC LAMP SERVER! You should only be doing a source-based installation if you need to alter settings in one or more components of the LAMP stack (e.g., you need a feature in PHP that isn’t in the default RPM). If you are just getting started with LAMP, use the binaries provided by your distribution – it is much simpler, and a lot easier to upgrade later.
    Most out-of-the-box Red Hat Linux installations will have one or more of the LAMP components installed via RPM files. I personally believe in installing things like this from source, so I get the most control over what’s compiled in, what’s left out, etc. But source code installs can wreak havoc if overlaid on top of RPM installs, as the two most likely won’t share the same directories, etc.
    If you have not yet installed your Linux OS, or just for future reference, do not choose to install Apache, PHP, or MySQL during the system installation. Then you can immediately proceed with the source-based install listed here.
    Note: to install applications from source code, you will need a C++ compiler (gcc++) installed. This is generally taken care of, but I’ve had enough queries about it that I’ve added this note to avoid getting more! You can use your distribution’s install CDs to get the proper version of the compiler. Or, if you are using an RPM based distro, you can use a site like http://www.rpmfind.net/ to locate the correct RPM version for your system. (You will obviously not be able to use/rebuild a source RPM to get the compiler installed, as you need the compiler to build the final binary RPM!) On a Fedora system, you can do this command:
    su – root
    yum install gcc gcc-c++
    Log in as root
    Because we will be installing software to directories that “regular” users don’t have write access to, and also possibly uninstalling RPM versions of some applications, we’ll log in as root. The only steps that need root access are the actual installation steps, but by doing the configure and makesteps as root, the source code will also be inaccessible to “regular” users.
    If you do not have direct access (via keyboard) to the server, PLEASE use Secure Shell (SSH) to access the server and not telnet!! Whenever you use telnet (or plain FTP for that matter), you are transmitting your username, password, and all session information in “plain text”. This means that anyone who can access a machine someplace between your PC and your server can snoop your session and get your info. Use encryption wherever possible!
    su – root
    Remove RPM Versions of the Applications
    Before we start with our source code install, we need to remove all the existing RPM files for these products. To find out what RPMs are already installed, use the RPM query command:
    rpm -qa
    in conjunction with grep to filter your results:
    rpm -qa | grep -i apache
    rpm -qa | grep -i httpd
    rpm -qa | grep -i php
    rpm -qa | grep -i mysql
    The ‘httpd’ search is in case you have Apache2 installed via RPM.
    To remove the RPMs generated by these commands, do
    rpm -e filename
    for each RPM you found in the query. If you have any content in your MySQL database already, the RPM removal step should not delete the database files. When you reinstall MySQL, you should be able to move all those files to your new MySQL data directory and have access to them all again.
    Get the Source Code for all Applications
    We want to put all our source code someplace central, so it’s not getting mixed up in someone’s home directory, etc.
    cd /usr/local/src
    One way application source code is distributed is in what are known as “tarballs.” The tarcommand is usually associated with making tape backups – tar stands for Tape ARchive. It’s also a handy way to pack up multiple files for easy distribution. Use the man tar command to learn more about how to use this very flexible tool.
    At the time of updating this, the current versions of all the components we’ll use are:
    MySQL – 4.1.22
    Apache – 1.3.37
    PHP – 4.4.6
    Please note: these are the only versions of these that I have set up myself, and verified these steps against. If you use another version of any component, especially a newer version, this HOWTO may not be accurate, and I won’t be able to provide free support under those circumstances. Paid support and assistance is always available however.
    wget http://www.php.net/distributions/php-4.4.6.tar.gz
    wget http://apache.oregonstate.edu/httpd/apache_1.3.37.tar.gz
    There may be an Apache mirror closer to you – check their mirror page for other sources. Then insert the URL you get in place of the above for the wget command.
    For MySQL, go to http://www.mysql.com/ and choose an appropriate mirror to get the newest MySQL version (v4.1.22).
    Unpack the Source Code
    tar zxf php-4.4.6.tar.gz
    tar zxf apache_1.3.37.tar.gz
    tar zxf mysql-4.1.22.tar.gz
    This should leave you with the following directories:
    /usr/local/src/php-4.4.6
    /usr/local/src/apache_1.3.37
    /usr/local/src/mysql-4.1.22
    Build and Install MySQL
    First, we create the group and user that “owns” MySQL. For security purposes, we don’t want MySQL running as root on the system. To be able to easily identify MySQL processes in top or a pslist, we’ll make a user and group named mysql:
    groupadd mysql
    useradd -g mysql -c “MySQL Server” mysql
    If you get any messages about the group or user already existing, that’s fine. The goal is just to make sure we have them on the system.
    What the useradd command is doing is creating a user mysql in the group mysql with the “name” of MySQL Server. This way when it’s showed in various user and process watching apps, you’ll be able to tell what it is right away.
    root is started (via the safe_mysqld script). Then child processes, owned by mysql are spawned from it. The parent controlling process watches the child processes and restarts them automatically if they get killed off, etc.
    –>Now we’ll change to the “working” directory where the source code is, change the file ‘ownership’ for the source tree (this prevents build issues in reported in some cases where the packager’s username was included on the source and you aren’t using the exact same name to compile with!) and start building.
    The configure command has many options you can specify. I have listed some fairly common ones; if you’d like to see others, do:
    ./configure –help | less
    to see them all.
    cd /usr/local/src/mysql-4.1.22
    chown -R root.root *
    make clean
    ./configure \
    –prefix=/usr/local/mysql \
    –localstatedir=/usr/local/mysql/data \
    –disable-maintainer-mode \
    –with-mysqld-user=mysql \
    –with-unix-socket-path=/tmp/mysql.sock \
    –without-comment \
    –without-debug \
    –without-bench
    18-Jul-2005: If you are installing MySQL 4.0.x on Fedora Core 4, there is a problem with LinuxThreads that prevents MySQL from compiling properly. Installing on Fedora Core 3 works fine though. Thanks to Kevin Spencer for bringing this to my attention. There is a workaround listed athttp://bugs.mysql.com/bug.php?id=9497. Thanks to Collin Campbell for that link. Another solution can be found at http://bugs.mysql.com/bug.php?id=2173. Thanks to Kaloyan Raev for that one.
    Now comes the long part, where the source code is actually compiled and then installed. Plan to get some coffee or take a break while this step runs. It could be 10-15 minutes or more, depending on your system’s free memory, load average, etc.
    make && make install
    Configure MySQL
    MySQL is “installed” but we have a few more steps until it’s actually “done” and ready to start. First run the script which actually sets up MySQL’s internal database (named, oddly enough, mysql).
    ./scripts/mysql_install_db
    Then we want to set the proper ownership for the MySQL directories and data files, so that only MySQL (and root) can do anything with them.
    chown -R root:mysql /usr/local/mysql
    chown -R mysql:mysql /usr/local/mysql/data
    Copy the default configuration file for the expected size of the database (small, medium, large, huge)
    cp support-files/my-medium.cnf /etc/my.cnf
    chown root:sys /etc/my.cnf
    chmod 644 /etc/my.cnf
    If you get an error message about the data directory not existing, etc., something went wrong in the mysql_install_db step above. Go back and review that; make sure you didn’t get some sort of error message when you ran it, etc.
    Now we have to tell the system where to find some of the dynamic libraries that MySQL will need to run. We use dynamic libraries instead of static to keep the memory usage of the MySQL program itself to a minimum.
    echo “/usr/local/mysql/lib/mysql” >> /etc/ld.so.conf
    ldconfig
    Now create a startup script, which enables MySQL auto-start each time your server is restarted.
    cp ./support-files/mysql.server /etc/rc.d/init.d/mysql
    chmod +x /etc/rc.d/init.d/mysql
    /sbin/chkconfig –level 3 mysql on
    Then set up symlinks for all the MySQL binaries, so they can be run from anyplace without having to include/specify long paths, etc.
    cd /usr/local/mysql/bin
    for file in *; do ln -s /usr/local/mysql/bin/$file /usr/bin/$file; done
    MySQL Security Issues
    First, we will assume that only applications on the same server will be allowed to access the database (i.e., not a program running on a physically separate server). So we’ll tell MySQL not to even listen on port 3306 for TCP connections like it does by default.
    Edit /etc/my.cnf and uncomment the
    skip-networking
    line (delete the leading #).
    Start MySQL
    First, test the linked copy of the startup script in the normal server runlevel start directory, to make sure the symlink was properly set up:
    cd ~
    /etc/rc.d/rc3.d/S90mysql start
    If you ever want to manually start or stop the MySQL server, use these commands:
    /etc/rc.d/init.d/mysql start
    /etc/rc.d/init.d/mysql stop
    Let’s “test” the install to see what version of MySQL we’re running now:
    mysqladmin version
    It should answer back with the version we’ve just installed…
    Now we’ll set a password for the MySQL root user (note that the MySQL root user is not the same as the system root user, and definitely should not have the same password as the system rootuser!).
    mysqladmin -u root password new-password
    (obviously, insert your own password in the above command instead of the “new-password” string!)
    You’re done! MySQL is now installed and running on your server. It is highly recommended that you read about MySQL security and lock down your server as much as possible. The MySQL site has info at http://www.mysql.com/doc/en/Privilege_system.html.
    Test MySQL
    To run a quick test, use the command line program mysql:
    mysql -u root -p
    and enter your new root user password when prompted. You will then see the MySQL prompt:
    mysql>
    First, while we’re in here, we’ll take care of another security issue and delete the sample databasetest and all default accounts except for the MySQL root user. Enter each of these lines at themysql> prompt:
    drop database test;
    use mysql;
    delete from db;
    delete from user where not (host=”localhost” and user=”root”);
    flush privileges;
    As another security measure, I like to change the MySQL administrator account name from root to something harder to guess. This will make it that much harder for someone who gains shell access to your server to take control of MySQL.
    MAKE SURE YOU REMEMBER THIS NEW NAME, AND USE IT WHEREVER
    YOU SEE “root” IN OTHER DIRECTIONS, WEBSITES, ETC.
    ONCE YOU DO THIS STEP, THE USERNAME “root” WILL CEASE TO
    EXIST IN YOUR MYSQL CONFIGURATION!
    update user set user=”sqladmin” where user=”root”;
    flush privileges;
    Now, on with the “standard” testing… First, create a new database:
    create database foo;
    You should see the result:
    Query OK, 1 row affected (0.04 sec)
    mysql>
    Delete the database:
    drop database foo;
    You should see the result:
    Query OK, 0 rows affected (0.06 sec)
    mysql>
    To exit from mysql enter \q:
    \q
    Build and Install Apache (with DSO support)
    The advantage to building Apache with support for dynamically loaded modules is that in the future, you can add functionality to your webserver by just compiling and installing modules, and restarting the webserver. If the features were compiled into Apache, you would need to rebuild Apache from scratch every time you wanted to add or update a module (like PHP). Your Apache binary is also smaller, which means more efficient memory usage.
    The downside to dynamic modules is a slight performance hit compared to having the modules compiled in.
    cd /usr/local/src/apache_1.3.37
    make clean
    ./configure \
    –prefix=/usr/local/apache \
    –enable-shared=max \
    –enable-module=rewrite \
    –enable-module=so
    make && make install
    Build and Install PHP
    This section has only been tested with PHP v4.x. If you are trying to build PHP 5.x, I do not have experience with this yet, and do not provide free support for you to get it working. Please note that there are many options which can be selected when compiling PHP. Some will have library dependencies, meaning certain software may need to be already installed on your server before you start building PHP. You can use the command
    ./configure –help | less
    once you change into the PHP source directory. This will show you a list of all possible configuration switches.
    cd /usr/local/src/php-4.4.6
    ./configure \
    –with-apxs=/usr/local/apache/bin/apxs \
    –disable-debug \
    –enable-ftp \
    –enable-inline-optimization \
    –enable-magic-quotes \
    –enable-mbstring \
    –enable-mm=shared \
    –enable-safe-mode \
    –enable-track-vars \
    –enable-trans-sid \
    –enable-wddx=shared \
    –enable-xml \
    –with-dom \
    –with-gd \
    –with-gettext \
    –with-mysql=/usr/local/mysql \
    –with-regex=system \
    –with-xml \
    –with-zlib-dir=/usr/lib
    make && make install
    cp php.ini-dist /usr/local/lib/php.ini
    I like to keep my config files all together in /etc. I set up a symbolic link like this:
    ln -s /usr/local/lib/php.ini /etc/php.ini
    Then I can just open /etc/php.ini in my editor to make changes.
    Recommended reading on securing your PHP installation
    Edit the Apache Configuration File (httpd.conf)
    I like to keep all my configuration files together in /etc, so I set up a symbolic link from the actual location to /etc:
    ln -s /usr/local/apache/conf/httpd.conf /etc/httpd.conf
    Now open /etc/httpd.conf in your favorite text editor, and set all the basic Apache options in accordance with the official Apache instructions (beyond the scope of this HOWTO).
    To ensure your PHP files are properly interpreted, and not just downloaded as text files, remove the # at the beginning of the lines which read:
    #AddType application/x-httpd-php .php
    #AddType application/x-httpd-php-source .phps
    If the AddType lines above don’t exist, manually enter them (without the leading # of course) after the line
    AddType application/x-tar .tgz
    or anyplace within the  section of httpd.conf.
    If you wish to use other/additional extensions/filetypes for your PHP scripts instead of just .php, add them to the AddType directive:
    AddType application/x-httpd-php .php .foo
    AddType application/x-httpd-php-source .phps .phtmls
    An example: if you wanted every single HTML page to be parsed and processed like a PHP script, just add .htm and .html:
    AddType application/x-httpd-php .php .htm .html
    There will be a bit of a performance loss if every single HTML page is being checked for PHP code even if it doesn’t contain any. But if you want to use PHP but be “stealthy” about it, you can use this trick.
    Add index.php to the list of valid Directory Index files so that your “default page” in a directory can be named index.php.

    DirectoryIndex index.php index.htm index.html
    You can add anything else you want here too. If you want foobar.baz to be a valid directory index page, just add the .baz filetype to the AddType line, and add foobar.baz to the DirectoryIndex line.
    Start Apache
    We want to set Apache up with a normal start/stop script in /etc/rc.d/init.d so it can be auto-started and controlled like other system daemons. Set up a symbolic link for the apachectl utility (installed automatically as part of Apache):
    ln -s /usr/local/apache/bin/apachectl /etc/rc.d/init.d/apache
    Then set up auto-start for runlevel 3 (where the server will go by default):
    ln -s /etc/rc.d/init.d/apache /etc/rc.d/rc3.d/S90apache
    Then start the daemon:
    /etc/rc.d/init.d/apache start
    You can check that it’s running properly by doing:
    ps -ef
    and look for the httpd processes.