In conventional IT systems, storage devices are connected to servers bymeans of SCSI cables. The idea behind storage networks is that these SCSI cables arereplaced by a network, which is installed in addition to the existing LAN. Server and storage devices can exchange data over this new network using the SCSI protocol
SAN Components
1. Server Hardware – The actual machines which are configured to use the data from the storage devices. The major vendors for Server hardware are ORACLE ( formerly SUN Microsystems) , IBM, HP, Fujitsu ..etc.
2. Storage Hardware – The actual Disk arrays from smaller size to enterprise size. The major Vendors for Storage Hardware are EMC, Oracle (formerly knows as SUN Microsystems), IBM, HP , DELL …etc.
3. HBA – Host Bus adapters acts as initiator in SCSI storage subsystem and main function is converting SCSI commands into Fiber Channel format and establish connection from server side to the SAN. You can compare HBA cards with Ethernet adapters in our regular networking. Major vendors are SUN, Emulex, Qlogic, JNI ..etc.
4. SAN Switches – These are similar to Ethernet switches , but do switching for fiber networks. Major vendors are Brocade, Cisco ….etc.
SAN Topology
In this exercise, we will be looking at various phases of SAN configuration and practically looking at the configuration of two Solaris machines with Emulex HBAs connected to a EMC Clarion Storage.
From the diagram you can notice two Solaris servers named “gurkul1 and gurkul2″, two SAN switches named “SN1 and SN2″ and EMC device with two Storage arrays “array1 and array2″.
The SAN Switch and the Storage array together comprises a fabric in the SAN.
PORTS
In regular networking we do have ports in Ethernet adapters to interconnect the networking devices using network cables, in the similar way in Storage Area Networking we do have different types of ports which are used to interconnect Servers , Switches and Storage. Below are the common type of ports that we commonly discuss while talking about SAN.
- ‘N’ port: Node ports used for connecting peripheral storage devices to switch fabric or for point to point configurations
- ‘F’ port: Fabric ports reside on switches and allow connection of storage peripherals (‘N’ port devices)
- ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches
- ‘G’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port. Its also capable of acting in an ‘L’ port capacity— known as a ‘G’ port
- ‘L’ port: Loop ports are used in arbitrated loop configurations to build storage peripheral networks without FC switches. These ports often also have ‘N’ port capabilities and are called ‘NL’ ports
Important point to remember here is End to end connection managed by N ports, and Switching and addressing handled by fabric Ports.
SAN Device Identification
World-wide name (WWN): Similar to an Ethernet address in the traditional networking world, this is a unique name assigned by the manufacturer to the HBA. This name is then used to grant and restrict access to other components of the SAN. There are two basic forms of WWN.
1. World Wide Node Name (WWNN)—Assigned to Fibre Channel node device by vendor
2. World Wide Port Name (WWPN) —Assigned to Fibre Channel Host BusAdapter port by vendor
Normally that the HBA vendor will provide a tool that allows the system administrator to query the wwn of the HBA (e.g. Emulex supplies the lputil application). The world-wide name of the symmetrix fibre adapters (FA) can be obtained from EMC, and inaddition to the WWPN for EMC we should also note the SCSI target ID assigned for specific FA in order to make configuration in Solaris Side.
For host bus adapters, the world wide name is typically displayed in the /var/adm/messages file after the fibre card and software driver have been installed. An example of for an Emulex PCI fibre HBA is as follows. In this case, the relevant value the WWPN, the World-Wide Port Name.
May 18 09:26:28 gurkul1 lpfc: [ID 242157 kern.info] NOTICE:
lpfc0:031:Link Up Event received Data: 1 1 0 0
May 18 09:26:31 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc0: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:31 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc0: WWPN:11:11:11:11:11:11:11:01
WWNN:20:22:22:22:22:22:22:22 DID 0×210913
May 18 09:26:31 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc0: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:31 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc0: WWPN:11:11:11:11:11:11:11:01
WWNN:20:22:22:22:22:22:22:22 DID 0×210913
May 18 09:26:31 gurkul1 lpfc: [ID 494464 kern.info] NOTICE:
lpfc1:031:Link Up Event received Data: 1 1 0 0
May 18 09:26:34 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc1: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:34 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc1: WWPN:11:11:1111:11:11:11:02
WWNN:20:22:22:22:22:22:22:02 DID 0×210913
lpfc1:031:Link Up Event received Data: 1 1 0 0
May 18 09:26:34 gurkul1 lpfc: [ID 129691 kern.notice] NOTICE:
lpfc1: Firmware Rev 3.20 (D2D3.20X3)
May 18 09:26:34 gurkul1 lpfc: [ID 664688 kern.notice] NOTICE:
lpfc1: WWPN:11:11:1111:11:11:11:02
WWNN:20:22:22:22:22:22:22:02 DID 0×210913
For the purpose of our configuration example I am adding the list of WWPNs numbers that we are going to use here ( we should collect this information before configuring the Server/Storage)
|
SAN Connections
The below diagram illustrates several principals common to most san topologies:
- There are two separate switched (“SN1″ and “SN2″). Each switch therefore comprises an independent “fabric”.
- The array provides multiple I/O paths to every disk device. In the diagram beside, for example, the disk device “Disk01″ is accessible via both FA1A and FA1B. Depending on the array, the two paths may be simultaneously active (active/active) or only one path may be valid at a time (active/passive).
- Each host has a connection into each fabric. Host-based software load-balances the traffic between the two links if the array supports active/active disk access. Furthermore, the host should adjust to a single link failure by rerouting all traffic along the surviving path.
Storage PATH for Gurkul1 Server
Take a moment to examine the diagram below and consider the host “gurkul1″. Assuming that it requires highly-available access to disk Disk02 in Array2 , note that there are two separate paths to that device.
Path1 : HBA1–> SN1 (P1) –> SN1 (P5) –> FA1B –> Array2 –> Disk02
Path2 : HBA2–> SN2 (P1) –> SN2 (P5) –> FA2B –> Array2 –> Disk02
Storage PATH for Gurkul2 Server
Similar way to Gurkul1 now we can examine the diagram beside to identify the storage paths available for ”gurkul2″. Assuming that it requires highly-available access to disk Disk01 in Array1 . And again here we have total separate paths to that device.
Path1 : HBA1 –> SN1 (P3) –> SN1 (P4) –> FA2A –> Array1 –> Disk01
Path2 : HBA2 –> SN2 (P3) –> SN2 (P4) –> FA1A –> Array1 –> Disk01
Target configuration that we want to configure
- For gurkul1: need access to the LUN02 (Disk02 ) to LUN05 (Disk05) from the storage array2, through FA1B and FA2B
- For gurkul2: need access to the LUN01 (Disk01) to LUN04( Disk04) from the storage array1, through FA1A and FA2A
- And both servers require access to disk00 – lun00 on each adapter they are connected
Solaris 8/9 host side configuration
Typically, there are two configuration files that need to be updated once the vendor’s hba software has been installed. The hba driver’s configuration file typically resides in the /kernel/drv directory, and must be updated to support persistent binding and any other configuration requirements specified by the array vendor. Secondly, the Solaris “sd” driver configuration file sd.conf must be updated to tell the operating system to scan for more than the default list of scsi disk devices. The examples below describe the process for configuring Emulex cards in to support an EMC Symmetrix array.
1. Configuring /kernel/drv/lpfc.conf
A. To configure Gurkul1 Server with below storage paths
Path1 : HBA1 (lpfc0) –> SN1 (P1) –> SN1 (P5) –> FA1B ( target 12 )–> Array2 –> Disk02
Path2 : HBA2 (lpfc1) –> SN2 (P1) –> SN2 (P5) –> FA2B ( target 22) –> Array2 –> Disk02
Path2 : HBA2 (lpfc1) –> SN2 (P1) –> SN2 (P5) –> FA2B ( target 22) –> Array2 –> Disk02
We Have to update below entries in /kernel/drv/lpfc.conf
fcp-bind-WWPN=”555555555555551B:lpfc0t12″,
“555555555555552B:lpfc1t22″;
“555555555555552B:lpfc1t22″;
B.To configure Gurkul2 Server with below storage paths
Path1 : HBA1(lpfc0) –> SN1 (P3) –> SN1 (P4) –> FA2A ( target 11) –> Array1 –> Disk01
Path2 : HBA2 (lpfc1) –> SN2 (P3) –> SN2 (P4) –> FA1A (target 21) –> Array1 –> Disk01
Path2 : HBA2 (lpfc1) –> SN2 (P3) –> SN2 (P4) –> FA1A (target 21) –> Array1 –> Disk01
We have to update below entries in /kernel/drv/lpfc.conf
fcp-bind-WWPN=”555555555555551A:lpfc0t11″,
“555555555555552A:lpfc1t21″;
“555555555555552A:lpfc1t21″;
2. Configuring /kernel/drv/sd.conf
By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the/kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWPN of the fiber array adapter is arbitrary. In our case, we’ve assigned scsi targets 11, 12, 21, and 22 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:
A. Gurkul1 Server:
By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the/kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWPN of the fiber array adapter is arbitrary. In our case, we’ve assigned scsi targets 11, 12, 21, and 22 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:
A. Gurkul1 Server:
# Entries added for host gurkul1 to “see” lun ( 0,2,3,4,5 ) on FA1B and FA2B with target 12 and 22
# FA1B = 555555555555551B
name=”sd” target=12 lun=0 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=2 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=3 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=4 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=5 hba=”lpfc0″ wwn=”555555555555551B”;
# FA1B = 555555555555551B
name=”sd” target=12 lun=0 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=2 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=3 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=4 hba=”lpfc0″ wwn=”555555555555551B”;
name=”sd” target=12 lun=5 hba=”lpfc0″ wwn=”555555555555551B”;
# FA2Bb = 555555555555552B
name=”sd” target=22 lun=0 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=2 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=3 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=4 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=5 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=0 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=2 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=3 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=4 hba=”lpfc1″ wwn=”555555555555552B”;
name=”sd” target=22 lun=5 hba=”lpfc1″ wwn=”555555555555552B”;
B. Gurkul2 Server:
# Entries added for host gurkul1 to “see” lun(0,1,2,3,4) on FA1A and FA2A with target 11 and 21
# FA1B = 555555555555551A
name=”sd” target=12 lun=0 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=1 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=2 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=3 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=4 hba=”lpfc0″ wwn=”555555555555551A”;
# FA1B = 555555555555551A
name=”sd” target=12 lun=0 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=1 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=2 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=3 hba=”lpfc0″ wwn=”555555555555551A”;
name=”sd” target=12 lun=4 hba=”lpfc0″ wwn=”555555555555551A”;
# FA2Bb = 555555555555552A
name=”sd” target=22 lun=0 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=1 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=2 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=3 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=4 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=0 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=1 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=2 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=3 hba=”lpfc1″ wwn=”555555555555552A”;
name=”sd” target=22 lun=4 hba=”lpfc1″ wwn=”555555555555552A”;
3. Update the /etc/system file as per EMC’s requirements for the Symmetrix.
set sd:sd_max_throttle=20set scsi_options = 0x7F8
* Powerpath? Enter
* No sd:sd_io_time=0×78
* Yes sd:sd_io_time=0x3C
set sd:sd_io_time=0x3C
4. Perform the Final Reconfiguration Reboot
Perform a reconfiguration reboot (e.g. “reboot — -r”) on all three servers.
And after the reboot see the outout for
And after the reboot see the outout for
#format
You should see the desired disks. Put a Sun label on them via the “format” command and the configuration is complete.
