Monday, July 26, 2010

Ethernet bonding

Ethernet bonding is a way to combine multiple ethernet channel into a single one. If, for example, you have 2 ethernet cards of 1 Gb/s speed each, you can combine them to have one virtual card of 2 Gb/s speed.

Suppose you have a cluster of several nodes, one of them is the master node and has 3 ethernet cards, one for the network 192.168.1.0/24 (file ifcfg-eth2) and the two others (file ifcfg-eth0 and ifcfg-eth1) bonded together (file ifcfg-bond0) for the network 192.168.7.0/24.

The ethernet card in the network should be configured with file ifcfg-eth2 as follows:
DEVICE=eth2
BOOTPROTO=none
HWADDR=00:05:48:A8:F2:H6
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
NETMASK=255.255.255.0
IPADDR=192.168.1.13
GATEWAY=192.168.1.254

Juste create in /etc/sysconfig/network-scripts, the file ifcfg-bond0 containing the following lines:
DEVICE=bond0
BOOTPROTO=none
BROADCAST=192.168.7.255
IPADDR=192.168.7.20
NETMASK=255.255.255.0
NETWORK=192.168.7.0
GATEWAY=192.168.1.254
ONBOOT=yes
USERCTL=no
TYPE=Ethernet

which is what you would have for a single ethernet card in a file called ifcfg-eth0. On the master node, you have to specify the gateway 192.168.1.254.

On the other nodes, you will specify the gateway to be 192.168.7.20 which is the IP address of the master node in the network 192.168.7.0/24:

DEVICE=bond0
BOOTPROTO=none
BROADCAST=192.168.7.255
IPADDR=192.168.7.22
NETMASK=255.255.255.0
NETWORK=192.168.7.0
ONBOOT=yes
USERCTL=no
TYPE=Ethernet
GATEWAY=192.168.7.20

You must also create the files ifcfg-eth0 and ifcfg-eth1, one file for each card in which you will put the following:

In ifcfg-eth0:
DEVICE=eth0
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=no
TYPE=Ethernet

In ifcfg-eth1:
DEVICE=eth1
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=no
TYPE=Ethernet

This way, both cards know who is the master.

On the master node, ifconfig should give:
# ifconfig 
bond0 Link encap:Ethernet HWaddr 00:02:55:C7:ED:73 
inet addr:192.168.7.20 Bcast:192.168.7.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:27451 errors:0 dropped:0 overruns:0 frame:0
TX packets:13703 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 
RX bytes:2542289 (2.4 MiB) TX bytes:1328584 (1.2 MiB)

eth0 Link encap:Ethernet HWaddr 00:02:55:C7:ED:73 
inet6 addr: fe80::202:55ff:fec7:ed73/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:13684 errors:0 dropped:0 overruns:0 frame:0
TX packets:6854 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:1239023 (1.1 MiB) TX bytes:663919 (648.3 KiB)

eth1 Link encap:Ethernet HWaddr 00:02:55:C7:ED:73 
inet6 addr: fe80::202:55ff:fec7:ed73/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:13767 errors:0 dropped:0 overruns:0 frame:0
TX packets:6849 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:1303266 (1.2 MiB) TX bytes:664665 (649.0 KiB)

eth2 Link encap:Ethernet HWaddr 00:03:47:B1:F9:E6 
inet addr:192.168.1.13 Bcast:192.168.1.15 Mask:255.255.255.240
inet6 addr: fe80::203:47ff:feb1:f9e6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:311014 errors:0 dropped:0 overruns:0 frame:0
TX packets:443837 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:110699104 (105.5 MiB) TX bytes:434496718 (414.3 MiB)
Memory:effc0000-effe0000 

lo Link encap:Local Loopback 
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:185354 errors:0 dropped:0 overruns:0 frame:0
TX packets:185354 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 
RX bytes:421703302 (402.1 MiB) TX bytes:421703302 (402.1 MiB)

On the slave nodes, ifconfig should give
# ifconfig 
bond0 Link encap:Ethernet HWaddr 00:02:55:C7:ED:EE 
inet addr:192.168.7.22 Bcast:192.168.7.255 Mask:255.255.255.0
inet6 addr: fe80::202:55ff:fec7:edee/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:14694 errors:0 dropped:0 overruns:0 frame:0
TX packets:3973 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 
RX bytes:1138254 (1.0 MiB) TX bytes:359734 (351.3 KiB)

eth0 Link encap:Ethernet HWaddr 00:02:55:C7:ED:EE 
inet6 addr: fe80::202:55ff:fec7:edee/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:7380 errors:0 dropped:0 overruns:0 frame:0
TX packets:1991 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:579813 (566.2 KiB) TX bytes:178136 (173.9 KiB)
Interrupt:193 Base address:0x6000 

eth1 Link encap:Ethernet HWaddr 00:02:55:C7:ED:EE 
inet6 addr: fe80::202:55ff:fec7:edee/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:7322 errors:0 dropped:0 overruns:0 frame:0
TX packets:1994 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:558969 (545.8 KiB) TX bytes:183558 (179.2 KiB)
Interrupt:201 Base address:0xa000 

lo Link encap:Local Loopback 
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:152 errors:0 dropped:0 overruns:0 frame:0
TX packets:152 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 
RX bytes:10980 (10.7 KiB) TX bytes:10980 (10.7 KiB)