lagg freebsd
2010-04-11
I recently configured a NFS/Samba server with FreeBSD’s Link Aggregation protocol. Here is how I set it up.
FreeBSD Configuration
/boot/loader.conf
I recommend adding the if_lagg_load=YES and kern.hz=“2000” to /boot/loader.conf.
The OS will automatically load the lagg kernel module when your network configuration loads, I prefer to explicitly set it to load.
ispfw_load="YES"
kern.hz="2000"
aio_load="YES"
hw.igb.rxd=4096
hw.igb.txd=4096
if_lagg_load="YES"
Since I am using the igb ethernet device (Intel 82575 and 82576 chipsets), I also set the max number of send and receive descriptors from the default from 256 to the maximum 4096. Give some thought to this step, increasing this will allocate more memory per interface. Since there are four in use in this setup, that is an order of magnitude higher than the stock configuration.
/etc/rc.conf
ifconfig_igb0="UP polling"
ifconfig_igb1="UP polling"
ifconfig_igb2="UP polling"
ifconfig_igb3="UP polling"
ifconfig_lagg0="create laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3 128.115.132.165 netmask 255.255.255.0"
Jumbo Frames
Normally, I set a MTU of 9194 for my igb/82575 ethernet controllers.
Setting an arbitrary MTU size above the default 1500 can cause the unexpected, and decrease the stability in your environment. This configuration is using the Intel 82575 Quad Port 1000 VT adapter, which has the maximum MTU of 9194. Not all ethernet controllers support the same MTU sizes, for instance, Broadcom chipsets have a max MTU of 9022. Also, verify your switch can support Jumbo Frames, and have the ports in use set to the appropriate MTU.
Some Notes from Intel’s 8257[5-6] README:
- Only enable Jumbo Frames if your network infrastructure supports them.
- To enable Jumbo Frames, increase the MTU size on the interface beyond
- The Jumbo Frames setting on the switch must be set to at least 22 bytes larger than that of the MTU.
- The maximum MTU setting for Jumbo Frames is 9216. This value coincides with the maximum Jumbo Frames size of 9234 bytes.
- Using Jumbo Frames at 10 or 100 Mbps may result in poor performance or loss of link.
Since the Cisco hardware used can be set to a max MTU of 9216, our igb interfaces can be set to 9194.
Of course, the lagg interface wont let me set a mtu higher than 1500. If anyone has additional information on this, that would be great to know.
Results
[root@laggy ~]> ifconfig lagg0
lagg0: flags=8843 metric 0 mtu 9194
options=1bb
ether 00:26:b9:62:ae:c8
inet 128.115.132.165 netmask 0xffffff00 broadcast 128.115.132.255
media: Ethernet autoselect
status: active
laggproto lacp
laggport: igb3 flags=1c
laggport: igb2 flags=1c
laggport: igb1 flags=1c
laggport: igb0 flags=1c
[root@laggy ~]> netstat -I lagg0 -w 1
input (lagg0) output
packets errs bytes packets errs bytes colls
87581 0 127400293 0 0 0 0
85126 0 123891200 0 0 0 0
84926 0 124237023 0 0 0 0
[root@laggy ~]> netstat -I igb0 -w 1
input (igb0) output
packets errs bytes packets errs bytes colls
38118 0 54749173 0 0 0 0
35833 0 51498282 0 0 0 0
[root@laggy ~]> netstat -I igb1 -w 1
input (igb1) output
packets errs bytes packets errs bytes colls
12889 0 18315538 0 0 0 0
16303 0 23159114 0 0 0 0
27672 0 39275792 0 0 0 0
[root@laggy ~]> netstat -I igb2 -w 1
input (igb2) output
packets errs bytes packets errs bytes colls
23709 0 35778378 0 0 0 0
24445 0 36901194 0 0 0 0
[root@laggy ~]> netstat -I igb3 -w 1
input (igb3) output
packets errs bytes packets errs bytes colls
11 0 1535 0 0 0 0
1 0 60 0 0 0 0
Cisco Configuration
interface GigabitEthernet 8/41
description (L5D13) laggy
switchport # Required for L2 Etherchannel
switchport access vlan 132 # VLAN assignment (optional)
spanning-tree portfast # Recommended
channel-group 4 mode active # Required assign channel # and mode
see table below
channel-protocol lacp # Required assign lacp or pagp
no shutdown
Verify the interface port-channel is not shutdown and add a description
interface Port-channel 4
description Test Channel 2/22/10 laggy
switchport
switchport access vlan 101
switchport trunk encapsulation dot1q
no shutdown
This command will cause load balancing to occur for the source and destination IP addresses and applied to all Etherchannels on the switch. Other load balancing options exists.
port-channel load-balance src-dst-ip
Use the following show commands for verifying the condition of the Etherchannel
show interfaces port-channel ‘channel number’ show etherchannel port-channel show etherchannel summary show Etherchannel load-balance
Switch# show interfaces port-channel 4
Port-channel4 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0017.9499.94ac (bia 0017.9499.94ae)
Description: TEST CHANNEL 2/22/10 laggy
MTU 9214 bytes, BW 4000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s
input flow-control is off, output flow-control is on
Members in this channel: Gi8/41 Gi8/43 Gi8/45 Gi8/47
Switch#show etherchannel port-channel
Channel-group listing:
-----------------------
Group: 4
----------
Port-channels in the group:
----------------------
Port-channel: Po4 (Primary Aggregator)
------------
Age of the Port-channel = 0d:05h:25m:33s
Logical slot/port = 14/1 Number of ports = 4
Port state = Port-channel Ag-Inuse
Protocol = LACP
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
1 11 Gi8/41 Active 2
2 22 Gi8/43 Active 2
3 44 Gi8/45 Active 2
0 88 Gi8/47 Active 2
Time since last port bundled: 0d:04h:47m:26s Gi8/41
Time since last port Un-bundled: 0d:05h:22m:14s Gi8/47
Switch#show etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
Number of channel-groups in use: 1
Number of aggregators: 1
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
4 Po4(SU) LACP Gi8/41(P) Gi8/43(P) Gi8/45(P) Gi8/47(P)
Switch#show Etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP