lagg freebsd

2010-04-11

I recently configured a NFS/Samba server with FreeBSD’s Link Aggregation protocol. Here is how I set it up.

FreeBSD Configuration

/boot/loader.conf

I recommend adding the if_lagg_load=YES and kern.hz=“2000” to /boot/loader.conf.

The OS will automatically load the lagg kernel module when your network configuration loads, I prefer to explicitly set it to load.

ispfw_load="YES"
kern.hz="2000"
aio_load="YES"
hw.igb.rxd=4096
hw.igb.txd=4096
if_lagg_load="YES"

Since I am using the igb ethernet device (Intel 82575 and 82576 chipsets), I also set the max number of send and receive descriptors from the default from 256 to the maximum 4096. Give some thought to this step, increasing this will allocate more memory per interface. Since there are four in use in this setup, that is an order of magnitude higher than the stock configuration.

/etc/rc.conf

ifconfig_igb0="UP polling"
ifconfig_igb1="UP polling"
ifconfig_igb2="UP polling"
ifconfig_igb3="UP polling"
ifconfig_lagg0="create laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3 128.115.132.165 netmask 255.255.255.0"
Jumbo Frames

Normally, I set a MTU of 9194 for my igb/82575 ethernet controllers.

Setting an arbitrary MTU size above the default 1500 can cause the unexpected, and decrease the stability in your environment. This configuration is using the Intel 82575 Quad Port 1000 VT adapter, which has the maximum MTU of 9194. Not all ethernet controllers support the same MTU sizes, for instance, Broadcom chipsets have a max MTU of 9022. Also, verify your switch can support Jumbo Frames, and have the ports in use set to the appropriate MTU.

Some Notes from Intel’s 8257[5-6] README:

  • Only enable Jumbo Frames if your network infrastructure supports them.
  • To enable Jumbo Frames, increase the MTU size on the interface beyond
  • The Jumbo Frames setting on the switch must be set to at least 22 bytes larger than that of the MTU.
  • The maximum MTU setting for Jumbo Frames is 9216. This value coincides with the maximum Jumbo Frames size of 9234 bytes.
  • Using Jumbo Frames at 10 or 100 Mbps may result in poor performance or loss of link.

Since the Cisco hardware used can be set to a max MTU of 9216, our igb interfaces can be set to 9194.

Of course, the lagg interface wont let me set a mtu higher than 1500. If anyone has additional information on this, that would be great to know.

Results

[root@laggy ~]> ifconfig lagg0

lagg0: flags=8843 metric 0 mtu 9194
	options=1bb
	ether 00:26:b9:62:ae:c8
	inet 128.115.132.165 netmask 0xffffff00 broadcast 128.115.132.255
	media: Ethernet autoselect
	status: active
	laggproto lacp
	laggport: igb3 flags=1c
	laggport: igb2 flags=1c
	laggport: igb1 flags=1c
	laggport: igb0 flags=1c

[root@laggy ~]> netstat -I lagg0 -w 1

            input        (lagg0)           output
   packets  errs      bytes    packets  errs      bytes colls
     87581     0  127400293          0     0          0     0
     85126     0  123891200          0     0          0     0
     84926     0  124237023          0     0          0     0

[root@laggy ~]> netstat -I igb0 -w 1

            input         (igb0)           output
   packets  errs      bytes    packets  errs      bytes colls
     38118     0   54749173          0     0          0     0
     35833     0   51498282          0     0          0     0

[root@laggy ~]> netstat -I igb1 -w 1

            input         (igb1)           output
   packets  errs      bytes    packets  errs      bytes colls
     12889     0   18315538          0     0          0     0
     16303     0   23159114          0     0          0     0
     27672     0   39275792          0     0          0     0

[root@laggy ~]> netstat -I igb2 -w 1

            input         (igb2)           output
   packets  errs      bytes    packets  errs      bytes colls
     23709     0   35778378          0     0          0     0
     24445     0   36901194          0     0          0     0

[root@laggy ~]> netstat -I igb3 -w 1

            input         (igb3)           output
   packets  errs      bytes    packets  errs      bytes colls
        11     0       1535          0     0          0     0
         1     0         60          0     0          0     0

Cisco Configuration

interface GigabitEthernet 8/41
 description (L5D13) laggy
 switchport                           # Required for L2 Etherchannel
 switchport access vlan 132           # VLAN assignment (optional)
 spanning-tree portfast               # Recommended
 channel-group 4 mode active          # Required assign channel # and mode
                                         see table below
 channel-protocol lacp                # Required assign lacp or pagp
 no shutdown

Verify the interface port-channel is not shutdown and add a description

interface Port-channel 4
 description Test Channel 2/22/10 laggy
 switchport                           
 switchport access vlan 101           
 switchport trunk encapsulation dot1q
 no shutdown

This command will cause load balancing to occur for the source and destination IP addresses and applied to all Etherchannels on the switch. Other load balancing options exists.

port-channel load-balance src-dst-ip

Use the following show commands for verifying the condition of the Etherchannel

show interfaces port-channel ‘channel number’ show etherchannel port-channel show etherchannel summary show Etherchannel load-balance

Switch# show interfaces port-channel 4
Port-channel4 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0017.9499.94ac (bia 0017.9499.94ae)
  Description: TEST CHANNEL 2/22/10 laggy
  MTU 9214 bytes, BW 4000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is on
  Members in this channel: Gi8/41 Gi8/43 Gi8/45 Gi8/47

Switch#show etherchannel port-channel
                Channel-group listing:
                -----------------------
Group: 4
----------
                Port-channels in the group:
                ----------------------
Port-channel: Po4    (Primary Aggregator)

------------
Age of the Port-channel   = 0d:05h:25m:33s
Logical slot/port   = 14/1          Number of ports = 4
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP

Ports in the Port-channel: 

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  1     11     Gi8/41   Active    2
  2     22     Gi8/43   Active    2
  3     44     Gi8/45   Active    2
  0     88     Gi8/47   Active    2

Time since last port bundled:    0d:04h:47m:26s    Gi8/41
Time since last port Un-bundled: 0d:05h:22m:14s    Gi8/47

Switch#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
4      Po4(SU)         LACP      Gi8/41(P)  Gi8/43(P)  Gi8/45(P)  Gi8/47(P)
Switch#show Etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-dst-ip
        mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address
  MPLS: Label or IP