Kea DHCP and Raspberry Pi's

2020-01-30

Introduction

I started a little home infrastructure project a year. It was mostly because I really wanted to check out ISC’s new DHCP server, kea. Primarily because it is API driven, and the configuration files are in JSON.

At work, I am on a team that is in charge of a few thousand servers all colocated in 4 datacenters across Chicago. For the longest time we have managed to keep those servers auto-registering and pxe booting with 2 VM’s running CentOS and ISC’s DHCPD server.

It gets better! We have moved away from CentOS. Since New Relic is a huge docker shop, we use CoreOS on bare metal. CoreOS is a provision once read-only kind of system. Well, CentOS was managed by puppet, and these two VM’s are literally some of the last CentOS hosts we have.

This honestly has kept me up at night.

Environment

First off, despite what I do at new relic, home.michaelc.dev is a FreeBSD shop. There are no containers here, just bare metal and jails.

I picked up two Raspberry Pi 3B’s for the cheap. I did want a nice load-balanced setup for a new Kea dhcp environemnt.

These Pi’s are running FreeBSD 12. I have Salt installed on them, and I have a simple salt state that at least gets the required packages up and running: 

base:
  '*':
    - salt.minion
    - cert
    - ssl
    - resolver
  'os:(FreeBSD|Arch|Debian|RedHat|Fedora|Ubuntu)':
    - match: grain_pcre
    - basepkgs
  'ca':
    - ca
  'infra-*':
    - ldap.server
    - chrony
    - dns
    - kea
    - data.pkg
  'plexer*':
    - data.safekeg
kea/init.sls 
kea:
   pkg:
      - installed
      - fromrepo: pkg
   service:
      - name: kea
      - running
      - require:
         - file: kea.rcng
         - file: kea-dhcp4.conf

kea-dhcp4.conf:
   file.managed:
      - name: {{ pillar.etc_prefix }}/kea/kea-dhcp4.conf
      - template: jinja
      - source: salt://kea/kea-dhcp4.conf.jinja
      - require:
         - pkg: kea

{% if grains.os == "FreeBSD" %}
kea.rcng:
   file.managed:
      - name: /etc/rc.conf.d/kea
      - source: salt://kea/kea.rcng
      - require:
         - pkg: kea
{% endif %}

so yeah, these two pi’s are doing a bit more than just DHCP. They are time servers (chronyd), DNS servers (I ultimately chose bind9. I tried a few others, but I was satisfied with BIND overall), and finally LDAP servers.

FreeBSD 12 images can be found here: https://download.freebsd.org/ftp/releases/arm64/aarch64/ISO-IMAGES/12.1/

First big tip I have is: Build your own packages

I have written about using Poudriere before. The only new update to cross-build packages, you need to setup a new jail with the aarch64 architecture (and have qemu installed)

JAILNAME  VERSION         ARCH          METHOD TIMESTAMP           PATH
12aarch64 12.0-RELEASE    arm64.aarch64 ftp    2019-03-03 12:27:56 /data/poudriere/jails/12aarch64
12amd64   12.0-RELEASE-p3 amd64         ftp    2019-03-03 00:51:12 /data/poudriere/jails/12amd64

With packages built, keeping both raspberry pi’s packages in sync is a lot more managable. What REALLY sucks is cross-building llvm and gcc. They can take days to compile.

Kea - ISC’s Next Gen DHCP Server

It is important to keep in mind, that Kea has a subscription model that comes with premium and subscriber only hooks.

Kea Free hooks

  • High Availablity
  • Lease Commands
  • Stat Commands

Kea premium hooks

  • Flexible Identifier
  • Host Commands
  • Forensic Logging

Kea subscriber-only hooks (+ Premium Hooks)

  • RADIUS integration
  • Host Cache
  • Subnet Commands
  • Client Classification
  • Configuration Backend

My home environment does not need these and I won’t go into them. Work environment did, and the subnet-cmd hook was well worth it, and we’re still working on the Configuration Backend but we found a bug

The setup is pretty simple. Both Pi’s I have (infra-1 and infra-2) are running Kea. Both have the HA hook loaded, and for ease of transition, I dumped the leases on my Ubiquiti router (which was serving as my home’s DHCP server) and created a bunch of static leases.

The Basics

{
  "Dhcp4": {
    "server-tag": "infra-1",
     "interfaces-config": {
       "dhcp-socket-type": "udp",
       "interfaces": [ "ue0" ],
       "re-detect": true
     },
    "calculate-tee-times": false,
    "authoritative": false,
    "boot-file-name": "",
    "min-valid-lifetime": 300,
    "valid-lifetime": 172800,
    "max-valid-lifetime": 216000,
    "match-client-id": false,
    "client-classes": [
      {
        "boot-file-name": "/ipxe/undionly.kpxe",
        "name": "Legacy PXE",
        "next-server": "0.0.0.0",
        "option-data": [ ],
        "option-def": [ ],
        "server-hostname": "",
        "test": "option[client-system].hex == 0x0000"
      },
      {
        "boot-file-name": "/ipxe/netboot.xyz.efi",
        "name": "EFI x86_64",
        "next-server": "0.0.0.0",
        "option-data": [ ],
        "option-def": [ ],
        "server-hostname": "",
        "test": "option[client-system].hex  == 0x0007"
      },
      {
        "boot-file-name": "",
        "name": "iPXE",
        "next-server": "0.0.0.0",
        "option-data": [ ],
        "option-def": [ ],
        "server-hostname": "",
        "test": "substring(option[77].hex,0,4) == 'iPXE'"
      },
      {
        "boot-file-name": "",
        "name": "PXE Host",
        "next-server": "0.0.0.0",
        "only-if-required": true,
        "option-data": [ ],
        "option-def": [ ],
        "server-hostname": "",
        "test": "member('Legacy PXE') or member('EFI x86_64') or member('iPXE')"
      }
    ],
    "control-socket": {
      "socket-name": "/tmp/kea-dhcp4-ctrl.sock",
      "socket-type": "unix"
    },
     "host-reservation-identifiers": [ "hw-address", "duid", "circuit-id", "client-id" ],
     "lease-database": {
         "type": "memfile",
         "persist": true,
         "lfc-interval": 3600
     },
     "match-client-id": false,
     "max-valid-lifetime": 216000,
     "min-valid-lifetime": 300,
     "next-server": "0.0.0.0",
     "option-data": [ ],
     "option-def": [ ],
     "reservation-mode": "all",
     "sanity-checks": {
       "lease-checks": "warn"
     },
     "server-hostname": "",
     "server-tag": "infra-1",
     "shared-networks": [ ],

HA Setup

{
  "library": "/usr/local/lib/kea/hooks/libdhcp_ha.so",
  "parameters": {
    "high-availability": [
      {
        "heartbeat-delay": 10000,
        "max-ack-delay": 5000,
        "max-response-delay": 60000,
        "max-unacked-clients": 5,
        "mode": "load-balancing",
        "peers": [
          {
            "auto-failover": true,
            "name": "infra-1",
            "role": "primary",
            "url": "http://192.168.1.2:8080/"
          },
          {
            "auto-failover": true,
            "name": "infra-2",
            "role": "secondary",
            "url": "http://192.168.1.3:8080/"
          }
        ],
        "send-lease-updates": true,
        "sync-leases": true,
        "sync-page-limit": 10000,
        "sync-timeout": 60000,
        "this-server-name": "infra-1"
      }
    ]
  }
}

That url cannot be a dns name. It has to be an IP address.

Subnet Setup

"subnet4": [
  {
    "authoritative": false,
    "id": 1,
    "match-client-id": false,
    "next-server": "0.0.0.0",
    "option-data": [
      {
        "always-send": false,
        "code": 3,
        "csv-format": true,
        "data": "192.168.1.1",
        "name": "routers",
        "space": "dhcp4"
      },
      {
        "always-send": false,
        "code": 15,
        "csv-format": true,
        "data": "home.michaelc.dev",
        "name": "domain-name",
        "space": "dhcp4"
      },
      {
        "always-send": false,
        "code": 119,
        "csv-format": true,
        "data": "home.michaelc.dev",
        "name": "domain-search",
        "space": "dhcp4"
      },
      {
        "always-send": false,
        "code": 42,
        "csv-format": true,
        "data": "192.168.1.2, 192.168.1.3",
        "name": "ntp-servers",
        "space": "dhcp4"
      },
      {
        "always-send": false,
        "code": 6,
        "csv-format": true,
        "data": "192.168.1.2, 192.168.1.3, 8.8.8.8, 8.8.4.4",
        "name": "domain-name-servers",
        "space": "dhcp4"
      }
    ],
    "pools": [
      {
        "option-data": [ ],
        "pool": "192.168.1.20-192.168.1.250"
      }
    ],
    "relay": {
      "ip-addresses": [ ]
    },
    "reservation-mode": "all",
    "reservations": [
    {
        "hw-address": "80:2a:a8:93:be:b8",
        "ip-address": "192.168.1.29",
        "hostname": "ubnt-ap1"
    },
    {
        "hw-address": "c4:8e:8f:fc:5d:75",
        "ip-address": "192.168.1.24",
        "hostname": "lap-mc"
    },
    {
        "hw-address": "88:41:fc:98:38:a7",
        "ip-address": "192.168.1.35",
        "hostname": "airi-ap"
    },
    {
        "hw-address": "0c:fe:45:99:e1:d4",
        "ip-address": "192.168.1.36",
        "hostname": "playstation4"
    },
    {
        "hw-address": "a4:77:33:48:18:ae",
        "ip-address": "192.168.1.50",
        "hostname": "chromecast"
    },
    {
        "hw-address": "08:05:81:e7:6a:50",
        "ip-address": "192.168.1.106",
        "hostname": "roku4"
    },
    {
        "hw-address": "b8:27:eb:be:a7:8f",
        "ip-address": "192.168.1.107",
        "hostname": "retropi"
    },
    {
        "hw-address": "2c:fd:ab:0e:f9:cc",
        "ip-address": "192.168.1.111",
        "hostname": "androidf9cc"
    },
    {
        "hw-address": "40:4e:36:82:ef:ff",
        "ip-address": "192.168.1.112",
        "hostname": "android-htc"
    },
    {
        "hw-address": "c4:1c:ff:ab:a1:73",
        "ip-address": "192.168.1.116",
        "hostname": "vizio4k"
    },
    {
        "hw-address": "bc:ff:eb:36:83:98",
        "ip-address": "192.168.1.204",
        "hostname": "t420-laptop"
    },
    {
        "client-id": "08:00:69:07:dc:e8",
        "ip-address": "192.168.1.205",
        "server-hostname": "indigo2",
        "boot-file-name": "/data/irix/i/22/disc1/stand/fx.ARCS"
    },
    {
        "client-id": "08:00:69:12:5d:cd",
        "ip-address": "192.168.1.206",
        "server-hostname": "octane",
        "boot-file-name": "/data/irix/i/30/disc1/stand/fx.64"
    }
],
"subnet": "192.168.1.0/24",
"t1-percent": 0.5,
"t2-percent": 0.875,
"valid-lifetime": 172800
}

The reservations are the gross part. I mostly did this to transition off of the old DHCP server, but also because I haven’t setup DDNS. I like to SSH to my hosts by name so keeping the *nix systems static is preferable.

Kea’s agent controller is what the dhcp4 process uses to check its peer member. My biggest complaint to date, is there is no built-in authentication. ISC recommends placing a proxy in front with some kind of authentication (which is acceptable for users or services interacting with the API to issue commands), however, the dhcp4 servers themselves need to communicate with the agent without credentails. So, unless you have additional firewall rules in place, this feels half baked and insecure by default.

Using Kea’s API

My home version doesn’t have too much going for it, but there are a few things I can do

Fetch Config

curl -X POST -H "Content-Type: application/json" -d '{ "command": "config-get", "service": [ "dhcp4" ] }' infra-1:8080 | jq

piping to jq just makes it fancy and readable. That will render the entire running config (and passwords if you are using a database backend for leases and the configuration database)

Leases

curl -X POST -H "Content-Type: application/json" -d '{ "command": "lease4-get-all",  "service": [ "dhcp4" ] }' infra-1:8080 | jq
[
  {
    "arguments": {
      "leases": [
        {
          "cltt": 1580517612,
          "fqdn-fwd": true,
          "fqdn-rev": true,
          "hostname": "pioneer-vsx-lx303-ef39e1",
          "hw-address": "00:09:b0:1e:e8:c7",
          "ip-address": "192.168.1.43",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        },
        {
          "cltt": 1580517859,
          "fqdn-fwd": true,
          "fqdn-rev": true,
          "hostname": "chromecast",
          "hw-address": "a4:77:33:48:18:ae",
          "ip-address": "192.168.1.50",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        },
        {
          "cltt": 1580515813,
          "fqdn-fwd": true,
          "fqdn-rev": true,
          "hostname": "roku4",
          "hw-address": "08:05:81:e7:6a:50",
          "ip-address": "192.168.1.106",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        },
        {
          "cltt": 1580517575,
          "fqdn-fwd": false,
          "fqdn-rev": false,
          "hostname": "",
          "hw-address": "2c:fd:ab:0e:f9:cc",
          "ip-address": "192.168.1.111",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        },
        {
          "cltt": 1580517756,
          "fqdn-fwd": false,
          "fqdn-rev": false,
          "hostname": "",
          "hw-address": "40:4e:36:82:ef:ff",
          "ip-address": "192.168.1.112",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        },
        {
          "cltt": 1580518048,
          "fqdn-fwd": true,
          "fqdn-rev": true,
          "hostname": "c02xk346jg5m",
          "hw-address": "38:f9:d3:3a:7a:a3",
          "ip-address": "192.168.1.135",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 216000
        },
        {
          "cltt": 1580517757,
          "fqdn-fwd": false,
          "fqdn-rev": false,
          "hostname": "",
          "hw-address": "bc:ff:eb:36:83:98",
          "ip-address": "192.168.1.204",
          "state": 0,
          "subnet-id": 1,
          "valid-lft": 172800
        }
      ]
    },
    "result": 0,
    "text": "7 IPv4 lease(s) found."
  }
]

Since I am using a flat file as the lease database, its written out as a simple csv: 

root@infra-2:~/json # cat /var/db/kea/kea-leases4.csv
address,hwaddr,client_id,valid_lifetime,expire,subnet_id,fqdn_fwd,fqdn_rev,hostname,state,user_context
192.168.1.107,b8:27:eb:be:a7:8f,,172800,1580691195,1,1,1,retropi,0,
192.168.1.21,70:bc:10:04:3d:59,,172800,1580691435,1,0,0,xboxone.,0,
192.168.1.36,0c:fe:45:99:e1:d4,,172800,1580691448,1,0,0,,0,
192.168.1.94,e0:d5:5e:6f:ce:58,,172800,1580691859,1,0,0,desktop-ke7rahc.,0,
192.168.1.20,58:cb:52:7b:4e:d8,,172800,1580692153,1,0,0,,0,
192.168.1.35,88:41:fc:98:38:a7,,172800,1580692171,1,1,1,airi-ap,0,
192.168.1.29,80:2a:a8:93:be:b8,,172800,1580692412,1,1,1,ubnt-ap1,0,
192.168.1.29,80:2a:a8:93:be:b8,,172800,1580692412,1,1,1,ubnt-ap1,0,
192.168.1.135,38:f9:d3:3a:7a:a3,,216000,1580737632,1,1,1,c02xk346jg5m,0,

Subnet and Remote Set - Subscriber only

The real good stuff is behind a subscription fee. I can’t use that at home, however, at work its been pretty fun.

I can’t share all of the tooling I helped write at work, I can at least show a mock up of a subnet.

If you are using the mysql configuration data backend, you can add a subnet by posting the following to the Kea Controller Agent: 

{
  "command": "remote-subnet4-set",
  "service": [ "dhcp4" ],
  "arguments": {
    "subnets": [
      {
        "id": 2,
        "subnet": "192.168.100.0/24",
        "shared-network-name": null,
        "pools": [
         {
          "pool": "192.168.100.20 - 192.168.100.250.254"
         }
        ],
        "option-data": [
         {
          "name": "routers",
          "data": "192.168.100.1",
          "always-send": false
         },
         {
          "name": "domain-name",
          "data": "enterprise.michaelc.dev",
          "always-send": false
         },
         {
          "name": "domain-search",
          "data": "enterprise.michaelc.dev",
          "always-send": false
         },
         {
          "name": "domain-name-servers",
          "data": "192.168.100.10, 192.168.100.11, 192.168.100.12",
          "always-send": false
         }
        ],
        "user-context": {
         "location": "fancy-workplace-with-free-lacroix"
        },
        "require-client-classes": [
         "PXE Host"
        ]
       }
      ],
    "remote": {
      "type": "mysql"
    },
    "server-tags": [
      "all"
    ]
  }
}

However, if you are not using the configuration database, and are still using the subnet_cmd hook, you have to use the subnet4-add command: 

{
 "command": "subnet4-add",
 "service": [
  "dhcp4"
 ],
 "arguments": {
  "subnet4": [
   {
    "id": 2,
    "subnet": "192.168.100.0/24",
    "pools": [
     {
       "pool": "192.168.100.20 - 192.168.100.250.254"
     }
    ],
    "option-data": [
     {
      "name": "routers",
      "data": "192.168.100.1",
      "always-send": false
     },
     {
      "name": "domain-name",
      "data": "enterprise.michaelc.dev",
      "always-send": false
     },
     {
      "name": "domain-search",
      "data": "enterprise.michaelc.dev",
      "always-send": false
     },
     {
      "name": "domain-name-servers",
      "data": "192.168.100.10, 192.168.100.11, 192.168.100.12",
      "always-send": false
     }
    ]
   }
  ]
 }
}

Also, when you use subnet4-add opposed to remote-subnet4-set you have to also issue a config-write command. Otherwise, it only stays in memory for the duration of the kea dhcp4 process: 

{ 
    "command": "config-write",
    "arguments": {
        "filename": "/etc/kea/kea-dhcp4.conf"
    },
    "service": [ "dhcp4" ]
}

a handy way to do this is in a script, here is an example in python: 

def write_config(server, path):
    payload={ 
                "command": "config-write",
                "arguments": {
                    "filename": path
                },
                "service": [ "dhcp4" ]
            }
    response = requests.post(str("http://") + server + str(":8080"), json=payload)
    response.raise_for_status()
    return response.json()[0]
    
def write_configs(servers):
    for server in servers:
        try:
            result = write_config(server, "/etc/kea/backups/kea-dhcp4-" + str(datetime.datetime.now()).replace(" ", "_") + ".json" )
            if result['result'] == 0:
                print(f"\nConfig written successfully")
            elif result['result'] == 3:
                print(f"\nConfig write was empty")
            elif result['result'] == 1 :
                print(f"\nConfig was unsucessful")
            else:
                print(f"\nUnknown result for config-write: {result}")
            
            result = write_config(server, "/etc/kea/kea-dhcp4.json")
            if result['result'] == 0:
                print(f"\nConfig written successfully")
            elif result['result'] == 3:
                print(f"\nConfig write was empty")
            elif result['result'] == 1 :
                print(f"\nConfig was unsucessful")
            else:
                print(f"\nUnknown result for config-write: {result}")
        except Exception as e:
            print(f"\nFailed to write config on {server}: "+str(e))

iPXE and netboot.xyz

At home, I decided to pull down netboot.xyz’s EFI dhcp image.

I haven’t been able to get a VMWare Fusion instance to PXE boot, but, my desktop did just fine:

Troubleshooting

tcpdump is your best friend, because no matter what debug level you set Kea to, it will not top a packet capture

tcpdump -i ue0 -vvv '((udp port 67) and (udp[8:1] = 0x1))'

16:35:15.789018 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 346)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:09:b0:1e:e8:c7 (oui Unknown), length 318, xid 0x36d2ac76, Flags [none] (0x0000)
          Client-Ethernet-Address 00:09:b0:1e:e8:c7 (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Client-ID Option 61, length 7: ether 00:09:b0:1e:e8:c7
            Requested-IP Option 50, length 4: 192.168.1.43
            Server-ID Option 54, length 4: 192.168.1.3
            MSZ Option 57, length 2: 576
            Parameter-Request Option 55, length 7:
              Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
              Domain-Name, BR, NTP
            Vendor-Class Option 60, length 12: "udhcp 1.23.2"
            Hostname Option 12, length 24: "Pioneer-VSX-LX303-EF39E1"
            END Option 255, length 0

Another testing utility is tftp. We had to verify out dhcp server(s) were serving the file:

# tftp localhost
tftp> get /ipxe/ipxe.efi-x86_64
Received 997856 bytes during 0.3 seconds in 1949 blocks
tftp> ^Droot@infra-2:~ #
# sha512 ipxe.efi-x86_64
SHA512 (ipxe.efi-x86_64) = 965917bd494c404783900325d976528969b76401781bcfeb81e583a3759d36a07a77fcfc00bca14c7a50f14206de8d93cacbc098f4bab54bac2270e07c9cb19f
# sha512 /usr/local/tftp/ipxe/ipxe.efi-x86_64
SHA512 (/usr/local/tftp/ipxe/ipxe.efi-x86_64) = 965917bd494c404783900325d976528969b76401781bcfeb81e583a3759d36a07a77fcfc00bca14c7a50f14206de8d93cacbc098f4bab54bac2270e07c9cb19f
#

you should see something like this on your dhcp/tftp server:

Feb  2 18:04:40 infra-2 in.tftpd[21310]: RRQ from 127.0.0.1 filename /ipxe/ipxe.efi-x86_64

#DHCP #FreeBSD #Geekyness #Work