building packages for freebsd

2013-04-05

I have held out on building packages on FreeBSD for a long time.

My experience with portmaster and portupgrade was not perfect, but it was pretty consistent, and I always knew how to recover from any failures. I normally had systems build packages via portmaster during the early am, or through some automated process that I didn’t have to look at.

At this point though, pkgng has become a much better tool than the pkg_ tools, and while portmaster and portupgrade support registering packages with pkgng, I felt this was a good time to start building local packages for our infrastructure.

The main tools I use are:

  • Poudriere : Sort of like Tinderbox, but it is specifically for pkgng (I didn’t like tinderbox and pkgng integration either)
  • Jenkins CI : I use jenkins to kick off and manage building packages, syncing them to a web server, and re-generating the package index
  • EJBCA : Our internal CA that I setup. I used it to generate the OpenSSL keypair that is used to sign packages. Its a wonderful tool and I like to mention it as much as possible. Its too complex to discuss here, perhaps at another date

Lets start with Poudiere

Poudriere

ZFS Setup

You’ll need a ZFS pool, which means you need zfs enabled:

kldload zfs
echo zfs_enable=YES > /etc/rc.conf.d

I do all of this on a VM (two in fact, I build desktop packages separate from server packages), and each vm has a dedicated virtual disk for poudriere buils:

zpool create data ada1

Install

Install ports-mgmt/poudriere through whatever mechanisms you want to (but I bet after this, you’ll start using pkg :) )

  • portinstall ports-mgmt/poudriere
  • cd /usr/ports/ports-mgmt/poudriere
  • pkg install poudriere

Configuration

A .sample file will get installed, and that has a few more options, but we’ll keep it basic:

ZPOOL=data
FTPHOST=ftp4.us.freebsd.org
RESOLV_CONF=/etc/resolv.conf
POUDRIERE_DATA=/data/poudriere_data
BASEFS=/data/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CRONDIR=${BASEFS}/cron
SVN_HOST=svn0.us-west.FreeBSD.org
NO_PACKAGE_BUILDING=no
PKG_REPO_SIGNING_KEY=/usr/local/etc/poudriere.d/pkgng.key

The ZPOOL value should be the name of the zfs pool created above. I like to use the name data, its just a preference of mine. The PKG_REPO_SIGNING_KEY is your key, and I just placed it in the .d directory.

Also, you may want to pick an ftp and svn host that is geographically close to you, but thats optional.

Once that is done, you’re all ready to create jails and a ports tree

Jails

Poudriere needs to build packages from a FreeBSD jail, and this is really cool. You can have a 9.1, 8.2, 10 (aka, CURRENT) and mix between i386 and amd64 platforms.

I myself only build 9.1-RELEASE amd64 packages. Our older 9.x systems are still using Puppet and Portmaster. When I upgrade them, I’ll switch them to Salt and pkgng.

First, create a ports tree for your jail(s) to use:

$ poudriere ports -c
$ poudriere ports -l
PORTSTREE            METHOD     PATH
default              portsnap   /data/poudriere/ports/default

The default update method is to use portsnap, but you can also use svn:

$ poudriere ports -c -m svn

Next, we create a jail. I’ll create a 9.1-RELEASE AMD64 jails:

$ poudriere jail -c -j 91amd64 -a amd64 -v 9.1-RELEASE
$ poudriere jail -l
JAILNAME             VERSION              ARCH    METHOD  SUCCESS FAILED  IGNORED SKIPPED QUEUED  STATUS
91amd64              9.1-RELEASE          amd64   ftp     0       0       0       0       0       idle:

Building a Port

The command you’ll be working with is poudriere bulk. I use the -f flag to pass it a file, and I’ll show that later, but to test everything out go ahead and build a single port:

poudriere bulk -j 91amd64 sysutils/sysrc

Here is the output:

====>> Mounting system devices for 91amd64
/etc/resolv.conf -> /data/poudriere/jails/91amd64/etc/resolv.conf
====>> Starting jail 91amd64
====>> Mounting ports from: /data/poudriere/ports/default
====>> Mounting packages from: /data/poudriere_data/packages/91amd64-default
====>> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/91amd64-options
====>> Appending to /etc/make.conf: /usr/local/etc/poudriere.d/91amd64-make.conf
====>> Populating LOCALBASE
====>> Calculating ports order and dependencies
====>> Sanity checking the repository
====>> Deleting stale symlinks
====>> Cleaning the build queue
====>> Building 1 packages using 1 builders
====>> [01] Starting build of sysutils/sysrc
====>> [01] Finished build of sysutils/sysrc: Success
====>> Stopping 1 builders
====>> Creating pkgng repository
tar: Removing leading '/' from member names
Generating repo.sqlite in /usr/ports/packages/: done!
====>> Cleaning up
====>> Umounting file systems
====>> Built ports: sysutils/sysrc

====>> [91amd64] 1 packages built, 0 failures, 0 ignored, 0 skipped

We can now see that package in /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz, and if you want to install if manually, just use pkg add:

$ pkg info -l -F /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz
sysrc-5.1 owns the following files:
/usr/local/libexec/sysrc/include/messages.subr
/usr/local/man/man8/sysrc.8.gz
/usr/local/sbin/sysrc
/usr/local/share/licenses/sysrc-5.1/BSD
/usr/local/share/licenses/sysrc-5.1/LICENSE
/usr/local/share/licenses/sysrc-5.1/catalog.mk
/usr/local/share/sysrc/common.subr
/usr/local/share/sysrc/sysrc.subr

$ pkg add /data/poudriere_data/packages/91amd64-default/sysutils/sysrc-5.1.txz
Installing sysrc-5.1... done

Otherwise, you can point a system to the generated repo file and use pkg install

Build-time options

Before we get into the entire automation process, did you notice this line at the begining of the build?

====>> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/91amd64-options
====>> Appending to /etc/make.conf: /usr/local/etc/poudriere.d/91amd64-make.conf

Poudriere will first look in /usr/local/etc/poudriere.d/${JAIL_NAME}-options for the build time options for a port (make options).

For the most part, I take the default options except for a few very important builds like Samba, Nginx, Apache, etc…

To set those build time options, just run poudriere options -j 91amd64 net/samba36

The other thing poudriere does is look at your local /etc/make.conf and appends it to /usr/local/etc/poudriere.d/${JAIL_NAME}-make.conf

My make.conf looks like this: 

WITH_PKGNG=yes
WITHOUT_X11=yes
.if empty(.CURDIR:M/usr/ports/databases/mongodb*)
CC=gcc
CXX=g++
CPP=cpp
.endif
CC=clang
CXX=clang++
CPP=clang-cpp

And, my make.conf for the vm that builds desktop pacakges looks like this: 

WITH_PKGNG=yes
CC=clang
CXX=clang++
CPP=clang-cpp
WITHOUT_NOUVEAU=yes
WITH_NEW_XORG=yes

As you can see, I make our server packages without X11 support, but the desktop packages get X11.

I try to build with clang, but there are a few cases when it just doesn’t work so I’ve attempted to use a little if statement (I’m not sure if it works thought…)

Managing with Salt

I have a fairly simple salt state to help the build process, and I only recently discovered that salt has a poudriere module to help manage everything I just wrote above.

Pillar Data

poudriere.sls

{% if grains['nodename'] == 'pkg-server' %}                                                  
poudriere_role: server
{% elif grains['nodename'] == 'pkg-desktop' %}
poudriere_role: desktop
{% endif %}

State

poudriere:
   pkg.installed

/root/plist:
   file.managed:
      - source: salt://pkgbuilds/{{ pillar.poudriere_role }}.plist

{{ pillar.etc_prefix }}/poudriere.conf:
   file.managed:
      - source: salt://pkgbuilds/poudriere.conf.jinja

{{ pillar.etc_prefix }}/poudriere.d/pkgng.key:
   file.managed:
      - source: salt://pkgbuilds/pkgng.key

AAAAdHQhyFaKapB1SGysz_SSHKEY_FOR_JENKINS_USER_9pzqHE310HGveDIvUixxxpS64C0BDuPdbjqOVgOGzzkxOtgqxDT5mWQq3:
  ssh_auth:
      - present
      - user: jenkins-ci
      - enc: ssh-rsa

ZPOOL=data
FTPHOST=ftp4.us.freebsd.org
RESOLV_CONF=/etc/resolv.conf
POUDRIERE_DATA=/data/poudriere_data
BASEFS=/data/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CRONDIR=${BASEFS}/cron
SVN_HOST=svn0.us-west.FreeBSD.org
NO_PACKAGE_BUILDING=no
PKG_REPO_SIGNING_KEY=/usr/local/etc/poudriere.d/pkgng.key
converters/p5-JSON-PP
databases/couchdb
databases/rubygem-couchrest
databases/mongodb
databases/memcached
databases/mysql55-client
databases/mysql55-server
databases/pecl-memcache
databases/postgresql92-client
databases/postgresql92-server
,,,
accessibility/atk
accessibility/atkmm
archivers/gtar
archivers/p7zip
archivers/rpm
archivers/rpm2cpio
archivers/squeeze
archivers/unzip
archivers/zip
audio/alsa-lib
...

Jenkins

Setup

I based most of what I have off of this great three part article Continuous package building with poudriere and Jenkins under FreeBSD

I eventually re-worked it and did it my own way, which I just found simpler to grok and manage.

You’ll need to enable two Jenkin’s plugins:

  • Jenkins PostBuildScript Plugin
  • Jenkins Publish Over SSH Plugin

Once those are installed, you need to setup a ssh host under Jenkins -> Manage Jenkins -> Configure System -> Publish Over SSH

PkgNg Job

Create a new Jenkins job. I titled mine FreeBSD-9.1-amd64-server, and the job type is Build a free-style software project

The job has 5 simple steps:

  • Update the ports tree
    • poudriere ports -u
  • Build the packages from the /root/plist file
    • poudriere bulk -J 8 -j 91amd64 -f /root/plist
  • Show the jail information
    • poudriere jails -i -j 91amd64
  • Rsync packages to separate web server:
    • rsync -ave “ssh -i /home/jenkins-ci/.ssh/id_rsa” –force /data/poudriere_data/packages/91amd64-default/ jenkins-ci@pkgng:/data/www/pkgng/data/9.1-freebsd-amd64/server/
  • Rebuild the repo index
    • pkg repo /data/www/pkgng/data/9.1-freebsd-amd64/server /data/www/pkgng/pkgng.key

To set that up in Jenkins, add these Build Steps:

  • Build
    • Send files or execute commands over SSH
      • SSH Publisher
        • name: pkg-server
        • Credentials:
          • Username: jenkins-ci
          • Passphrase/Password: *************
          • Path to key: /home/jenkins-ci/id_rsa.publish
        • Transfers:
          • Source files: /bin/sh
          • Remote Prefix:
          • Remote Directory:
          • Exec command:
            • sudo poudriere ports -u
            • sudo poudriere bulk -J 8 -j 91amd64 -f /root/plist
            • sudo poudriere jails -i -j 91amd64
          • Exec timeout: 0 (ms)
  • Post-build actions
    • [PostBuildScript] - Execute a set of scripts
    • Build steps
      • SSH Publisher
        • name: pkg-server
        • Credentials:
          • Username: jenkins-ci
          • Passphrase/Password: *************
          • Path to key: /home/jenkins-ci/id_rsa.publish
        • Transfers:
          • Source files: /bin/sh
          • Remote Prefix:
          • Remote Directory: /root
          • Exec command: /root/pkg-rsync.sh
          • Exec timeout: 0 (ms)
      • SSH Publisher
        • name: pkgng
        • Transfers:
          • Source files: /bin/sh
          • Remote Prefix:
          • Remote Directory: /data/www/pkgng/data/9.1-freebsd-amd64/server/
          • Exec command: pkg repo . /data/www/pkgng/pkgng.key

Clients

Now that the packages are built, and then rsync’d over to a web server (I’m using nginx), your clients will need to have a usable pkg.conf, as well as the public key for the signed packages

Salt

In our basepkgs state, we manage the pkg.conf file, and push out the public certificate:

basepkgs_init.sls

{% if grains['kernel'] == 'FreeBSD' %}
pkg:
  pkg.installed

{{ pillar.etc_prefix}}/pkg.conf:
  file.managed:
    - require:
      - pkg: pkg
    - source: salt://basepkgs/pkg.conf.jinja
    - template: jinja
    - context:
      role: {{ pillar['role']['host'] }}

{{ pillar.etc_prefix }}/pkg/ssl/pkgng.pub:
   file.managed:
      - require:
         - pkg: pkg
      - source: salt://basepkgs/pkgng.pub

{% endif %}

pkg.conf.jinja

PUBKEY: /usr/local/etc/pkg/ssl/pkgng.pub

PACKAGESITE:    https://pkgng.bayphoto.com/{{ grains.osrelease }}-{{ grains.os|lower }}-{{ grains.cpuarch }}/{{ role }}/`

The pillar data for host.role is used to differentiate if a client will use the Desktop package repo (built with X11 support and CUPS), or the Server repo (trimmed down options and almost no support for X11, GTK+, CUPS, etc…)

The final file that ends up on the system will look like this: 

PUBKEY: /usr/local/etc/pkg/ssl/pkgng.pub

PACKAGESITE:    https://pkgng.bayphoto.com/9.1-freebsd-amd64/server/

Update / Upgrade

Now, we can finally update our repo index, and upgrade packages:

root@pkg-server:/root # pkg update
Updating repository catalogue
repo.txz                                                                             100%  275KB 275.2KB/s 275.2KB/s   00:01    
root@pkg-server:/root # pkg upgrade
Updating repository catalogue
Repository catalogue is up-to-date, no need to fetch fresh copy
New version of pkg detected; it needs to be installed first.
After this upgrade it is recommended that you do a full upgrade using: 'pkg upgrade'

The following packages will be upgraded:

	Upgrading pkg: 1.0.9_2 -> 1.0.11

The installation will require 6 kB more space

1 MB to be downloaded

Proceed with upgrading packages [y/N]: y
pkg-1.0.11.txz                                                                       100% 1524KB   1.5MB/s   1.5MB/s   00:01    
Checking integrity... done
Upgrading pkg from 1.0.9_2 to 1.0.11... done
root@pkg-server:/root # pkg upgrade
Updating repository catalogue
Repository catalogue is up-to-date, no need to fetch fresh copy
The following packages will be upgraded:

	Upgrading gamin: 0.1.10_4 -> 0.1.10_5
	Reinstalling cairo-1.10.2_5,2
	Reinstalling pango-1.30.1
	Upgrading libxml2: 2.7.8_5 -> 2.8.0_1
	Upgrading vim: 7.3.669 -> 7.3.669_1
	Upgrading subversion: 1.7.8 -> 1.7.9
	Upgrading gdk-pixbuf2: 2.26.5 -> 2.26.5_3
	Upgrading portmaster: 3.14_9 -> 3.16

The installation will free 5 MB

12 MB to be downloaded

Proceed with upgrading packages [y/N]: y
gamin-0.1.10_5.txz                                                                   100%   70KB  69.6KB/s  69.6KB/s   00:00    
cairo-1.10.2_5,2.txz                                                                 100%  524KB 523.8KB/s 523.8KB/s   00:00    
pango-1.30.1.txz                                                                     100%  553KB 553.0KB/s 553.0KB/s   00:01    
libxml2-2.8.0_1.txz                                                                  100%  774KB 774.1KB/s 774.1KB/s   00:00    
vim-7.3.669_1.txz                                                                    100% 5253KB   5.1MB/s   2.3MB/s   00:01    
subversion-1.7.9.txz                                                                 100% 4485KB   4.4MB/s   4.4MB/s   00:01    
gdk-pixbuf2-2.26.5_3.txz                                                             100%  518KB 518.0KB/s 518.0KB/s   00:00    
portmaster-3.16.txz                                                                  100%   43KB  43.0KB/s  43.0KB/s   00:00    
Checking integrity... done
Upgrading gamin from 0.1.10_4 to 0.1.10_5... done
Reinstalling cairo-1.10.2_5,2 done
Reinstalling pango-1.30.1 done
Upgrading libxml2 from 2.7.8_5 to 2.8.0_1... done
Upgrading vim from 7.3.669 to 7.3.669_1... done
Upgrading subversion from 1.7.8 to 1.7.9... done
Upgrading gdk-pixbuf2 from 2.26.5 to 2.26.5_3... done
Upgrading portmaster from 3.14_9 to 3.16... done
root@pkg-server:/root #

Ta Da!

As a FreeBSD and general SysAdmin, this process has been pretty cool. I’ve discovered a few hiccups, so hopefully with the above information, you will not have to deal with those.