using couchdb with puppet and bacula

2012-01-25

On aspect that I was never happy with the Bacula environment I built while at LLNL was the fact that I could no look up certain values for each client. Values like:

  • Passwords
  • Storage Devices
  • Certificates (if you are using Encryption)

Well, over the past few week’s I’ve been able to work around this problem by storing additional information in a CouchDB DB.

It is not the ideal solution, but it is a start and I’m okay with that. I should also warn you, I do HORRIBLE things here with Bash and JSON. Since Bash doesn’t know about JSON, I rely upon awk. I know, I know, I should re-write all of this in a nice new language like Python or Ruby…

First thing is first, I had to create a new database: 

$ curl -kX PUT https://puppet.bayphoto.local/bacula_meta
{"ok":true}

I’m not going to worry about that name, to me it is a database that contains some metadata of our clients.

My client creation tool that I posted in my previous Bacula article has been updated to do a little bit more. Aside from no longer using TEMPLATE files, I’ve added some additional code to push a few details into this new bacula_meta database. Here is my “write_json” function:

# Some NEW Variables:
export COUCH_SERVER="https://puppet.bayphoto.local"
export DB="bacula_meta"
export CERTDIR="$BDIR/certs"

write_json()
{
   curl -H "Content-Type: application/json" -kX PUT -d '{ "_id": "'${HOSTNAME}'","passhash": "'${PASSHASH}'" }' $COUCH_SERVER/$DB/$HOSTNAME
}

The document I create is simple, it is named after the short hostname of the client added to backups, and for this first run we store that and the password.

The “main” function of the script first tests to see if a document in the bacula_meta db exists, and if not it will create a new client. If it does exist, you can either continue and re-create the bacula client’s configuration, or quite:

TEST=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME`
if [[ $TEST == *not_found* ]]
then
       # Generate a bacula password.
       export PASSHASH=`dd if=/dev/random bs=6 count=4 2>/dev/null | openssl base64`

       # This is the actual .conf configuration
       print_client_conf
 
       # Create a new client document in $DB
       write_json
       
       # Create SSL key-pair
       create_keys

       # Adding the client .conf file for the director to source.
       echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf

       echo 'created client definition: '$BDIR/clients.d/$HOSTNAME.conf
       echo 'for '$HOSTNAME
else
       echo 'client '$HOSTNAME 'already exists.'
       echo 'Do you want to override the current configuration for:'
       echo '      '$HOSTNAME
       read -p "[y/N] " prompt
       prompt=${prompt,,}
       if [[ $prompt =~ ^(yes|y)$ ]]
       then
               # if we choose to override, a new client conf will be generated and added and commited.
               
               # Lets re-obtain our stored client password first
               export PASSHASH=`curl -k  -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F: 'gsub("{|}","") { print $5 }'`

               # print out a new cliend.conf
               print_client_conf
               
               # Push the clients key-pair back to couchdb       
               curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$FQDN-fd.pem?rev=$DOC_REV --data-binary @$CERTDIR/$FQDN-fd.pem  -H "Content-Type: application/octet-stream"

                grep -w $HOSTNAME $BDIR/clients.conf
                if [ $? -eq 0 ]
                then
                        echo 'client '$HOSTNAME 'already exists...'
                else
                        echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf
                fi
 
       else
               echo "Ok, no clients were modified or added!"
       fi
fi

The other addition was a create_keys function. Our clients encrypt their data to the storage node (we send some backup volumes to S3 storage, which is over http and not stored in any sort of encrypted format), and we needed a decent way to distribute the keys (using Puppet…).

This was difficult for me to do. What I failed to understand about adding attachments to CouchDB is you have to reference the current document _rev, and after a LOT of trial and error I finally got it. The DOC_REV variable grabs the current documents revision: 

DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`
Which is then used when I actually PUT the file in there: 
curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"

create_keys()
{
  DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`
  C="US"
  ST="California"
  L="Santa Cruz"
  O="Bay Photo Lab"
  OU="IT"
  CN="${HOSTNAME}.bayphoto.local"
  EMAIL="bayit@bayphoto.com"
  
openssl genrsa -out ${CERTDIR}/${CN}.key 2048
openssl req -new -key ${CERTDIR}/${CN}.key -x509 -out ${CERTDIR}/${CN}.cert <<EOF
${C}
${ST}
${L}
${O}
${OU}
${CN}
$EMAIL
EOF
echo ""

cat ${CERTDIR}/${CN}.key ${CERTDIR}/${CN}.cert > ${CERTDIR}/${CN}-fd.pem

curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"
}

So what does adding a new client look like using this updated tool?

# ./cclient.bash -s Standard -h client-a
INSERT 0 1
{"ok":true,"id":"client-a","rev":"1-0841684988ec85c6d2b16cb941a739ac"}
Generating RSA private key, 2048 bit long modulus
..............................................................+++
..............+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Name (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (eg, YOUR name) []:Email Address []:
{"ok":true,"id":"client-a","rev":"2-21d4e7bc019c2176dfa2583b320387ab"}
created client definition: /usr/local/etc/bacula/clients.d/client-a.conf
for client-a
And my new record in CouchDB has all the right data: 
curl -kX GET https://puppet.bayphoto.local/bacula_meta/client-a
{"_id":"client-a","_rev":"2-21d4e7bc019c2176dfa2583b320387ab","hostname":"client-a","passhash":"y9WBgacrd8JbZjrefeZHKbPk9Kda5UQc","_attachments":{"client-a.bayphoto.local-fd.pem":{"content_type":"application/octet-stream","revpos":2,"digest":"md5-kqi8ODloPxT6D6IxZbCoVg==","length":3411,"stub":true}}}

Thats ugly… how about a nice screenshot!

todo

Now that we have the Bacula tool pushing passwords and and certificates, we need to get Puppet to pull the data.

I found a github project called couchdblookup: couchdblookup

I placed that couchdblookup.rb file into one of my Puppet modules (etc/puppet/environments/production/bacula/lib/puppet/parser/functions/couchdblookup.rb), and created a bacula::fd::cert class: 

class bacula::fd::cert inherits bacula::fd {

  # Pull bacula client password from our
  # CouchDB server
  $couchdb_bind_address = "puppet.bayphoto.local"
  $couchdb_port = "5984"
  $couchdb_base_url = "https://${couchdb_bind_address}:${couchdb_port}"
  $bacula_meta = "${couchdb_base_url}/bacula_meta/${hostname}"
  $bacula_fd_cert = "${couchdb_base_url}/bacula_meta/${hostname}/${fqdn}-fd.pem"

  $bacula_fd_passhash = couchdblookup($bacula_meta, "passhash")

  file { "master.cert":
    name    => $operatingsystem ? {
      FreeBSD  => "/usr/local/etc/bacula/certs/master.cert",
      windows  => "C:\Program Files\Bacula\master.cert",
      default  => "/etc/bacula/certs/master.cert",
    },
    owner   => 0,
    mode    => 0640,
    source  => "puppet:///bacula/master.cert",
  }

  exec { "fd.cert":
    path    => ["/usr/bin","/usr/local/bin","/bin","/sbin","/usr/sbin","/usr/local/sbin","/usr/local/libexec","/usr/libexec"],
    command => $operatingsystem ? {
      FreeBSD  => "fetch -o /usr/local/etc/bacula/certs/${fqdn}-fd.pem $bacula_fd_cert",
      windows  => "C:/scripts/curl.exe -sk $bacula_fd_cert -o \"\Program Files\Bacula\\${::fqdn}-fd.pem\"",
      default  => "curl -sk $bacula_fd_cert -o /etc/bacula/certs/${fqdn}-fd.pem",
    },
  }

}

As you can see, I’m also working on getting Windows systems into our Puppet environment.

It is incredibly immature right now, and Windows lacks a lot of tools I take for granted. It would make my life a lot easier if Microsoft just tool all the BSD licensed userland tools like diff, fetch (or curl), md5, ssh, etc… to make my Puppet automation easier. You NEED diff.exe to use Puppet on windows, otherwise templating won’t work.

Aside from the windows side of things being a pain, this has been working out well enough.

#Bacula #CouchDB #Featured #FreeBSD #Geekyness #Puppet