configuration management with salt stack


I’ve often made the remark that the open source community is a fickle crowd. There is always a new fork, or a new hot development team that everyone is clamoring to be involved in.

I strive for stability, but I’m not going to stick with something for historical reasons. If I find a better tool, I’ll spend a good amount of time working with it before I call it production.

For example:

  • UFS -> ZFS
  • Apache -> Nginx
  • PHP -> Ruby/Python
  • Slackware -> RHEL -> Fedora -> Ubuntu…
  • KDE -> Gnome -> Xfce
  • FreeBSD -> FreeBSD
    • If it ain’t broke dont fix it :)

After hearing about Salt, and distributed execution engine over a year ago, my co-worker and I had been keeping close tabs on it. We attended a LSPE meeting where all sorts of folks talked about their own Configuration Management experience. The overall message was “We used Puppet, but we felt Salt had a more accommodating community”.

Another good description is “Salt is a configuration management tool done right the first time”.

Once I switched to pkgng, which salt supported, and we had a handful of new systems to test and deploy with, it quickly became the preferred tool.

My big headaches with Puppet were the following:

  • Out of the box, Puppet did not scale. You had to setup Passenger and Apache to get any reasonable amount of performance
    • I figured this out early, and I never had production scaling issues
  • Ruby made the manifest execution order unpredictable, and the only way to work around that was to have a chain of dependencies, notifies and requries that became difficult to keep track of
  • For a while, you had to be careful about which version of Puppet and mod_passenger you were running (and of course, Ruby), but this had stabilized
  • Some folks in charge of puppet are kind of arrogant, and it shows by how the products are managed. Their buisness plan and how they aggressively push the commercial version of Puppet was a turn off.
  • After 2.5, I was never able to actually use the external node database. They came out with PuppetDB to solve this issue, but it was not an official released tool in FreeBSD’s Ports. I had to pull it down from github and manually build it
    • It was riddled with sloppy linux-isms ( silly things like assuming /bin/sh is actually bash, or not specifying gmake vs. cmake vs. make)
  • There was a disdain for FreeBSD/BSD in general
  • It seemed like every release broke how I used variable lookups. Puppet never really felt stable in that sense, so it was frustrating.

Salt on the other hand:

  • “Batteries Included” mindset. The single py-salt port/package has the master and minion. It works great out of the box and scales very well
    • Using a fast task queing system like ZeroMQ that is built in makes a difference
  • Its FAST. Python is fast, and its parallel execution is noticibly faster than Puppet ever was
    • It also helps that I switched from Portmaster/Portupgrade to our own pkgng environment
  • Execution of states works the same time, everytime.
  • I like how I can have Private data per host or group of hosts (Pillars), static system information (grains) and use those in the State file and the template engine.
    • Probably my killer feature. I LOVE that I can keep bind credentials as a pillar and never let it seep into the state files

The Saltstack team are very receptive to feedback, and I have yet to see Hatch turn down a pull request. Following the mailing list, you constantly see Hatch saying things like “Thats a good idea, lets work on that”. filed a feature request myself and it was well recieved and resolved in a few days.

Basically, there have yet to be any bikeshed’s and I like that. That is probably more because Salt is still new and there are no real ego’s involved yet.

It also helps that there is strong FreeBSD support and there are active patches.

Did I mention it is new?

Yeah, there are some areas that I hope they improve quickly. Logging and validation are two biggies for me. Right now, if you mess up a pillar, your minions will not show ANY pillar data, and they won’t tell you why either. It you muck up a template file, its is difficult to see where exactly.

I really enjoyed Puppets validate flag. I have a Jenkins job that quickly and simply validates the basic syntax of my .pp files. This quickly points out most of the issues we have while we are building new modules.

Lets get down to the real core reason why anyone should switch a critical peice of infrastructure software.

Does this help us deploy systems better and faster? Is there more or less accountibility between the new and old system? How often am I surprized by version changes?

I don’t really know, not yet at least. Rob likes it a lot more than Puppet, so that virtue by itself doubles the productivity of the system.

What I can do, is show two different ways of doing something.

To start off easy, lets do the defacto ssh server, which seems to be the first thing anyone ever does.

The Puppet way:


├── manifests
│   ├── client.pp
│   ├── init.pp
│   └── server.pp
└── templates
    └── sshd_config.erb


class ssh {
    include ssh::server, ssh::client 
class ssh::client inherits ssh {
    if $bootstrap::is_public != "yes" {
            file { "/etc/pam.d/sshd":
                owner   => 0,
                group   => 0,
                content => template("ssh/pam/$operatingsystem/sshd.erb"),
class ssh::server inherits ssh {
    service { "sshd":
        name    => $operatingsystem ? {
            Ubuntu    => "ssh",
            default   => "sshd",
        enable  => true,
        ensure  => running,
        require => File['/etc/ssh/sshd_config'],

    file { "/etc/ssh/sshd_config":
        ensure  => present,
        owner   => 0,
        group   => 0,
        mode    => 600,
        content => template("ssh/sshd_config.erb")
        notify  => Service['sshd']

    group { "local-allow":
        ensure    => present,
        name  => 'local-allow',
        gid       => '1000',
        provider  => $osfamily ? {
            FreeBSD              => pw,
            /(Redhat|Debianc)/   => groupadd,
<% if scope.lookupvar("bootstrap::is_public") == "yes" %>
<% else %>
ListenAddress <%= ipaddress %>
<% end %>

Protocol 2

<% if operatingsystem == "FreeBSD" %>
<% else %>
SyslogFacility AUTHPRIV
<% end %>

KerberosAuthentication yes
KerberosOrLocalPasswd yes
KerberosTicketCleanup yes

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

UsePAM yes


<% if kernel == "FreeBSD" %>
Subsystem   sftp    /usr/libexec/sftp-server
<% elsif kernel == "Linux" %>
Subsystem        sftp    /usr/libexec/openssh/sftp-server
<% else %>
Subsystem       sftp    /usr/libexec/sftp-server
<% end %>

AllowGroups wheel local-allow <% if scope.lookupvar("bootstrap::is_public") == "yes" %>syncwww<% end %>

Pretty Simple. We manage a service, one that is notified if the sshd_config template is changed.

What is missing here is the openssh-server package that is required on some linux distro’s.

The Salt Way


├── init.sls
└── pkg.conf.jinja
├── absent.sls
└── init.sls
├── absent.sls
├── init.sls
├── keys
│   ├── absent.sls
│   └── init.sls
└── server
    ├── absent.sls
    ├── init.sls
    └── sshd_config.jinja
└── init.sls

Right off the bat, you can see I implemented things a bit different with Salt.

The Puppet way, was that each module should be an isolated and independent state. So, a ssh module had to manage the group, keys, users… all of it, because calling facts or variables between classes was not really encouraged (but I did it to a degree anyway. It bit me a few times)

With Salt, the entire collection of states is part of “top”, so I can be pretty granular (pun intended) with the states.



{% if grains['kernel'] == 'Linux' %}
  - pkg:
    - installed
{% endif %}

   - ssh.keys

# SSH keys for users
  - ssh

        - present
        - user: mikec
        - enc: ssh-rsa
  - ssh

    - running
    - enable: True
    - watch:
      - file: /etc/ssh/sshd_config

    - source: salt://ssh/server/sshd_config.jinja
    - template: jinja


Protocol 2

KerberosAuthentication yes
KerberosOrLocalPasswd yes
KerberosTicketCleanup yes

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

UsePAM yes


Subsystem   sftp    /usr/libexec/sftp-server

{% if pillar['allowed_users'] %}AllowUsers {% for user in pillar['allowed_users'] %} {{ user }} {% endfor %}{% endif %}


      {% for user in pillar['allowed_users'] %}
      - name: /home/{{ grains.realm.split('.')[0] }}/{{ user }}
      - user: {{ user }}
      {% endfor %}
      - mode: 700

      - gid: 1000
         - system: True

That last bit right there with the AllowUsers directive is a simple example of using Pillars for unique and private minion information in a state.

The Pillar that makes that possible is this:


      - mike
      - rob
      - service_account

Thats a simple SSH module, and you’ll notice my Salt implementation needs more work since I don’t have any special conditions about OS specific settings for sshd.

Perhaps in another post I’ll show what I’ve done for some more complex states ( internal vs dmz systems, ftp with ldap authentication, etc… )