Utah Open Source Planet

Image via Wikipedia

Pat Eyler sent me a note asking if we were still using puppet. In the spirit of trying to leverage my response to him for the widest possible audience, I thought I'd blog the answer.

Kynetx has been using Puppet (and a little Capistrano) to automate our infrastructure since this summer. While we haven't gotten everything as automated as we'd like, we're quite a ways down the road and it's making life as a small startup bearable.

My ultimate goal is to register a node in the system that delivers Kynetx Network Services (KNS) to the machine database (iClassify) and have it come up, configure itself, and be ready to put in production in less than 15 minutes. While there are still a number of manual steps in that process we are remarkably close and getting closer all the time.

I haven't found learning puppet to be difficult. There's plenty of documentation, including recipes that if not usable as is, are great resources for figuring out best practices. Also, there's a very helpful group on IRC (#puppet on irc.freenode.net) that I've turned from time to time.

The best thing about Puppet is that I can count on nodes under puppet management to be consistently configured and the puppet recipes represent an unambiguous record of what should be happening.

A recent example, we needed to ensure that data files from an OEM partner were on all of our initialization servers. I used Puppet in two ways:

1. Puppet is used to distribute the data files from the ops server to each of the initialization servers.
2. Puppet is used to manage the cron job that grabs updated data from the partner and put it in the right directory for distribution.

If I add another initialization node puppet will make sure it has the right data files just because I declare it to be an initialization node. Nothing else need occur.

I recently released Kynetx's operational procedures. For a small company with no fulltime engineering staff other than me, I think we do a pretty good job operationally. Much of that we owe to Puppet.

Needless to say, I remain, more than ever, a Puppet fan.

Tags: puppet kynetx

I wrote about puppet and how we're starting to look into it at work a couple of days ago. Yesterday, I learned about an awesome little tool that puppet provides — ralsh, the Ruby Abstraction Layer SHell. To understand just how cool ralsh is, you need to know a little bit more about how puppet works. puppet allows you to describe how a server should 'look' using a declarative language (this is

I've been trying to introduce Ruby at work for a long time, and have had some success here. I've been able to show many of the developers the value of Ruby, and it's made it's way into the organization as a valuable tool. I've never been able to make many inroads into the systems administration team that I work with though, and this has bothered me because I see myself as much more of a sys

The code that makes Kynetx work is a couple of custom Apache modules written in Perl. So, configuring machines via puppet, naturally requires ensuring that a set of Perl modules are loaded.

For a long time, I was using a private bundle, but I found that was unreliable. For the most part it worked fine, but then sometimes I'd get an error that Perl couldn't find the bundle, even though it was clearly in the path.

I didn't want to spend a lot of time debugging it because I figured it was a dead end. Eventually, I've got to be able to control more of what happens. So, I decided to define a custom Puppet type that used CPAN directly to load Perl modules. Here's what I came up with:

define cpan_load() {
  exec{"cpan_load_${name}":
     command => "perl -MCPAN -e 
             '\$ENV{PERL_MM_USE_DEFAULT}=1; 
               CPAN::Shell->install(\"${name}\")'",
     onlyif => 
          "test `perl -M${name} -e 'print 1' 2>/dev/null || \
                 echo 0` == '0'",
     require => [Package["perl-CPAN"],
                    File["/root/.cpan/CPAN/MyConfig.pm"]],
     }
 }

The hardest part to get right was the onlyif clause. The purpose of this clause is to ensure that the command is idempotent--that is that it only runs if it needs to. The truth is that CPAN itself is idempotent, so I could have skipped it, but I saved 80 seconds of processing time for a "do nothing" run of puppet by installing it.

I got something more too. CPAN will report things are installed when they aren't really. I don't know how it gets in this state, but it does. My onlyif clause only reports a module is present if a perl program can load it.

This still isn't perfect since when the load fails and you rerun the puppet file, you see it trying to install them over and over and it's not going to happen. Then manual intervention is required. I'd rather have the system back out and fix the error but that's beyond me right now.

I think the answer may be to move to Tasks, but I'm not sure. If you know better ways of managing Perl module installs with Puppet, please let me know.

Tags: puppet cpan perl kynetx

The whole idea of Puppet is to put your machine configuration scripts in a versioned repository. This is good because I've found that a syntax error on a manifest not even used by the current machine will stop the puppet updates from running. One error anywhere kills the whole thing everywhere. So, being able to back out of a change is good.

We can go one better however and keep people from checking in files that aren't syntactically correct using the pre-commit hook in SVN. If you don't use SVN, your repository probably has something similar.

I followed these instructions for the hook and these instructions for installing it.

I ran into a problem however. The pre-commit script wasn't reporting an error condition correctly. I'm not sure what was wrong. Running the puppet command on the command line and then checking $? gave a "1", but inside the script it wasn't working that way. So, I punted and wrote the error output to a file and checked whether or not the file existed and then just cat'd that out to standard err. Messy, but it worked and I need to go home. Here's the file as modified by me:

#!/bin/bash
# SVN pre-commit hook to check Puppet syntax for .pp files
PATH="/usr/bin:/bin"
REPOS="$1"
TXN="$2"
tmpfile=`mktemp`
errfile=`mktemp`
export HOME=/
SVNLOOK=/usr/bin/svnlook
$SVNLOOK changed -t "$TXN" "$REPOS" | awk '{print $2}' \
   | grep '\.pp$' | while read line
do
  $SVNLOOK cat -t "$TXN" "$REPOS" "$line" > $tmpfile
  if [ $? -ne 0 ]
  then
    echo "Warning: Failed to checkout $line" >&2
  fi
  /usr/bin/puppet --color=false --confdir=/tmp \
     --vardir=/tmp --parseonly --ignoreimport $tmpfile \
     >$errfile 2>/dev/null
  if [  -s $errfile ]
  then
    echo "Puppet syntax error in $line." >&2
    cat $errfile >&2
    exit 2
  fi
done
res=$?
rm -f $tmpfile
rm -f $errfile
if [ $res -ne 0 ]
then
  exit $res
fi

Note: you may need to combine lines separated by "\\" into a single line.

This works great. Syntax errors are caught before they make it into the repository. I'm still looking for a good way to thoroughly test puppet scripts before they get into the production servers. If you have any best practices in that area, I'd love to hear about them.

Tags: puppet kynetx svn

I'm continuing to explore the use of puppet for managing systems. I'm convinced that it's the best system for managing large groups of interacting servers and keeping them all in sync. For example, today I was trying to figure out why a cronjob that processes log files wasn't working. Turns out that it was a system issue and there was also a problem with mail aliases, so we weren't even finding out.

I create a simple puppet recipe for all our machines that simply ensures crond is running. That's basic and now it's ensured on all our machines. This was simple:

class cron {
      service { crond:
	      ensure => running,
	      enable => true
      }		
}

Next I created a manifest to installed the correct crontab entry on the right machines to process the logs along with setting the right environment variables for the processing daemon. There's a cron "type" in puppet so it's pretty easy:

class kobj_log_daemon {

      cron { kobj_log_daemon:
           command => "bin/kobj_log_daemon -v",
	   user => www,
	   hour => 0,
	   minute => 5,
	   environment => ["WEBROOT=/www", "MAILTO=root"],
      }
}

Finally, I took care of mail aliases by putting /etc/aliases under the control of puppet. Here's the class definition that does that:

class aliases {
 file { "/etc/aliases":
   mode  => 640,
   source => 
     "puppet://puppet.kobj.net/dist/apps/sendmail/aliases",
   owner => "root",
   group => "root",
   before  => Exec["create aliases db"]
 }

 exec { "new_aliases":
   command => "/usr/bin/newaliases",
   alias => "create aliases db",
   subscribe => File["/etc/aliases"],
   refreshonly => true,
 }
}

Now there's one place--under version control--where all of this is controlled for every machine we own. Managing this any other way would be a recipe for disaster.

Tags: system+configuration kynetx puppet

The final talk of the day (hope I make my flight) was by Luke Kanies of Reductive Labs on Puppet.

Most automation tools are based on SSH and as a result, they suck. The problem is that the intersection of administrator and developers is very small. Luke wanted Puppet to be so good it was like bringing a gun to a knife fight. The goal: manage lots of machines with very little effort.

Luke makes an analogy about the transition from assembly to C and moving from commands and files to "resources." Resources are abstract and portable. Abstraction is the most important idea here. Why do we have to know how to, for example, update packages on Fedora and Debian.

Packages are the basic unit. You can install, uninstall, update, etc. packages. There are 23 package types. Resources say what to do for a given package. Here's an example for SSH:

class ssh {
  package { ssh: ensure => installed }
  file { sshd_config:
    name => $operatingsystem ? {
      Darwin  => "/etc/sshd_config",
      Solaris => "/opt/csw/etc/ssh/sshd_config",
      default => "/etc/ssh/sshd_config"
    },
    source => "puppet://server.domain.com/files/ssh/sshd_config"
  }
  service { ssh:
    name => $operatingsystem ? {
      Solaris => openssh,
      default => ssh
    },
    ensure    => running,
    subscribe => [Package[ssh], File[sshd_config]]
  }
}

The class provides intent; by creating a class for ssh, you're saying it should be installed and running. Note that the installation, configuration, and service all have their own definitions.

Nodes allow you to associate hosts, by type, with resources. Transactions make sure everything is in it's correct state. Transactions are idempotent--they don't have an effect if machines are in the right state. Idempotency allows management throughout the lifecyce. Puppet should manage a machine from it's birth to death.

Tags: velocity08 puppet automation infrastructure

Puppet is a great looking configuration management tool written in Ruby, from Luke Kanies at Reductive Labs. Recently, James Turnbull has written a book, Pulling Strings with Puppet, about it. I’ve interviewed both Luke and James about Ruby, Puppet, and James’ book. So, it’s only fair that I turn an eye to the book as well. In the interest of full disclosure I should mention that Apress

Recently, I’ve been reading about Puppet because of James Turnbull’s excellent Pulling Strings with Puppet. James has been good enough to do a short interview with me as well. I hope you enjoy reading it as much as I enjoyed chatting with James (you might want to check out my interview with Luke Kanies too). I normally think of you as a Pythonista, yet Puppet comes from the land of Ruby.

I’m currently reading Pulling Strings with Puppet, the new book from Apress. So far I’m really enjoying it, look for a review in a week or two. The book whetted my appetite to learn more about Puppet though, so I decided to go to the source and talk to Luke Kanies of Reductive Labs, the development team behind Puppet. Hey, I've just posted my interview with James Turnbull too. Here’s my