Utah Open Source Planet

This weekend, I installed Scalix on a client's mail server. I had installed Scalix before, but it was a clean install with no old e-mail to migrate. This time, however, I had a lot of old e-mail to migrate.

Most of the users used Eudora for Windows. Others used the openwebmail web mail client. As a result, I had e-mail from three different sources to populate with:

• Eudora locally-saved mailboxes (an almost mbox-format)
• IMAP folders on the server, in mbox format
• Standard mbox mailspool boxes (inboxes).

In preparation, I had read on a page on the Scalix site there were scripts to import mbox files into Scalix, but as it turned out, these scripts were for restoring a proprietary-to-Scalix backup mailbox format. Bleh.

The next best solution was to use an e-mail client that could import mbox mailboxes and move the messages to an IMAP folder. Thunderbird only imports a few formats on Windows and only imports Netscape Communicator on Linux. Bleh.

So, I tried kmail. It worked beautifully. At one point, I had three instances of kmail running in three different VNC sessions on the server funneling messages into Scalix. The only problem with kmail is that it required constant babysitting as I moved from mailbox to mailbox and user to user.

So, while kmail was cranking along on a large mailbox, I looked at CPAN to see what modules were available for dealing with IMAP and mbox file formats. It didn't take long before I had a script that would upload a user's messages. That worked great for the mbox mailspool files- there was one file per user.

Next were the IMAP/webmail mailboxes. I had copied them all into separate folders for each user, so to automate it, I needed a script that could go into each user folder and upload the messages from each of the mailboxes it found there. However, there were some mailboxes I didn't want to upload (e.g. Spam, Virus, Trash boxes). Others, I needed to rename to go into Scalix equivalents.

The following Perl script is what I came up with.

#!/usr/bin/perl

use Smart::Comments;
use Mail::IMAPClient;
use Mail::Box::Mbox;

my $users = [
    {   username   =>  'user1',
        password   =>  'password1', },
    {   username   =>  'user2',
        password   =>  'password2', }, ];

my $folder_translation = {
    'sent-mail'     =>  'Sent Items',
    'sentmail'      =>  'Sent Items',
    'saved-drafts'  =>  'Drafts',
    'saved-messages'=>  'Drafts',
};

my $ignore_folders = [ 
    'spam-mail', 
    'virus-mail', 
    'mail-trash', 
    'Junk',
    'Junk E-mail', 
    'Trash', ];

foreach my $user (@$users) { 
    if( -d "/export/imap/$user->{'username'}") {

        warn "Looking at $user->{'username'}\n";
        my $imap = Mail::IMAPClient->new(  
            Server => 'mail.example.com',
            User    => $user->{'username'} . '@example.com',
            Password=> $user->{'password'},) or 
            die "Could not log in as $user->{'username'}";

       
        opendir DH, "/export/imap/$user->{'username'}";
        my @files = readdir DH;

        FILELOOP:
        foreach my $file (@files) {
            if($file !~ m/^\./) {   # Skip hidden
                foreach my $ign (@$ignore_folders) {
                    if($file eq $ign) {
                        warn "Ignoring $file\n";
                        next FILELOOP;
                    }
                }
                my $translated_name = $file;
                foreach my $tr_key (keys %$folder_translation) {
                    if($tr_key eq $file) {
                        $translated_name = $folder_translation->{$tr_key};
                    }
                }

                # Ready to upload messages
                warn "Uploading $file to $translated_name\n";

                my $folder = Mail::Box::Mbox->new(
                    folder => '/export/imap/' . $user->{'username'} .  '/' . $file);

                my @folders = $imap->folders();
                if(! grep /^$translated_name$/, @folders) {
                    $imap->create($translated_name);
                }

                foreach my $msg ($folder->messages) { ### Uploading msgs |===[%]           |
                    my $uid = $imap->append($translated_name, $msg->string);
                    if(! $uid) {
                        warn "Could not append message\n";
                    }
                }

                $folder->close() or warn "Could not close Mbox connection $@\n";
            }
        }
        closedir DH;

        $imap->disconnect() or warn "Could not close IMAP connection $@\n";
    }
    else {
        next;
    }

}

As I mentioned in yesterdays post, I have begun studying perl.  I’m actually enjoying it quite a bit, and picking things up fairly quickly I think.  I’m sure my familiarity with bash scripting helps quite a bit.

This morning I hacked together an irssi script for origami users that will let you display your current work unit progress percentage within irssi chat.  I’d like some feedback on it.

If you are using irssi and have origami running on the same machine please check out my plugin and let me know what can be improved.

Download origami.pl v0.2

Instructions:

download and save the above script into ~/.irssi/scripts

To load the script:

/script load origami.pl

To get the origami work unit status:

/origami

To unload the script:

/script unload origami

Other Points of Interest

• November 17, 2008 — Regex Challenge (24)
• November 12, 2008 — Origami 0.6.9 Released (0)
• April 20, 2008 — Origami (previously folding.sh): Now In PPA! (7)
• March 27, 2008 — Quick Poll Regarding folding.sh (0)

Over the weekend I started studying Perl and quickly realized I was going to be better off if I reviewed regular expressions before I got too far into things.  I went back and found my copy of “Mastering Regular Expressions” and dove right in.  Now, maybe its just me, but I find that I really enjoy the problem solving aspect of regular expressions.  I thought it might be fun to put up a regular challenge on the blog that needs to be solved via regular expressions.

(Obligatory xkcd reference)

So, I figure I’d start off with one that caught me today.  Here is the situation:

You’ve got a regular text file filled with usernames.  You want to be able to read this file into a program to populate an array, but there are random blank lines throughout the file.  What regex would you use to find and remove all empty lines in the file?

For consistency sake, I’ve populated just such a file and made it available here.

Rules: Use any tool you want (perl, sed, vim, etc)  The file must contain all original usernames (total of 15), one per line, with no blank lines start to finish.  Please share your solution in the comments!

Other Points of Interest

• November 18, 2008 — Hacking Perl (0)

The code that makes Kynetx work is a couple of custom Apache modules written in Perl. So, configuring machines via puppet, naturally requires ensuring that a set of Perl modules are loaded.

For a long time, I was using a private bundle, but I found that was unreliable. For the most part it worked fine, but then sometimes I'd get an error that Perl couldn't find the bundle, even though it was clearly in the path.

I didn't want to spend a lot of time debugging it because I figured it was a dead end. Eventually, I've got to be able to control more of what happens. So, I decided to define a custom Puppet type that used CPAN directly to load Perl modules. Here's what I came up with:

define cpan_load() {
  exec{"cpan_load_${name}":
     command => "perl -MCPAN -e 
             '\$ENV{PERL_MM_USE_DEFAULT}=1; 
               CPAN::Shell->install(\"${name}\")'",
     onlyif => 
          "test `perl -M${name} -e 'print 1' 2>/dev/null || \
                 echo 0` == '0'",
     require => [Package["perl-CPAN"],
                    File["/root/.cpan/CPAN/MyConfig.pm"]],
     }
 }

The hardest part to get right was the onlyif clause. The purpose of this clause is to ensure that the command is idempotent--that is that it only runs if it needs to. The truth is that CPAN itself is idempotent, so I could have skipped it, but I saved 80 seconds of processing time for a "do nothing" run of puppet by installing it.

I got something more too. CPAN will report things are installed when they aren't really. I don't know how it gets in this state, but it does. My onlyif clause only reports a module is present if a perl program can load it.

This still isn't perfect since when the load fails and you rerun the puppet file, you see it trying to install them over and over and it's not going to happen. Then manual intervention is required. I'd rather have the system back out and fix the error but that's beyond me right now.

I think the answer may be to move to Tasks, but I'm not sure. If you know better ways of managing Perl module installs with Puppet, please let me know.

Tags: puppet cpan perl kynetx

I read Is learning Perl the hard way the easy way? recently.

Andy Lester links to Learning Perl the Hard Way.

Reading the PDF, Allen B. Downey makes a great suggestion when learning new languages.
He says, “As you are experimenting with a new language, I suggest that you make deliberate errors in order to get familiar
with the most common error messages.”

That is some of the simplest, best advice I’ve heard in a while.

The system we're building at Kynetx includes a domain specific language that uses rules to create JavaScript programs that get sent down to the browser. I've documented our decision to use a domain specific langauge and our choice of Perl in other posts.

When I started this project, I was reading Mark Dominus' book Higher Order Perl and started using his HOP parser to play around with. One thing led to another an before you know it I had a full blown language parser in HOP without giving much thought as to whether or not I'd made the right choice.

I found the HOP parser to be pretty flexible, but it has it's quirks. More importantly, I didn't like the BNF specification format and so I was constantly trying to keep the spec and the implementation in sync. Better if I could just use the spec as the implementation 'ala Bison. Don't get me wrong, this is a great book with lots of wonderful ideas, but I wanted something else for the parser.

As I added more and more features to the language, it got to where I'd dread making the parser changes. Recently, I decided I had to significantly beef up the predicate expressions and thought it would be a good time to change out the parser as well.

A few months ago I picked up Christopher Frenz's Pro Perl Parsing in anticipation of just this day. Reading through it illuminated my choices and ultimately, I picked Damian Conway's Parse::RecDescent, a recursive descent parser over the other contender, Parse::Yapp. The reasons for my choice were partly esthetic and partly a trust in Damian. The main thing I was after was a parse spec that I could read and compile and RecDescent seemed great in that regard.

The biggest downside of RecDescent is that there's no associated Lexer. For most things that's not a big deal since terminals can be specified as regular expressions. The place where it really bit me was comments. Removing comments is trickier than you'd think because you don't want to process "start of comment markers" inside any quotes. With a lexer, that's easy; without one, it's more problematic. Writing the regexp to remove comments took me a while to get right. I ended up using a modified version of the solution given in this FAQ. The problem with most solutions, including Regexp::Common, which has a language comment module, is that they don't account for comment markers in quotes.

All in all, rewriting the parser was a good exercise and I'm happy with the choice of RecDescent. Here's a sample production from my file:

decl: VAR '=' VAR ':' VAR '(' expr(s? /,/) ')'
      {$return =
       {'lhs' => $item[1],
        'type' => 'data_source',
        'source' => $item[3],
        'function' => $item[5],
        'args' => $item[7]
       }
      }
    | VAR '=' 'counter' '.' VAR
      {$return =
       {'lhs' => $item[1],
        'type' => $item[3],
        'name' => $item[5]
       }
      }
    | VAR '=' HTML
      {$return =
       {'lhs' => $item[1],
        'type' => 'here_doc',
        'value' => $item[3]
       }
      }
    | 

This production for decl has three alternates. Each has a separate return value (a hash) that represents the portion of the abstract syntax tree created for that part of the input.

If you decide to give Parse::RecDescent a try, here are some resources:

• The Parse::RecDescent documentation
• The Parse::RecDescent tutorial
• The Parse::RecDescent FAQ
• An O'Reilly getting started-style tutorial
• An example showing SQL CREATE TABLE statements

Reading the documentation and the FAQ thoroughly is highly recommended. There's lots of little tricks that can make your job easier.

My job, replacing an existing parser, was made easier by the fact that I'd previously built a pretty thorough test suite in Perl for the parser and some related modules. So once I got the language spec pretty much complete, I started running the tests and correcting errors as they cropped up. In a few hours, I'd solved all the problems and was confident my parser was ready to go. Definitely the way to go.

At any rate, now I've for a shiny new parser that I can go modify. Fun!

Tags: kynetx perl programming+languages parsing

When we purchased our house, there was a semi-neglected little garden in the back yard. It had been planted in the spring but pushed down in importance as the house was put up for sale during the summer. Having grown up with a three quarter acre gar

Dave Cross has a piece on why corporations hate Perl. He's being a little hyperbolic (as he admits)--not everyone hates Perl, but he's right in noting that there is a backlash against it. He says:

I was talking to people from one such company last night. The Powers That Be at this company have announced that Perl is no longer their language of choice for web systems and that over time (probably a lot of time) systems will be rewritten in a combination of Java and PHP. Management have started to refer to Perl-based systems as "legacy" and to generally disparage it. This attitude has seeped through to non-technical business users who have started to worry if developers mention a system that is written in Perl. Business users, of course, don't want nasty old, broken Perl code. They want the shiny new technologies.

And so, in a matter of months, the technical managers at this company have create a business environment where Perl is seen as the cause of most of the problems with the current systems. It's an impressive piece of social engineering

From Why Corporates Hate Perl - O'Reilly ONLamp Blog
Referenced Wed Aug 20 2008 10:43:00 GMT-0600 (MDT)

He labels this kind of things "anti-Perl social engineering." I've run into similar feelings.

At Kynetx, our systems are built in Perl and Ruby. We use Rails to create the user facing pieces, but the engine, that part that needs to run fast, is written as an Apache module in Perl.

I've noticed that when I explain the language choice to people I almost always have to answer the spoken or unspoken question: "Perl?? Really? Why Perl?" After I explain the architecture and the reasons behind the decision, they're almost always satisfied, but there's a definite stigma.

I'm doing a talk next week at the Utah Open Source Conference (Friday morning at 10am) on using Apache as an application server. There are some significant advantages and Apache offers some real great application support that is largely untapped. But to talk advantage of it, you're pretty much stuck with C...or Perl. Given that choice the decision is a no-brainer. I'll be putting my thoughts together for the talk later this week or next week and promise to blog the ideas.

Still, the idea that Perl is an old crusty language couldn't be more wrong. I've been programming in Perl for nigh on 15 years now and I don't think it's ever been more interesting. Perl sports the best collection of libraries of any dynamic language and supports modern engineering practices like modules, objects, and testing. I've seen ugly Perl code--heck I've written it--but that's not Perl's fault.

I spoke with a local company last week that has been a Perl shop but is moving to Java. I smiled because I know that means there will be a lot of unhappy Perl programmers there who I can hire later! :-)

I'll continue to proudly carry the torch for Perl.

Tags: kynetx perl programming+languages

Last night, a close, personal friend sent me e-mail asking me for a "script fu" favor. It would seem that my close, personal friend had somehow acquired a collection of audio files and these files were in a format that his personal media player device would not play. The audio files were encoded in the MPEG-4 Audio (M4A) format and my close, personal friend's personal media player device supports a wide range of formats including FLAC, WAV, Ogg Vorbis, MP3, and perhaps some others I can't remember at the moment. My friend (who is my close, personal friend) asked me if I could "whip something up" that could convert all his files to MP3 format.

What a nice challenge!

For a few moments, I considered tackling this problem with a shell script using time-tested command line utilities like find, sed, and grep, but ultimately, I decided to engage this challenge using Perl.

I chose Perl over shell scripting mostly because the directory tree that needed to be traversed to access all the files had file and directory entries that contained an arbitrary number of whitespace characters and other not-so-friendly-to-shell characters. While I'm sure this could have been accomodated, it didn't seem like fun and I got excited thinking about how this could be handled with Perl.

It's a fun exercise to write Perl scripts that use opendir, readdir, and other standard Perl functions to interact with the host filesystem, but I knew there were some valuable CPAN modules, maybe even some "indistinguishable from magic" modules maintained by Damian Conway, I could use.

The first module I decided to use was File::Find::Rule which provides an alternative interface to File::Find.

use File::Find::Rule;

...

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( '/path/to/root/of/files' );

File::Find is somewhat of a relic in terms of how it operates. It doesn't provide any kind of object oriented interface for using it and requires the user to pass subroutine references which isn't very pretty. File::Find::Rule, on the other hand, works relatively nicely.

To convert the files, once we had them in a list, I figured we'd use the veritable bastion of audio versatility that is mplayer and use its built-in ability to construct standalone WAV files from media files. To do this at the command line, use the pcm audio output option and specify a filename:

mplayer -ao pcm:file=myfile.wav someotherfile.m4a

There are a couple quadrillion other options and parameters you could also add, but this is the general gist of it.

After each MPEG-4 audio file is decoded and dumped into a WAV file, we can use lame to encode the WAV to MP3 format.

The lame utility, in its simplest form, works like this:

lame myfile.wav output.mp3

Like mplayer, there are a ridiculous number of options, switches, parameters, chants, and secret handshakes you can provide to make lame do its job faster, slower, on one foot, etc.

One of the, uhm... inconveniences, yeah, of calling other programs from a Perl script is that it isn't easy to tell what's going on or how things went on... or off, or whatever. The same is generally true in a shell scripting environment, but that's not important right now. What is important is that our good man Damian has done a fantastic job of helping make this easier by providing the Perl6::Builtins module to the Perl community. Apparently, this incongruent behavior when calling external applications is not an issue in the long-forthcoming next major version of Perl (Perl 6). Damian has just ported the nice behavior back to Perl 5.

The Perl6::Builtins module gives us a new system function we can use which behaves like a good system function should.

use Perl6::Builtins qw(system);

...

system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
   die "Could not dump $file to WAV: $!";

Using the standard system function, the above code would almost always result in a call to die because the normal exit status of the system call, while sensibly being zero because there are no errors during program execution, means something else entirely to Perl. Instead, Perl detects failure.

With Damian's indistinguishable-from-magic help, sanity is restored.

So, below is the whole script, with some minor things changed to protect the... uhm... lonely.

I'm not proud of the code I wrote to create the destination paths. It works, but not gracefully.

#!/usr/bin/perl

use Readonly;
use File::Find::Rule;
use Perl6::Builtins qw/system/;

Readonly my $sourcetree = 
    '/home/friend/audio/Zarry Lotter (Cantonese)';
Readonly my $sourcetree_exp = 
    '\/home\/friend/audio\/Zarry Lotter \(Cantonese\)';
Readonly my $desttree => 
    '/home/friend/audio/zarry_lotter_cantonese_mp3';

my @files = File::Find::Rule->file()
                            ->name('*.m4a')
                            ->in( $sourcetree );

if( ! -d $desttree) {
    mkdir $desttree || die "Could not make directory: $!";
}

foreach my $file (@files) {
    my $dest = $file;
    $dest =~ s{$sourcetree_exp}{$desttree};
    $dest =~ s{m4a}{mp3};

    my @path_components = split /\//, $dest;
    # Remove common leading components
    for (1 .. 5) { 
        shift @path_components ; 
    }
    # Remove filename
    pop @path_components;

    my $path = $desttree;
    foreach my $comp (@path_components) {
        if(!  -d "$path/$comp") {
            warn "Making directory [$path/$comp]";
            mkdir "$path/$comp";
        }
        $path = "$path/$comp";
    }

    # Use mplayer to dump file to WAV
    system('/usr/bin/mplayer', '-ao', 'pcm:file=/tmp/out.wav', $file) or 
        die "Could not dump $file to WAV: $!";

    # Use lame to make an mp3
    system('/usr/bin/lame', '/tmp/out.wav', $dest) or 
        die "Could not convert $file to MP3: $!";
}

It doesn't matter what language you do your server-side web development in, presentation templates just make so much sense. Here are some reasons why:

• Templates (usually) let you re-use commonly used blocks of code.
• Templates (usually) make it easy (or easier) for a web designer (e.g. not a developer) to work on the presentation layout of your application.
• Templates allow you to separate presentation from business logic and will help you separate them in how you think of your application as well.

By templates, I mean files that contain HTML data along with some special coded method of interpolating dynamic data into the HTML.

By this definition, PHP and ASP code are template languages themselves. That's one of the more substantial reasons I've come to dislike these languages! You'll come to understand as you read more below.

Here's a very simple template example:

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>$TITLE</title>
    </head>
    <body>
        <h1>Welcome to the $SITE_NAME website!</h1>
    </body>
</html>

The above example uses the syntax defined by the CGI::FastTemplate Perl module. I used this module for a couple of projects in 2000 or so and while it does make it easy to design a page's look and feel around the dynamic content your web application is going to drop into the page, it works best when your content doesn't need to be nested inside HTML. For example, dynamics data in a table does not work well with CGI::FastTemplate. Neither do lists of data.

Some templating systems, like CGI::FastTemplate to some extent, can handicap you too much. Others, like HTML::Embperl or ePerl, in my opinion, provide too much capability at the template level.

You don't want to be able to write your entire application in a template. You might as well be writing your application in PHP, JSP, or ASP (or Apache::ASP).

I think a templating system should provide you with just enough logic so that you can affect how data is presented and not much more logic than that. I'm not alone in thinking these things. See Wikipedia topics: Web template system, Model-view-controller, and Separation of concerns.

So, as a result of study of templating systems for Perl, I recommend Template (Template Toolkit for Perl).

Note: Template Toolkit does allow some things I don't think have any place in templates, but it doesn't do so by default. You have to make a conscious choice to do those things, read the documentation, etc. and that should (hopefully) discourage you from doing so.

How to use Template Toolkit

Let's take a look at how Template Toolkit would be used to mimic the CGI::FastTemplate example above.

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% title %]</title>
    </head>
    <body>
        <h1>Welcome to the [% site_name %] website!</h1>
    </body>
</html>

It's not a whole lot different, is it? And, actually, it's a little more typing to key in those beginning and ending brackets instead of just a preceding dollar sign.

Now, here's an example of how this template file (we'll call it page1.tt) would be used in Perl code, say, in a CGI script.

#!/usr/bin/perl

use CGI qw/:standard/;
use Template;

my $q = CGI;
my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/'});

my $vars = {};

$vars->{'title'} = 'Joe Schmoe Shoe Repair - Home';
$vars->{'site_name'} = 'Joe Schmoe Shoe Repair';

print $q->header('text/html');
print $tt->process('page1.tt', $vars) || 
    die $tt->error();

Great! Now we know Template is just as good as CGI::FastTemplate! What else can it do?!

Wrapping an application

One thing I often do with Template Toolkit is use it to wrap applications so that every page has the same look and feel. You do this by defining a wrapper template in the hash reference you pass to the Template constructor:

my $tt = Template->new({
    INCLUDE_PATH    =>  '/var/www/templates/',
    WRAPPER         =>  'sitewrapper.tt', });

Now every page generated by the process() function will include the contents of sitewrapper.tt around it.

The wrapper template needs to contain a special template directive in it: [% content %]. This is where the contents of your specified templates are placed.

Here is an example of a wrapper template:

<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>[% template.title %]</title>
        <link rel="stylesheet" type="text/css" href="style.css" />
    </head>
    <body>
        <div class="sidebar">[% INCLUDE sidebar.tt %]</div>
        <div class="body">[% content %]</div>
        <div class="footer">[% INCLUDE footer.tt %]</div>
    </body>
</html>

Notice the use of the [% content %] directive in the wrapper template.

There are a couple other interesting uses of Template Toolkit directives in this example, notably the INCLUDE directive (for inserting the parsed output of other templates) and a special directive within the HTML title tags.

The [% template.title %] directive should make sense once we see how a normal template might look when using a wrapper:

    [% META title = "Contact us" %]
    <h1>Contacting Joe Schmoe Shoe Repair!</h1>
    <p>See the list below for our telephone numbers:</p>
    <ul>
    [% FOREACH phone = phone_numbers %]
        <li>[% phone.name %]: [% phone.number %]</li>
    [% END %]
    

The META directive let's us pass data into the wrapper template.

Flow control

The above template example also introduces to one of the reasons I love Template Toolkit: flow control. In this case, a for-each loop. The variable phone_numbers could be a list of hashes we get from a database and pass to the template. Each hash contains a name-value pair named name and one named number. The FOREACH directive allows us to iterate through the phone_numbers list and assign a reference the current hash to a variable we've named phone. Inside the loop, we can reference the individual name-value pairs by name.

This is only one of the flow control structures Template Toolkit offers. There are also IF-THEN-ELSIF-ELSE structures, CASE/SWITCH structures, and WHILE loops.

These should give you just a taste of what Template Toolkit can do. With the Template Perl module installed on your system, you get all the documentation you could hope for as POD and man files. The Template Toolkit website also has all this documentation online as well.

Below is a bit of a more complex template I created years ago for the utahisps.com website.

      <h3>[% company.name %]</h3>
      <table cellpadding="6">
        <tr>
          <td>Services</td>
          <td>
                [% has_services = 0 %]
                [% FOREACH service_name = services.keys %]
                  [% IF services.$service_name %]
                    <a href="/isp/[% company.company_id %]/[% service_name %]">
                    [ % service_name %]</a><br/>
                    [% has_services = 1 %]
                  [% END %]
                [% END %]
                [% IF has_services == 0 %]
                  N/A
                [% END %]
                  <!-- None listed -->

              </td>
            </tr>
        <tr>
          <td>Address</td>
          <td>
            [% company.addr1 %]<br/>
        [% IF company.addr2.length %]
        [% company.addr2 %]<br/>
        [% END %]
        [% company.city %], [% company.state %] [% company.zip %]
          </td>
        </tr>
        <tr>
          <td>Phones</td>
          <td>
            [% IF company.phone1.length %]
                [% company.phone1 %] ([% company.phone1_desc %])
        [% END %]
            [% IF company.phone2.length %]
        <br/>
                [% company.phone2 %] ([% company.phone2_desc %])
        [% END %]
          </td>
        </tr>
        [% IF company.fax.length %]
        <tr>
          <td>Fax</td>
          <td>[% company.fax %]</td>
            </tr>
            [% END %]
            <tr>
              <td>Website</td>
          <td>< <a href="[% company.www_url %]" 
          target="_new">[% company.www_url %]
          </a> ></td>
            </tr>
        [% IF company.info_email.length %]
        <tr>
          <td>Info E-mail</td>
              <td>< <a href="mailto:[% company.info_email %]"
              target="_new">
              [% company.info_email %]</a> ></td>
            </tr>
            [% END %]
        <tr>
          <td>Payment</td>
              <td>
                [% IF company.credit_cards.length %]
                  [% company.credit_cards %]
                [% ELSE %]
                  Cash/check only
                [% END %]
            </tr>
        [% IF company.oper_since %]
        <tr>
          <td>Oper since</td>
              <td>[% company.oper_since %]</td>
            </tr>
            [% END %]

        <tr>
              <td><a href="http://www.angio.net/rep/" 
              target="_new">Utah
              REP</a><br/>member</td>  
              <td>[% IF company.ut_rep %]Yes[% ELSE %]No[% END %]</td>
            </tr>
            
        [% IF company.os.length %]
        <tr>
          <td>Primary OS</td>
              <td>[% company.os %]</td>
            </tr>
            [% END %]

        [% IF company.upstreams %]
        <tr>
          <td>Upstream feeds</td>
              <td>[% company.upstreams %]</td>
            </tr>
            [% END %]


        [% IF company.notes.length %]
        <tr>
          <td>Notes</td> 
              <td><tt>[% company.notes %]</tt></td>
            </tr>
            [% END %]
     </table>

Perl is over twenty years old, but it continues to evolve as a programming language. When Perl 6 is released to the world, we're going to see an incredibly modern language that is adaptable to tackle many of the challenges faced by programmers today including internationalization/localization, Internet networking, object orientation, and more.

But this blog post isn't about Perl 6. This is the first in a series of posts designed to help people ease into Perl as a web development language and is influenced largely by tutorials and seminars I've given in person and online about using Perl as a web development tool.

Perl was embraced for web development in the early and mid 1990s because it had already become a powerful language for text-processing and system administration. As such, it was a natural first choice for developing web applications.

Perl programmers at the time adapted public domain algorithms into Perl code to process HTML forms, but the results were not pretty.

my $hashRef = {};
my $buffer = "";

if ($ENV{'REQUEST_METHOD'} eq 'GET') {
    $buffer = $ENV{'QUERY_STRING'};
}
else {
    read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
}

foreach (split(/&/, $buffer)) {
    my($key, $value) = split(/=/, $_);
    $key   = decodeURL($key);
    $value = decodeURL($value);
    %{$hashRef}->{$key} = $value;
}

sub decodeURL {
    $_ = shift;
    tr/+/ /;
    s/%(..)/pack('c', hex($1))/eg;
    return($_);
}

No wonder Perl is often labeled as "unreadable!"

Thankfully, Lincoln Stein came along and gave us CGI.pm-- a Perl 5 module that makes several tasks the Perl web developer is commonly faced with, easier:

• Generating good HTTP headers
• Parsing form data
• Generating HTML
• Handling file uploads
• Debugging HTML form processes
• ...and more

Web developers often fail to realize they take on a greater level of responsibility when they leave the realm of static HTML pages, images, MP3 files, etc. served up by a HTTP server like Apache and enter the world of dynamic content generation, interactive web applications, and forms and form processors.

Web developers must write code to generate headers for the client that would otherwise be handled by the web server. This includes, at a minimum, a Content-type header, but may also include Content-length, Content-disposition, or Location. Developers must also provide the client with an HTTP status code (e.g. 200, 404, 302, etc.).

You may use CGI.pm using object-oriented syntax or not. I prefer to use the object-oriented syntax because it means I'm in step with other Perl modules I employ in my code.

use CGI;

my $q = new CGI;
print   $q->header('text/html'),
        $start_html(    -title =>   'React' );

my $answer = $q->param('foo');
if($answer eq 'bar') {
    print $q->p("You answered correctly!");
}
else {
    print $q->p("That is incorrect.");
}

print $q->end_html;

This example generates different content based on the value of a form variable (foo) passed in. If this script were named react.cgi then it might be called with a URL like react.cgi?foo=bar.

Notice the code above uses the header() function to generate a valid HTTP header. There are many options you may pass to this function but if you pass one scalar string as in this example, it will use that data to generate a Content-type header.

The above above demonstrates some of CGI.pm's HTML-generation capabilities (the use of the p() function to generate paragraph tags around content and the use of the start_html() and end_html() functions to properly form the beginning and end of an HTML document and generate the necessary title tags in the header.)

The above example also shows how you may use CGI.pm to access form variables (the param() function).

For a couple years, I used CGI.pm quite heavily in my web development tasks. I grew very attached to its HTML-generation capabilities and the fact that if your HTML is generated from CGI.pm code, it will always be valid, start tags will always have matching end tags, and so forth. For example, here is a snippet of code I wrote circa 1999-2000:

print   $q->start_form(
           -method=> 'POST',
            -action => 'add_event.cgi'),
         $q->hidden({-name=>'r'}),
         $q->table(
            $q->Tr(
                $q->td(
                    _event_form($q)),
                $q->td({
                    -valign=>'top',
                    -align=>'left'}),
                    $q->submit({
                        -name=>'c',
                        -value=>'Save Event Info'}),
                    $q->br,
                        $q->submit({
                            -name=>'c',
                            -value=>'Cancel'}))),
         $q->end_form;

There are definite advantages to writing code this way, but the disadvantages are numerous, especially if you intend for your application to be maintained by someone other than yourself. If you intend to have a web designer who has no Perl competence work on the layout, this type of code will have them running away, screaming, and flailing their arms wildly.

That being said, if you need to throw together a simple, self-encapsulated interactive web page, doing everything with CGI.pm isn't a bad way to go.

In most cases, CGI.pm is already installed on your Linux system. To read all about it, type perldoc CGI at your shell command prompt, or go to http://search.cpan.org/perldoc?CGI with your favorite web browser.

What's next?

Well, what's next partly depends on the type of feedback I get from this post, if any. Generally, I plan to talk about templating, database interaction, and more. But, if people have some specific things they would like to see my spin on, I'd be more than happy to oblige.

I've been using the mt-twitter plugin to automatically publish blog articles to Twitter. I find that I get more readers that way than RSS or my newsletter at this point. One problem is that you don't get any good stats that way. I've modified the mt-twitter plugin to use bit.ly now to solve that problem. With bit.ly you can click on the "info" link and get good stats about who clicked from where.

This is the code I added to the _update_twitter function:

 my $bitly = LWP::UserAgent->new;
 my $url_response = 
       $bitly->get("http://bit.ly/api?url=" . $obj->permalink);
 my $small_url;
 if($url_response->is_success) {
    $small_url = $url_response->content;
} else {
   $small_url = $obj->permalink;
}

Of course, you also have to change the line that creates the twitter message to use the new shortened URL ($small_url) instead of the permalink directly.

Tags: blogging perl movabletype

I’ve been following various development mailing lists lately, and I’m seeing a trend of developers starting to bail on Twitter. This is a scary thought, because when the developers bail, so will the users. It all started with a conversation on the Twitter Developers’ mailing list with the subject, “Shame” by a developer named, “nath“, in which he said,

“Well, twitters always down or unusable due to the speed; the api’s
keep breaking and are down just as often; the groups now packed full
of spam which is littering my inbox.

“It’s a real shame to see such a great app crumble and die like this :(”

Alex Payne, a developer for Twitter, responded by saying,

We own Twitter’s speed a stability; my our metrics, it’s been pretty
solid over the last few days.

We do not, however, own spam prevention for this group. That’s up to
Google, and if it’s a hard problem for them, I’d imagine it’d be a
hard problem for anyone.

I go through and clear out spammy posts, but time they reach my inbox,
they’ve reached everyone else’s as well. There’s just not much I can
do about it. Please make use of Google’s “report as spam” features.

After which another developer that goes by “rlansky” responded:

Sorry, but I have to agree with the original author, it is a shame
that the service and the API are so unreliable. The potential for the
services that could be built on an API like the one offered by twitter
are endless. They really are.

Statements like this:

> my our metrics, it’s been pretty solid over the last few days.

don’t do much to boost my confidence. When you make an API available,
you are essentially saying to the world, “here’s our service, come and
build something great on top of it.” You can’t build anything of any
real value or widespread use on something that “has been *pretty
solid* over the last couple days (emphasis mine) .” You just can’t.
You need something that is rock solid all the time.

I’m not trying to start a flame war or bash twitter at all. Like I
said, I think it is a shame because the potential is so great. The
idea is great, the acceptance is great, the use is great, the
possibilities are awesome. But they just can’t be fulfilled given the
reliability of the service as it is today; try to build something on
top of the API that will see wide-spread use and you’ll find that when
you push the gas, the wheels fall off the car… at least that’s been
my experience. It’s been *extremely* frustrating and disappointing.

Peace.

After following a few threads on the Perl development library for Twitter, Net::Twitter, I recently found out that Net::Twitter’s original maintainer too has jumped ship. He has handed it over to a new maintainer, but developments like this are not a good sign for Twitter! It is very clear that frustration amongst Twitter developers has hit a maximum level and I fully expect to see this only increase in the short term.

At the same time, developers like Kee Hinckley are giving advice to Twitter, and they are graciously accepting it seems. Some great tips are being given on ways to enhance the API, and I even suggested they do a public bug tracker which they seemed to like. Twitter clearly doesn’t seem to have enough expertise in-house, although they do keep saying they are hiring. Their jobs page doesn’t seem to have any upper-management positions though which I think is really what they need right now.

I’m very worried for Twitter. As more developers jump ship and work on other platforms such as Plurk and FriendFeed (which really isn’t a direct competitor to Twitter), this great tool is going to be left in the dust with no new development and large networks of people moving elsewhere. Twitter’s largest traffic comes from the API itself, and as that traffic dies down, so will Twitter. Imagine, for instance, if Seesmic were to stop development on Twhirl due to the costs associated with keeping up with API flaws? That would be quite a chunk of Twitter’s users being forced over to the other Twhirl clients, FriendFeed and Seesmic itself - it’s such an easy transition were Twitter support to be dropped! What happens when Twhirl begins supporting Plurk?

Twitter needs to do something, and they need to do it fast. I agree they need to get their infrastructure in place, but before even doing that they really need to put every hack possible in place to keep the API up, keep it working, and work with the developers to ensure they are staying happy. A large revolution is about to take place, and I’m afraid it won’t be pretty.

UPDATE: See the little FriendFeed box below? Click “show” and join the discussion on FriendFeed about this right on my blog! Subscribe to my updates here.

Just yesterday, Facebook announced their second F8 conference, to occur July 23, 2008. This Developer-targeted event is said to possibly include some major announcements, including the new Profile redesign, more information about the fbOpen platform, and most significantly, possibly the launch of their E-Commerce platform. What hasn’t been announced or shared however is the odd timing of the event.

The event occurs right smack dab in the middle of O’Reilly’s Open Source Convention, scheduled to occur for about the past year now from July 21 through July 25. This conference is known as an essential “Mecca” for Open Source developers around the globe, and has presentations from such players as Google, MySQL, Sun, Meebo, and even SixApart. Everyone who is a developer (unless you solely develop for Microsoft) or Sysadmin will be at this conference.

As a developer, this is tough news to hear that Facebook will make me choose between OSCON and them. Frankly, I would by default choose OSCON if I were any smart developer, as I would get more. So why isn’t Facebook just joining OSCON and doing an “F8″ track there? Do they really want to tick off Open Source developers? You better bet that OpenSocial will have a presence there. If Facebook really wanted to target the Open Source crowd, as they have “claimed” to do with their fbOpen Platform and a few other contributions back to the community, they would try to have a presence at this conference and not interrupt it as they are currently doing. I was actually going to go to OSCON to promote my FBML Essentials book to potential Facebook developers for O’Reilly. Now I’m forced with a decision. I’ve contacted Facebook with no response, and I’m getting a little frustrated as a Social Media developer. Which conference will you choose?

Listen to any kind of syndicated talk radio program and you'll usually hear about some companion website the program has. Usually, there are a handful of free things you can get on a program's website, but many of these sites have a pay-to-play members' area where the really good content is. This includes MP3 downloads of the shows, access to live audio and/or video streams, special behind-the-scenes content, forums, desktop backgrounds, etc.

The MP3 downloads are very convenient for people who don't have the luxury of sitting in front of a radio (or driving a car) for a solid three hours while a radio program is broadcast (with advertisements). It's also a boon for people who find radio advertisements annoying.

The only problem with the MP3 downloads is that theme music and produced portions of the program can not, by law, be included in the MP3 file because otherwise the MP3 would be a copyright violation.

Live streams, on the other hand, are not subject to the above described restriction because they're like a broadcast in nature. They're not a time-shift of the original program. So, if you listen to the live stream or even listen to a pre-recorded program as a stream, music and produced segments may be included.

I listen to the Glenn Beck radio program quite often. I used to download the MP3 files to listen to in the car, but it got annoying everytime Glenn and his producers would put together a segment like "Sportscasters at the 2031 animal-human hybrid baseball games", or "The History Of the Democratic Superdelegates" and I would hear Glenn say, "Listen to this... [pause] Oh man! That was great! Wasn't that great, Stu? Oh yeah! Alright! Dan? Wasn't that just the best? Yeah. Oh yeah."

I decided I needed to figure out how to save a stream.

I knew it was possible. Lots of software applications exist for any operating systems that will convert audio from a live stream into a static WAV file or similar. The open source program mplayer is one such example.

Breaking it down

First of all, I needed to figure out how the stream content made its way to my computer.

After I've logged into the Glenn Beck website as an Insider, I can click a link to listen to a stream of a particular hour of the program (or the whole program) in Windows Media format or RealAudio format. I figured I'd have better luck extracting the audio from the Windows Media format, so I went that route. Instead of just clicking the link and letting my web browser find some program that could handle the content, I saved the content to a file and then looked at the file.

The file it saved was a fairly straightforward XML file that looked something like this:

<ASX VERSION="3.0">
  <TITLE>Glenn Beck</TITLE>
  <AUTHOR>Premiere Radio Networks</AUTHOR>
  <COPYRIGHT>Copyright 2008</COPYRIGHT>

 <ENTRY>

    <TITLE>Glenn Beck 1</TITLE>

    <AUTHOR>Premiere Radio Networks</AUTHOR>

    <COPYRIGHT>Copyright 2008</COPYRIGHT>
 

    <REF HREF="mms://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603.WMA?auth=blahblahblahblahblah" />

    <REF HREF="http://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603.WMA?auth=blahblahblahblahblahblah
  </ENTRY>

  <ENTRY>

    <TITLE>Glenn Beck 2</TITLE>

    <AUTHOR>Premiere Radio Networks</AUTHOR>

    <COPYRIGHT>Copyright 2008</COPYRIGHT>

    

    <REF HREF="mms://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603_CLIP01.WMA?auth=blahblahblahblahblahblah" />

    <REF HREF="http://a0011.v67134.c6713.g.vm.akamaistream.net/7/0011/6713/v08060322/glennbeck.download.akamai.com/6713/_!/shows/2008/06/03/GLENNBECKWIN20080603_CLIP01.WMA?auth=blahblahblahblahandblah" />

  </ENTRY>

...and so on.

This XML defines the MMS URLs for each segment of the show. There are several segments each hour. These individual MMS URLs are what I needed to feed to the application that was going to convert the audio stream to a file. In my case, I decided to use mplayer because it's just so good at everything it does!

The command line for doing the stream-to-file conversion looks like this:

mplayer -vc null -vo null -ao pcm:fast:file=dumpfile.wav \
    'mms://a0011.v67134.c6713.g.vm.akamaistream.net/blahblahblah...'

The real magic in the above command is where I use -ao pcm to tell mplayer to use the PCM file writer audio output driver (instead of sending the audio to my speakers).

This gives me a WAV file which I'll want to convert to an MP3 or Ogg-Vorbis file.

To convert a WAV file generated by the mplayer command above to an MP3 file, I use the open source lame tool:

lame -mf -q2 dumpfile.wav GlennBeck.mp3

Or, convert it to Ogg-Vorbis (the completely open and better-sounding-than-MP3 lossy audio codec):

oggenc -q2 --downmix -o GlennBeck.ogg dumpfile.wav

I've now covered the basic mechanical components of converting an audio stream into an MP3 or Ogg-Vorbis file. Next I automate it all.

Automation

Because I'm a long-time Perl junkie, I investigated how I could use a Perl script to act as the glue between the components and get the whole process of capturing a stream and converting it to MP3 or Ogg-Vorbis.

In the above walk-through, I manually logged into the Glenn Beck website with my web browser. To really completely automate this puppy, I wanted the script to log in for me. It didn't take me very long to figure out the Perl CPAN module WWW::Mechanize was what I needed to use.

WWW::Mechanize does several handy things for the programmer. It loads and parses web pages and can follow links, populate forms, and other basic kinds of interaction. It keeps track of its own cookies and session data too.

To get into the Insider area of the Glenn Beck website, members must enter their username and password on the Insider login page.

Looking at the HTML source for this page, I learned the form was named "aform", the username field was named "iUName", and the password field was named "iPassword".

I now had all the information I needed for WWW::Mechanize to log in:

my $agent = WWW::Mechanize->new(
    cookie_jar  => {},
);
   
my $resp = $agent->get('http://www.glennbeck.com/content/insider');
   
if($resp->is_success) {
    $resp = $agent->submit_form(
        form_name   =>  'aform',
        fields      =>  {   'iUName'    =>  'myusername',
                                'iPassword' =>  'shhhhhhhh!', },
        button      =>  'submit');

Walking through the code above: First, I create the WWW::Mechanize object with an in-memory cookie jar (cookie_jar => {}). Next, I use the object to get() the log-in page. If everything works well so far, I tell the object to find the form named "aform", fill in the username and password fields, and submit the form.

One thing I realized as I was debugging my script was that after I logged in on the Insider page, I was immediately redirected to another page. In order for my script to work, it needed to follow the redirect. This was an easy fix:

my $agent = WWW::Mechanize->new(
    cookie_jar  => {},
    redirect_ok => 1,
);

The page I got redirected to has the links on it for the streaming audio, so I'm exactly where I want to be if I want to capture and convert the latest and greatest Glenn Beck Program audio stream.

WWW::Mechanize can find links within the page with a variety of methods. One of these leverages Perl's excellent support for regular expressions. You can also search for links by the order in which they appear. The link I'm looking for looks like this:

<a href="http://www.premiereinteractive.com/cgi-bin/members.cgi?stream=shows/GLENNBECKWIN20080604&site=glennbeck&type=win_show"><img src="http://media.glennbeck.com/images/common/header_media5off.jpg" name="icon5" width="26" height="34" border="0" id="icon5" onMouseOver="MM_swapImage('icon5','','http://media.glennbeck.com/images/common/header_media5on.jpg',1)" onMouseOut="MM_swapImgRestore()" /></a>

So, my script has the following:

$link = $agent->find_link( url_regex => qr/${datestr}.*win_show$/);
$resp = $agent->get($link);

This assumes I have a scalar variable $datestr that contains a formatted date for the show I want to capture.

Originally, I was going to use one of Perl's several XML-parsing modules to make sense of the XML in the stream link, but in the end all I needed was a regular expression to extract the mms: URLs.

my $xml = $resp->decoded_content;
my (@urls) = $xml =~ m/HREF="(mms:[^"]+)"/msg;

This gives me a list of URLs stored in @urls. Now I just need to feed them to mplayer:

$i = 1;
foreach my $u (@urls) {
    my $seq = sprintf("%02d", $i);
    my @cmd = ( 'mplayer', 
            '-vc', 'null', 
            '-vo', 'null',
            '-ao', "pcm:fast:file=${datestr}-${seq}.wav", 
            $u);
    system(@cmd);
    if ($? == -1) {
        print "failed to execute: $!\n";
    }
    elsif ($? & 127) {
        printf "child died with signal %d, %s coredump\n",
        ($? & 127),  ($? & 128) ? 'with' : 'without';
    }
    else {
        printf "child exited with value %d\n", $? >> 8;
    }

    $i++;
}

This little ditty creates an output file for each of the segment streams. These are named something like 20080604-05.wav.

When the loop is finished, I have several WAV files sitting on the disk. Now I need to somehow sew them all together into one big WAV file so I can convert it to an MP3 or Ogg-Vorbis file. For this, I turn to sox. I decided to have the Perl script generate a shell script to run all the sox and lame commands needed.

open FH, ">/tmp/${datestr}.sh";
foreach my $j (1..($i-1)) {
    my $seq = sprintf("%02d", $j);
    print FH 'sox ', "${datestr}-${seq}.wav", " -t raw - | cat >> /tmp/${datestr}.raw", "\n";
}
print FH 'sox -w -s -c 1 -r 22050 ', "/tmp/${datestr}.raw ${datestr}.wav\n";
print FH "lame -mf -q2 ${datestr}.wav ${datestr}.mp3 ";
print FH "--tt \"Glenn Beck Show - $datestr\" ";
print FH "--ta \"Glenn Beck\" --add-id3v2\n";
close FH;

Then, I run the shell script:

system('sh', "/tmp/${datestr}.sh");

Finally, I do a little cleanup:

unlink "/tmp/${datestr}.sh", "/tmp/${datestr}.raw", map({"${datestr}-$_.wav"} (1..($i-1)));

And, I'm done. There are many other ways I could have gone about doing this, but I found a way that worked and ran with it. I'd love to hear from people who have done something similar and how they did it.

I’ve been contemplating for awhile now a good way to share what I know about Social Software Development and helping business owners, marketers, and developers learn how to set up their own social apps. Especially for developers, I know there are many out there looking for howtos and ways to learn more about starting their own App, promoting it, and getting it off the ground. As the author of FBML Essentials, I feel I am well suited for the task so in the next few days I’m going to start doing howtos and overviews on how you can get your own Apps together. If you’re “the business type”, I may get a little technical on you, but I do recommend you keep watching and forward these onto your IT personell - your CIO, CTO, and the like should read these so they can learn what’s possible to integrate into your existing environments. I’ll also try to throw in a little goodie here and there for “the business type”.

So, I’ve created a new category to the right, “Social Coding” - if you want to track just that, click on the category name and add it to your RSS. I’ve also started a new FriendFeed Room where those involved or that want to get involved in Social Coding can discuss, learn, and talk with each other. You can subscribe to that here.

Let’s start by going over the types of sites I could cover. Here are just a few - let me know if you have a particular interest in learning about how to code for any one in particular:

• Facebook
• OpenSocial
• Google Friend Connect
• Twitter
• FriendFeed
• Pligg
• Digg
• LinkedIn
• MySpace
• Wordpress
• MoveableType
• Google App Engine
• Bungee Connect

Stay tuned! I’ll keep posting news and other rants as we go forward - I’ll just be adding in some good howtos at the same time. Oh, and if you’re a developer and would like to do a howto in your preferred language for us, contact me - I’d love to let you do a guest post.

(Sorry it’s been awhile since my last blog - it took me several days to figure out how to get my Flip video imported and exported to and from iMovie. To make a long story short, if you want to export from iMovie and have both picture and sound, you must import your source as something other than MP4 or AVI.)

This is the first article in my “Utah Startup Series“. Starting today I will be circling Utah to find the best and most innovative startups in Utah, and featuring them here on Stay N’ Alive. If you have a hot startup (early to even late stage) and would like to demo for me what your product can do, please contact me - if I have the time and like your idea I’d love to come out and take a look at it!

While at Web 2.0 Expo I had the opportunity to meet with Bungee Labs, a local, well funded Utah company who had “Platform as a Service” down before Google even started thinking about their App Engine. In our meeting they demoed their Bungee Connect “IDE” (written entirely on the web). You can see the video below.

My thoughts - you have to see this stuff in person to understand the full ramifications of what they’re doing. One of the cool things about their service vs. Google’s is they actually integrate with Amazon’s EC2 service (which was announced during Web 2.0 Expo), so you can actually host your other stuff on Amazon’s EC2 platform with the same licensing as your Bungee Connect account. Their licensing structure is very appealing as well - Bungee only charges based on the number of registered user sessions using their platform, not traffic, not bandwidth. If I understand correctly, it’s all based on the number of users actively using your application on their platform. For Facebook and Social Media developers this is appealing, as most Applications are rated based on Application use, not number of users or traffic. With Bungee you only pay for the users that actively use your system.

Overall, the guys at Bungee were Rockstars at Web 2.0 Expo. With their announcements about EC2 integration, flexible licensing terms, features on TechCrunch, EWeek magazine, and a dozen other publications, you can bet Google has a watchful eye on them. Ironically, it was interesting seeing Kevin Marx, head guy over the OpenSocial (and other) efforts at their party on Thursday evening.

Bungee will be presenting at our Social Media Developers meeting this coming Tuesday, showing us a simple “Hello World” example on how to build a Facebook App using their platform. Follow me on Twitter and if we can stream it live you can watch it via my Ustream channel. After demo I may just write my own Facebook App to try out their system - it should be interesting.

Bungee Connect Demo - Web 2.0 Expo from Jesse Stay on Vimeo.

I've been trying to use Capistrano for application deployment over the last few days, writing rules to do some common tasks, figuring out how it works, etc. One problem I ran into is that I have a private CPAN bundle that I use to ensure a machine has all the right Perl libraries when I deploy to it.

The problem is that CPAN is often run interactively and so module writers often assume the user will be present. That means that it stops in the middle and asks questions about skipping tests, etc. I searched for a while to figure out how to get a default answer to questions. It's not Capistrano's job and CPAN didn't seem to have a configuration option that worked. Turns out it's in MakeMaker.

MakeMaker is the Perl library that the CPAN modules use to automate the build process. There's an environment variable called PERL_MM_USE_DEFAULT that when true causes the MakeMaker prompt function to assume the default answer.

So, here's the task from the capfile I came up with.

task :load_bundle, roles => :local do
     run "cd /web/lib/perl/etc/kynetx-private-bundle; 
          sudo perl -MCPAN -e 
             '$ENV{PERL_MM_USE_DEFAULT}=1;
              install Bundle::kobj_modules'"
end

This works fine. Of course, you also need to make sure the account you're using for installs can sudo without a password or this will fail as well. Maybe there's a better way to do sudo inside Capistrano? I'd like to know about it.

Tags: kynetx sysadmin ruby perl

I have disliked the Twitter syntax since I’ve been on it (you can find me via @JesseStay on Twitter - go ahead and follow me!). As a long-time IRC user, everything seems backwards! I have often referred to Twitter as “IRC 2.0″. I’m not sure I can fully embrace that concept though.

For those unfamiliar with IRC, it predates even instant messaging. It brought out the original concept of a “chatroom”, and exists even today on various servers throughout the world. Ustream.tv currently uses it for its users’ channel chatrooms. It is the home for almost any “live” activity of any open source project (log into irc.freenode.net to see - I’m often in #utah there, as well as recently #codeaway). Traditions have been established, and virtual friendships have been bonded. In many ways it could have been the original concept of a “social network”, the first concept of linking friends together in a single place on the internet.

I was at a Perl conference just last year, and was happy to see the #YAPC chatroom in irc.perl.org open during the banquet. We had a ton of fun with that! Now, just this year, when I go to conferences, I see speakers leaving up Twitter, and answering questions via Twitter. The two seem to be serving similar purposes, in different ways.

That’s why I was astonished when I got on Twitter for the first time, and started seeing public messages directed to individuals with “@” signs in front of them! Is there a source for that that I’m not aware of? I know of no known documentation that Twitter themselves created to establish that tradition. In IRC you simply type “username:”, and then your message, and it gets highlighted in that user’s chat window in most IRC clients. Better yet, I can start typing the username and it tab-completes. You can’t do that in Twitter. That tradition and method has been around for years, yet Twitter seems to break the mold for some reason.

IRC also supports commands - I can type “/nick newnickname”, and it switches my username, automatically! It’s a basic standard that all clients support, open, and available for all to use. Twitter I have to go entirely to their website to do anything, and it’s extremely limited in what you can do. To direct message someone on Twitter, I have to type, “dm username message”. In IRC it’s just a simple command, like all other commands, and I can always type, “/help” if I don’t know what the commands available are. I simply type, “/msg username message”, and it messages the user, and again, it tab-completes the username!

Why couldn’t Twitter just use the IRC standard in their platform, and then expand upon it to improve the IRC standard and bring it to a mobile world? By all means many of their scalability issues may have been taken care of had they done so. Not just that, but they would now be able to support groups, and less development would be needed to manage their platform. Twitter says they have an open API - I question that openness. It’s not based on much of an open standard, and IMO, it’s causing them problems now because of it.

Looking to start a project? Always look at the open solutions that are out there first, then build upon them - you’ll have much fewer headaches if you do.

(Photo courtesy GapingVoid.com)

I’ve been analyzing various Social Applications Analytics tools lately, and have recently stumbled upon Sometrics. Sometrics handles full Analytics for your Facebook, Bebo, and MySpace applications, and will actually utilize the Facebook API to retrieve demographic info about those visiting your Application. As I examine the other Analytics solutions for Facebook and other Social Network Applications, I’ll try to post my findings of their strengths and weaknesses here, OpenSocialNow, and FacebookAdvice.com. If you’re not a techie, you may want to skip the next part, or forward it onto your IT department.

One thing I noticed about Sometrics is it seems to only provide code to paste on your Application pages for PHP, Ruby, and ASP.net. The code they provide is relatively simple, but in case you’re wondering how to do it in Perl, here is how I did it in Template Toolkit under Catalyst on Perl:

Enter this on all Application pages (I do it in my “footer” file):


[% IF Catalyst.request.param("installed") %]

<fb:iframe width='1' height='0' frameborder='0' src="http://halo.sometrics.com/fb_tracer.html?src=fb&installed=1&session=%7B%22session_key%22%3A%22[% Catalyst.request.param("fb_sig_session_key") %]%22%2C%22uid%22%3A[% Catalyst.request.param("fb_sig_user") %]%2C%22expires%22%3A0%2C%22secret%22%3A%22%22%7D&t=[% date.now %]"></fb:iframe>

[% ELSE %]

<fb:iframe width='1' height='0' frameborder='0' src="http://halo.sometrics.com/fb_tracer.html?src=fb&session=%7B%22session_key%22%3A%22[% Catalyst.request.param("fb_sig_session_key") %]%22%2C%22uid%22%3A[% Catalyst.request.param("fb_sig_user") %]%2C%22expires%22%3A0%2C%22secret%22%3A%22%22%7D&t=[% date.now %]"></fb:iframe>

[% END %]


Then add this in the “post-remove url” subroutine for your Applicaiton (or create one and add the URL in your App’s config):

=head2 remove

  Page that handles App removal

=cut

sub remove : Local {

  my ( $self, $c ) = @_;

  if ($c->req->param(”fb_sig_uninstall”)) {

    $c->res->redirect(qq{http://halo.sometrics.com/met.gif?a=u&app=}.$c->req->param(”fb_sig_api_key”).qq{&uid=}.$c->req->param(”fb_sig_user”).qq{&age=&sex=&city=&state=&country=&friend=&src=fb});

    $c->detach();

  }

  return;

}

I’ve been following the issue list for Google App Engine (just realized it doesn’t have an “s” in the official name), and the two top issues are a dead heat between Perl and Ruby in the requests to have Ruby or Perl support. Ruby, as of this writing is at 361 votes, and Perl is right on it’s tail at 347 votes. Perl until a few hours ago was pretty far ahead of Ruby. PHP is only at 70 votes, and Java is at 247 votes.

Does this mean Perl is making a comeback? Did we ever really leave Perl? As an avid Perl developer this makes me happy, as Perl can do anything Ruby or even Rails can do, and even more (Perl XS and tie-ins to C are very powerful!). All of my current Facebook Apps and OpenSocial Apps I do in Perl on an MVC Framework called Catalyst - it’s very scalable! It never made sense to me when people said that “Perl was Dead”. Is this just a reflection of the type of Audience Google supports, or is it reflective of what new media developers are actually developing in?

I’m hesitant in posting this, as it could bring more Ruby voters to the mix, but hey, let’s keep it fair. If you want to vote for Perl, click on the star here. If you want to vote for Ruby, click on the star here. Not a developer of either? Then you’re on your own.

I wonder how Python would fare if it got equal treatment.

UPDATE: Within just a day after this post things have gone back to how I would expect them to be. Java has a strong lead over all the others, followed by PHP, then Ruby, then Perl. Perhaps the issues just needed a little exposure. Based on the interest, Perl is still far from dead though.

Okay, so I was wrong - it was worth a try. I do still expect more large announcements related to Social Media from Google. Just recently, Google announced their “Google Apps Engine” (will it be nicknamed, “GAE”?). It is essentially a competitor with Amazon’s EC2, S3, and SimpleDB, but at a much higher level. You’ll be required to interface with the service via the Python Programming language at first, but it is intended to make scalability and server set up much easier. Google does say that the underlying infrastructure is entirely language neutral, so we should expect more languages in the future. The advantage over Amazon is Google takes care of all the server set up for you - this is essential for a small business that can’t afford to hire an expensive Linux Admin as Amazon requires.

The Service is only available to the first 10,000 developers that apply at http://code.google.com/appengine/, and will be available starting at 9pm PST tonight. You can read more at Venturebeat and TechCrunch here and here.

I just caught this article from Mashable and I just had to pipe in. In the article, Mashable’s Kristen Nicole claims Guy Kawasaki paid too much for the development of AllTop, at $10,000. They compare it to Askablogr.com, claiming Chris DeVore only paid $7500 for the development of Askablogr, with its rich feature-set.

I was blown away by this! Not that Guy Kawasaki paid $10,000, but that Chris DeVore only paid $7500 for Askablogr. Now, I don’t know Chris, so take this with a grain of salt, but some call it a deal. I say he’s a cheapskate! For something that will be your primary revenue source and your main line of business, $10,000 for something like Alltop.com is a steal! The fact that Chris DeVore only paid $7500 for his development means he’s either hiring offshore, doing the development himself (in which those costs are way under-inflated), or he’s very much underpaying a bunch of gullible developers that probably don’t believe much in the product they’re working on.

As a business owner, when supporting a technology-based business, it is of utmost importance that you put your developers and IT staff at first priority. They are your bottom-line, and should be the superstars of your business. You have to keep in mind that for top notch developers and technology, you’re competing with the likes of Google, Facebook, Yahoo, and others to get the best talent. By not paying your developers, you will either a) lose your developers very quickly, b) have a revolution at one time in your future and your developers will all back out on you in rapid succession, or c) not get the best work and skills you could be getting, and you’ll definitely run into scalability issues as your site grows in the future.

I recently finished the book, “My Startup Life“, by Ben Casnochas. I bet Guy’s read it and Chris hasn’t. In it, Casnochas talks about the lessons he learned by not paying his lead developer well. He quickly had threats of the staff to leave, and they quickly ran into scalability issues due to the unexperienced offshores they were hiring overseas. In building a technology-based business it is of utmost importance that you pay and treat your IT staff well or it will come back to bite you in the future.

So, Kristen, I say Guy is the smart one in this case. I am willing to bet his site scales better, his developers are happier, and more likely to work with him in the future. Guy’s likely to get millions for Alltop.com in the future, should it succeed, so $10,000 is a very small price to pay to get good developers on staff.

I wrote a quick little perl script/irssi plugin that allows you to update twitter from irssi. It also has autocompletion for names from your friends and follower list. You can get it h