Utah Open Source Planet

A few weeks ago Yammer walked away with top honors at TechCrunch 50. I'd been hearing about it, so last week I went over and signed up. I'm really liking it.

Yammer is Twitter for work. The first person to sign up using an email from a particular domain establishes a sandbox for yammers from people in that domain. Since I signed up using my kynetx.com email, Yammer automatically created a domain for Kynetx and made me an administrator. Very low friction.

I sent out invites and soon had the whole Kynetx gang yammering away about work. Only people within your domain (i.e. those with emails you control) can see what the others are saying. Yammer has a Adobe Air client that makes keeping up with what folks are saying easy (you can also use the Web). In addition, Yammer has a number of features that are geared for business like the ability to input and display detailed contact info and organizational information as a crude social network.

We've been using Yammer for a week now and everyone feels it's been a good addition to our communications patterns. This is especially true with a distributed workforce like we have now.

So what of Twitter? Most of us still use Twitter too. Twitter is my public space. Yammer is my work space. That separation has real value and enables conversations and uses for Yammer that just wouldn't happen on Twitter.

We've made real decisions over Yammer that would have previously taken a phone call or email chain. Generally people are pleased with the connectivity it provides. For us, Yammer is a keeper.

Tags: yammer twitter microblogging work kynetx

We promised we'd at least try to get some audio and video from UTOSC up as quickly as possible, so I'm sitting here at my computers waiting for my Windows computer to write a modified 23GB AVI so I can start editing the video for the keynote presentations tonight.

So, while that slow process continues, I'll write a little about how today went.

Well, for me, it began very STRESSFULLY!

I went over to Salt Lake Community College (where UTOSC is being held) on Wednesday afternoon to get things set up for shooting video starting Thursday and to help with other UTOSC-related tasks as I am a UTOSC core team member. At 4:30 p.m., a client called me to tell me their server just went down. Right after the call, the Nagios alerts came into my phone saying the same thing. Nick was with me, so I sent him out to get their server back up while I continued setting up equipment. I figured it just needed to be powered back up or something, but we weren't so lucky this time.

This particular server has been pesky and super-sensitive ever since we installed it, making it an annoyance for both us and the client. We weren't really every sure what the problem was, but I strongly suspected the motherboard was just bad even though it worked most of the time.

Nick couldn't get the server to do much. It would love the RAID controller BIOS and then reboot, or it wouldn't display anything at all on the monitor. Finally, I told him to just remove the server and bring it to me and I'd work on it later at home.

Later, when I got the server to my house, I could not get it to do anything. It would power up, but would not POST. I tried all the usual tricks: removing the power cables, disconnecting the motherboard power connector, resetting the CMOS power jumper, chanting a voodoo chant. Nothing worked, so this morning, I made replacing the motherboard my first task. I had hoped I'd be able to get it done quickly and still make it to SLCC to be of some help in the preparation for UTOSC to start at 12:30.

I made it to Universal Systems around 9:00 and they had one socket 1207 motherboard in stock, a Supermicro H8DME-2 dual-processor board. I guess I was pretty lucky they had one. I knew USI was more of an Intel shop, but I thought they'd have more than one AMD board for sale. Lucky for me, they had one. It wasn't cheap, but it was a Supermicro so that's generally a good thing.

I took the server and the new board back to the office and proceeded to install it. The Supermicro board was an EATX board which means it's about as huge as a motherboard can be. The I-Star case I was installing it in could take an EATX motherboard, but it was a tight fit. It took me about an hour or so to get the new board in, everything connected, and powered up. The box didn't have a manual in it, so I downloaded a PDF and printed off the necessary pages for jumpers and connectors.

The LSI Logic RAID controller really slowed down the boot process. I eventually just yanked it out of its PCI-X slot so I could get through BIOS and boot-up issues without waiting.

The Supermicro motherboard had a different onboard SATA chipset than the old board, so I had to install a new initial RAMdisk (initrd) for the Linux kernel. The server was running Fedora Core 6, which I didn't have any media handy for, so I downloaded a rescue disk ISO and burned it to a CD. I ripped a CD drive out of an old desktop so I could boot to the rescue disk. This, of course, all took a little time... more than I anticipated. Finally, I got the system booting by getting the new initial RAMdisk installed by way of the rescue CD. Then, I realized I had to reconfigure the networking for the server because it used a bonded ethernet configuration. All the ethernet addresses would be different, so I had to go through a tedious process of making Fedora Core forget the information it had stored about the previous ethernet ports and learn about the new ones. Finally, I had a system that was ready to go back to the client and it was about ten minutes before noon.

Things went relatively well at the client's office. I had to do a couple other little things to get things working the way they should, but I was out of there shortly after 12:30. All the hustling made me a little shakey, so I hit a local Maverik and got some hot cheesy bread. I made it back to SLCC a little before 1.

Matt Asay was well into his presentation, but Nick had both cameras rolling and I stepped in on one and took control.

Everything else throughout the day went, I thought, very smooth. We shot video for Nathan Blackham's Nagios presentation and would have shot video for Jared Smith's Asterisk presentation, but it got moved to Friday. As a result, we had a little extra time and I would have rounded up a couple people to do some on-camera interviews, but I didn't bother to shave and looked like a wild man, so we didn't do that. Instead, we loaded up the equipment and moved over to the Student Center to get set up for the evening keynote presentation.

It was good we headed over there early. It was more work than either Nick or I expected packing our equipment up, moving it, and setting it back up, so we learned a lot from that.

Dinner was pretty good. More people should have attended the dinner and the keynote presentations. A lot of people did, but I still saw empty chairs. It seems like the SLCC students didn't make it out en force to the dinner and they should have. Free food!

I got home a little after 10 p.m. and started working on this video. Now it's about 12:30 and I'm done talking about my day and this video conversion thing is still going. We'll have to see if I have the patience to get this out tonight. If nothing else, I'll get audio from the presentations to someone to make them available.

Imagine you have a repository. At time t₀ someone copies the code from the repository, starts another repository from there, and commences development. You also continue development. At time t you want to pull in the changes the other guy has made.

One approach could be you simply merge other guys code into yours with one big commit calculated from time t₀. Now suppose you want to keep other guy’s history from t₀ on, or even worse, you want to pull only some changes and you want to push some of your changes to other guy.

This is a true story. It all started with CVS. A webapp was developed, cloned and branched. Now there are 4 of them, mostly the same, but subtly different, and each with it’s own CVS repo. Each one at some point in time is identical to another, but the history behind those points in time are different, thanks to copy and commit cloning.

So after converting the CVS to Darcs, I came up with a way of tricking darcs into treating time t₀ in each repository as the same t₀, thus enabling cross-pollination, and here I describe that method.

First you find t₀, and tag liberally. By liberally I mean twice. I’ll explain why later. Of course you put the same two tags on both repos, both at the same point in time. Then you do a darcs opt --checkpoint on each so you can do a partial get on them.

Now you have to decide which repo you’re going to keep, and which one you are bringing in. You probably want to keep the one with past history, and bring in the one that did copy and commit cloning. Do a darcs get --partial on each repo, keeping track of which is what. I like to use a for the destination, and b for the bad repo.

Now go into b/_darcs and we’ll play around with these guts to make it think it’s a, but we want it to have it’s future patches first. So we take a snapshot before pulling the other stuff. I just made a temp directory and did cp -r inv* patches temp, these are the three things in the _darcs directory we care about. Here’s where the liberal tagging comes in, the partial at the later tag only depends on the first tag, so that makes your inventory and patches much simpler. Now you’re ready to pull in all the subsequent patches from the cloned branch.

Once that’s done, you go back into b/_darcs, this time you take anything in temp/patches and temp/inventories and remove the corresponding files from patches and inventories. Then you replace those files with the files from a/_darcs/{inventories,patches}. These two steps I accomplish like so:

ls temp/inventories | while read i; do rm inventories/$i; done

ls temp/patches | while read i; do rm patches/$i; done

for f in inventories patches; do cp ../../a/_darcs/$f/* $f; done

One last step remains. Now you edit inventory and replace that very first patch number, the one that’s a tag, with the one that’s in the inventory file from a. Now b effectively has the exact same past history as a, and has all it’s future patches, and now you can share between your repo and other guy’s repo freely.

One might think this is an uncommon situation, but I can’t help but think it’s more common than I think… Hopefully this guide will save others the headache of trying to wrap their minds around the darcs inner guts, as well as save me some headache should I ever encounter this again (which I’m pretty sure I will, in only a few weeks!).

We’ve found and hired a general manager as my replacement at work. I started training him yesterday but have yet to start looking for my own new job. If you have or know of any openings for a tech savvy business guy, please let me know.

As many casual readers of the Fozzolog may know, I’m now working for a new company called KnowledgeBlue. Things are done differently at KnowledgeBlue and it’s not unexpeced that one company would do things differently than another, but the way things are done at KnowledgeBlue has been a bit of a shock to my system. I think my Iodynamics colleagues are experiencing similar feelings.

Robb and Chris are veterans of the corporate IT world and I don’t mean the IT department of your standard local business but the IT world of a large corporation listed on the Fortune 500 list. As a result, they bring an amount of discipline and regimen to workflow that I’m not acustomed to working with.

That being said, I’ve been learning a lot and enjoying it immensely. (I’m not just saying that to be a brown-noser.) While the regimented workplace is, without question, overkill for the size of company KnowledgeBlue is right now, the company will be growing significantly in the coming months. Without these policies and procedures in place, it would be easy for things to get out of control.

In all honesty, the structure and rigidity has been frustrating and the amount of work and responsibility piling up on my desk has been putting my confidence through its paces (pacing through the doldrums, perhaps?) That being said, I haven’t been challenged like this in a long time and despite my confidence scraping the bottom of the proverbial barrel, those I’m working for are supporting me and expressing confidence in me. So, there’s that.

Project planning

One thing I’ve been requested to do is do project and resource planning for some of the work that is upcoming. In the “corporate world“ the de facto computer application for such a task would be Microsoft Project. Anyone who knows me knows... that ain’t gonna happen. Fortunately, there are many alternatives to choose from in the immense software catalog of the Open Source universe.

Here’s a preliminary rundown of what I’ve come across.

Kplato was one of the first applications I looked into and I actually used it for a few days before moving on. It does a fair job of generating WBS (Work breakdown structure) codes and tracking who is assigned to what tasks. The one thing that kind of became a dealbreaker for me is that kplato doesn’t seem to be able to associate prerequisite task information. That is, there doesn’t seem to be an easy way to say “Task A must be finished before Task B can begin.” You can only specify task starting and ending constraints by date.

Planner, formerly known as MrProject, is a project management app that has been around for a while in various incarnations. Planner has most of the features I’m looking for and, unlike kplato, does support the concept of “predecessor tasks.” Additionally, Planner features the ability to import data from Microsoft Project XML format files. There is a port of Planner for Windows also available. One thing that bugs me about Planner is that it’s a GNOME application and suffers from the UI braindeadness that accompanies most GNOME applications. Otherwise, it’s probably one of the most capable of the applications I’ve looked at.

TaskJuggler is a bizarre entry to this field of applications. It completely deviates from the project management software genre. Users are required to enter information about projects, tasks, resources, etc. using a programming language-like syntax into a text file which is then compiled by the application. The authors of TaskJuggler claim their program is leaps and bounds better than “traditional” project management software. I’m a bit reluctant to go down that road. I should definitely come back to TaskJuggler at some point. For now, I’m sticking with the traditional applications.

OpenProj is the first Java-based desktop app of the mix I’ve looked at. It features nice import/export compatibiliy with Microsoft Project and its cross-platform nature, thanks to Java, also makes it attractive. It also boasts task predecessor support. It’s not very easy to use compared to the others I’ve looked at and it runs slow. It does have a lot of screens and reports, though.

GanttProject is another Java desktop application I looked at. It offers good import/export capabilities for people who are dealing with Microsoft Project data. It doesn’t offer quite as many views as OpenProj, but tries to make up for this with lots of export options (HTML. PDF, CSV, PNG, and JPEG). GanttProject also supports the concept of prerequisite or predecessor tasks. One disappointment, however, is that GanttProject doesn’t seem to let you get any more precise on the length of effort of a task beyond the number of days. For example, you can’t express that a task is only going to take 2 hours to complete. It also doesn’t handle some errors very well.

I’d love to hear about other applications that I haven’t listed above.

On my way home from work today, I stopped at the Sinclair gas station up the street to fill up my cup with a fresh batch of Diet Mountain Dew. I was running a couple minutes later than I would have liked, but still felt an obligation to be polite and minimally sociable with the clerk. In my haste, however, I completely munged my departure remark and perhaps in considering saying “Have a good one,” or “Take care,” I blurted out, “Take a good one, man,” as I walked out the door and got in my car.

“Take a good one?!” I said to myself aloud as I got in my car. *sighs*

Sean O'Steen, one of the hard-working audio editors on IT Conversations, had an article at Web Worker Daily on co-working in Berkeley. Sean's article discusses the concept of co-working in general and the Berkeley Coworking site in particular.

If you're not familiar with it, co-working is the trend for developers, writers, independents, and other professionals to spend at least part of their day in a shared, public space. Starbucks, Borders, libraries, and other places that have Wi-Fi have been popular hang outs for some time and now people have started to create specific co-working spaces like the Berkeley one.

Of course, this trend has been enabled though a number of technologies: laptops, wi-fi, email, IM, cell phones, VoIP, and even things like Twitter.

The draw to co-working is being able to work without the isolation that just working from home can bring or the structure that a standard office entails. I'm surprised at the number of people I know who spend at least a few hours a week working from a Starbucks or the library. Personally, I like to spend some time working at Borders when I can. I see the same group of people there all the time.

Tags: itconversations work

Those who attended the dinner and/or first keynotes from tonight’s Utah Open Source Conference session may have seen a press release on display at KnowledgeBlue’s exhibition area. It announces the merger of Iodynamics into KnowledgeBlue.

So, no more secrets! I’m a KnowledgeBlue employee now as well as Mike, Adam, and Thom - my chums from Iodynamics.

KnowledgeBlue is based in Salt Lake City and is in the business of (my words here) gluing together open source software packages into systems that offer great value to businesses.

Another thing KnowledgeBlue is letting people know at UTOSC is that we’re hiring. If you’re a can-do person with a firm understanding and appreciation of Open Source software, check out the jobs available on the KnowledgeBlue careers page.

Occasionally I need to play with an experimental machine using VMware Workstation on my Linux host. The virtualizaton already sanboxes my disks, memory, and other resources. However I only have 3 choices for networking: Bridged, NAT, and host-only.

• Bridged is often unacceptable for experimentation due to it’s unrestricted nature. It also does not pass through the standard netfilter interfaces in the kernel to be filtered.
• NAT is often unacceptable because I cannot control it’s filtering policies. It runs a separate daemon to handle the address translation. This blocks many of my filtering options in iptables.
• Host-only is almost never acceptable since I rarely do any experimentation that doesn’t require a network interface to my network outside my machine.

My favorite solution is to use the host-only networking option, and configure my host machine to NAT and route the traffic. This gives me extreme control over the network policies, addresses, etc. All with a familiar iptables interface.

• Configure your Virtual Machine to use Host-only networking
• Enable IP forwarding on your host.

  echo 1 > /proc/sys/net/ipv4/ip_forward

• Add the address you want your virtual machine to use on your network as an alias to your real interface.

  ifconfig eth0:0 10.49.220.40 netmask 255.255.252.0

• Add a NAT rule with iptables to translate packets to this new address.

  iptables -t nat -A POSTROUTING -i vmnet1 -o eth0 -j SNAT --to-source 10.49.220.40

• Add any rules you wish to impose to the FORWARD chain in the default filter table. Example here defaults to DROP all packets, but allow DNS to a DNS server, and all traffic to a host for the experiment.

  iptables -P FORWARD DROP
  iptables -A FORWARD -d 10.49.1.25 -p udp --dport 53 -j ACCEPT
  iptables -A FORWARD -d 10.49.1.26 -j ACCEPT

Now your experiment will come from your chosen IP as you would have wanted with bridged mode, but you get the awesome power and flexibility of filtering it via iptables. Great for playing with Windows and it’s included vulnerabilities.

I discovered today that I left my procmail deliveries logging all kinds of information. I had logs that went back a month and a half. I thought why not parse them up, and generate some stats.

My procmailrc sorts most of my mail into folders for me. When I was writing the script to parse I decided to categorize my folders to make the statistics more meaningful. This leaves me with 4 types of mail: work (automated reports, logs, and such), spam (SpamAssasin, and discarded mail), lists (mailing lists), and Inbox (everything else).

These stats turned out to be quite interesting, at least to me. Since I am the sysadmin for an ISP, I get tons of email. I get the output for any and all cron jobs, interesting snippets of logs, and all mail addressed to common aliases (postmaster, root, webmaster, abuse, daemon, security, etc). This will cause my work category to be quite large. You can see that my work mail accounts for more than half of all deliveries. If you leave out the work category, my spam accounts for about 80% of all of my email, and that doesn’t count all the crap that SpamAssassin or my own filters don’t catch. Holy cow. Spam is a huge problem.

The big dip this week is caused by my experimentation with new anti-spam techniques. I tried out OpenBSD’s spamd. It is amazing. It reduces spam quite a bit, as you can see here. It would show even better results, but I only used it on one of several balanced incoming mail servers. It is a great implementation of greylisting. However, this technique causes some legitimate mail to be delayed by 5min - a few hours. We had a few complaints from customers about delayed mail, so I had to turn it off. I highly recommend this technique for anyone who is battling spam, doesn’t have extremely picky users, and don’t mind slightly delayed mail from time to time.

Tags: procmail, stats, spam, greylisting

The flat green plateau was a bittorrent I left on, and the massive mountain is Tor.

Last week I used quite a bit of bandwidth on my colocated server. Part of it was due to leaving a bittorrent of a few episodes of “Weeds” running for a few days. Those sure are popular, but I have the max upload rate limited to 800KBps. Not enough to cause problems, but enough to share my generous amount of bandwidth.

While Bittorrent can eat up quite a bit of bandwidth, it wasn’t the culprit. The problem was Tor! I misread the documentation for BandwidthRate and BandwidthBurst. I thought it was in bps (bits), but, in fact, it was in Bps (bytes). Whoops! When I was going through the normal everyday sysadmin duties I noticed that our primary backbone link was a lot closer to being full than it normally should be. A little poking around and I found that my own server was eating up about 14Mbps. I shut it off immediately, and headed for the documentation. To my surprise FAQ 5.17 is very clear about the units for the BandwidthRate. I am a total idiot. Config fixed, tor restarted.

I am not one of those privacy nuts that won’t do anything that isn’t anonymized or encrypted. In fact I tried using tor myself for about 2 minutes. It was painfully slow. I don’t really care who sees most of my traffic, and the traffic I do care about is already either SSL’d or ssh tunneled. However, I understand the occasional need for tor, and it’s obvious benefits. Since I have so much bandwidth available that no one will be using, I thought I would share. Ain’t it nice to be sysadmin of your own ISP.

Interesting note: So far in the first 10 days of this month I have used 981GB on my server. Holy bejeezus