Utah Open Source Planet

Image via Wikipedia

I've been heads down for the last few weeks getting a project out the door for a new customer. As I mentioned, this involves creating a virtual appliance. I decided, due to the circumstances of this deployment that the best option was the build an appliance factory that is capable of churning out new virtual machines at will. I'm going to describe how I did that in this post.

There are bascially three steps to creating a new image that runs the Kynetx Network Service (KNS):

1. Create a new virtual machine
2. Install packages and Perl libraries, create users, and otherwise configure the machine to run KNS
3. Deploy the KNS code and test it

I was exporing Kickstart files for automatically installing Fedora and CentOS when someone pointed me at Cobbler. Cobbler is a Linux installation server that is simply amazing. It includes templated kickstart files, DHCP and DNS servers, the ability to manage multiple distros and repositories, and a database for keeping it all straight.

You start by importing distros and images, then define profiles that combine those with kickstart files, and finally create system definitions for each machine refering to profiles. I pnly needed one distro, one repo, and one kickstart, so I ended up with multiple systems hanging off of one profile. Once that's done, a command called koan (kickstart over a network) is used on the Dom0 machine to create virtual machines as defined by the system definitions cobbler.

I carefully edited the kickstart file to create just the machine I wanted with the right packages installed. At this point, I was building new VMs and taking them down 20-30 times a day as I tested this. That's the beauty of automation--tacking up a machine is just dirt simple.

I was lucky that I'd already invested considerable effort in Puppet recipes for building the environment that KNS need to run, so the second step was almost done. In fact, with just a few edits, I had Puppet building the new VMs up.

The third step was also one that I'd spent some time on. I have a custom deploy script (in Perl) that deploys KNS code based on server role and takes care of all the little details like setting up the configuration files for the various servers.

Every system is slightly different, but I think there's a definite distinction between machine setup, system configuration, and code deployment. The first creates a fairly standard environment, the second configures it to a specific purpose, and the third manages the code.

Some thoughts on all of this:

• Some have asked "Why not put the code in Puppet (i.e. why use a deployment system)?" My answer is that code deployment is a dynamic process that I want more control of than puppet's automatic configuration provides. You could probably press Puppet into this, but it didn't seem to fit for me.
• I had to create a simple YAML-based configuration file for KNS to pull everything together. YAML was the right answer for this. I chose to put that configuration file in Puppet, but I think I'll pull it into the deployment process in the future.
• One missing piece is a database that everything can read system configurations from. Cobbler provides a light-weight one that may serve our purposes for a while, but something like iClassify is more flexible. Right now there's system information in Cobbler, Puppet, and the deploy script. There's a way to put additional attributes in Cobbler that we could use in other places.
• All of this--Cobbler, Puppet, and the deploy script--were installed and running on a virtual machine that we call the factory. That one image, once installed in Xen is capable of creating as many copies of each type of machine we run as needed.
• This can all be done on physical boxes too, of course, but I prefer the flexibility of virtual machines--even when only one will be running on the physical hardware. They can be moved, replicated, and managed with a lot more ease that physical hardware. Plus I have the ability to fire up new ones for QA or whatever without buying and installing new physical hardware. When a 8 core, 32 Gb box costs $4K, you can amortize that investment a lot with virtual machines.

Startups need to be lean. Achieving that goal in a compute-intensive business requires automation. Fortunately with tools like Cobbler and Puppet, automating the build-side of your infrastructure is not only possible, but fairly easy. We manage several dozen machines with only a few hours a week of effort. What's more, adding a new box for load or experimenting is as easy as typing a few commands and waiting 20-30 minutes.

Tags: kynetx system+administration cobbler puppet

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the nineteenth installment. You may find my efforts instructive. Or you may know a better way--if so, please let me know!

I have a theory that the third deal matters more than the first two. Here's why.

The first time you sell your product--your first deal--is always exciting. But let's be honest, it could be a fluke. If you beat the bushes long enough you're likely to find someone who'll buy almost anything.

The second deal feels good because you at least can convince yourself that the first deal wasn't an accident.

But the third time you sell your product you have confidence around a few important things:

• Proven repeatability - to get to the third deal you've proven that you understand what your selling and you're able to explain it in a way that people connect with.
• Turn the crank - at this point you ought to be able to "turn the crank" operationally and deliver. If you're still doing one-offs by the third deal, you need to ask yourself what will change by the fourth, eighth or 100th deal? You can't achieve scale without operational excellence.
• Know your price - On the first deal you're always a little unsure of the price you've set. Is it too high? Too low? Will you get laughed out of the room? By the third deal, you can go into pricing discussions with confidence. After all, two other customers have paid it--why won't everyone?

I've found that to get any deal you usually have to put your ego aside. The sweetspot is when you've found (a) something you're good at, (b) something you like to do, and (c) something someone will pay for. A deal implies (a) and (c). If you have to cave at all on (b), then your ego's likely to get in the way of the deal.

Putting ego aside is not always easy for techies to do. After all, you've spent years working on this and generally have dreams and even fantasies about how people will use it. Take a deep breath and realize: someone's willing to pay money for something you built. That's a good feeling. Go with it and enjoy the ride.

Tags: kynetx startup sales

Image via Wikipedia

This Friday at 11:30, Kynetx will host an open house and demo lunch. If you'd like to see what we're up to, stop by, eat some pizza, see some demos, and chat. We're especially interested in getting more people developing on our platform, so if you'd like to try it out, we'll have sign up cards for anyone who wants them.

We'll be at Kynetx World Headquarters in Thanksgiving Point. This map will give you directions; we're in Suite 275 (metal door). Please RSVP so we know how much pizza to buy.

Tags: kynetx events utah

Image via Wikipedia

This Friday at 11:30, Kynetx will host an open house and demo lunch. If you'd like to see what we're up to, stop by, eat some pizza, see some demos, and chat. We're especially interested in getting more people developing on our platform, so if you'd like to try it out, we'll have sign up cards for anyone who wants them.

We'll be at Kynetx World Headquarters in Thanksgiving Point. This map will give you directions; we're in Suite 275 (metal door). Please RSVP so we know how much pizza to buy.

Tags: kynetx events utah

Image by MrsWoman via Flickr

This week I've been working on creating a virtual appliance (VA) version of the Kynetx engine. This is a necessary step for customers who need a version of the Kynetx Rules Engine (KRE) running behind their firewall (think SAS70 compliance, for example).

This post documents some of what I discovered. (N.B. Since I'm working with Xen, much of what appears below could be Xen specific; I haven't taken the time to generalize it.)

First, and most important, there's no standard definition of what a virtual appliance is or how they are built. There are companies like rPath and others who have developed processes, but there are literally dozens of ways you might do it. They can be categorized into three camps:

• You can, in the spirit of an appliance, hand over a disk image (or images) and an XML machine descriptor which can be used, over and over again if properly set up, to reconstitute a machine (using virt-image for example).
• You can create a VA factory that when fired off creates a new machine from scratch, including the loading of the OS, configuration, and so on.
• You can do a hybrid of these two that includes a mostly configured image that then bootstraps itself to a final configuration.

Let's make a few assumptions:

1. You want to install multiple copies of the VA at the customer site.
2. The installation needs to be "appliance-like" implying that this isn't just a custom software installtion but is relatively independent of customer environment and is configurable.

Given these requirements and some choices we're already made--such as using a rather large puppet recipe collection to configure machines--I opted for for the VA factory method of deploying our appliances.

To make that all work, there are a number of pieces you need: DHCP, Kickstart files (if you are using RedHat, Centos, or Fedora), repositories, puppet recipes, and bootstrap scripts. Setting all these up individually can be done, but there's a better way: Cobbler.

Cobbler is a Linux installation server that takes care of putting all these together for you into a nice package that you configure and then it takes care of the details of making things work. It took me most of a day to play with it, get it set up how I wanted, and then see success in creating a usable image.

You could put Cobbler on Dom0 (the host machine in Xen), but I chose to put Cobbler and it's associated repositories on a virtual machine, which functions like a VA factory. That primordial system can be installed on a Xen host machine and then used to create as many copies and configurations of the VA as we need. The factory contains the Linux distro, the repositories we need, the kickstart file, and other configuration data. I haven't decided whether to put the puppet recipes there or not, but I'm leaning that way.

For the record, here's what I did in Cobbler. After installing and making sure the installation was good (cobbler check), I imported a distro, created a repository, created a profile for the particular kind of system I wanted to build, and added system definitions for the specific VA's I was going to create. (N.B. the double \\ indicates that the command ought to be on one line, but I've warpped it for presentation here.

cobbler import --mirror ~/fc8 --name fc8
cobbler repo add --mirror=http://archive.fedoraproject.org/...\\ 
  --name=fc8-newkey-repo --keep-updated=Y \\
  --rpm-list="puppet facter ruby-libs..."
cobbler profile add --name=base-fc8 --distro=fc8-xen-i386 \\
  --kickstart=/root/base-fc8.ks  --repos=fc8-newkey-repo
cobbler system add --name=kib0 --mac=00:16:3E:4B:4B:40 \\
  --ip=192.168.122.160 --profile=base-fc8 --hostname=kib0
cobbler system add --name=kib1 --mac=00:16:3E:4B:4B:41 \\
  --ip=192.168.122.161 --profile=base-fc8 --hostname=kib1
cobbler system add --name=kib2 --mac=00:16:3E:4B:4B:42 \\
  --ip=192.168.122.162 --profile=base-fc8 --hostname=kib2
cobbler system add --name=kib3 --mac=00:16:3E:4B:4B:43 \\
 --ip=192.168.122.163 --profile=base-fc8 --hostname=kib3

I customized things the most by editing the default Kickstart file that comes with Cobbler to customize the build to what I needed. After that was all set up, installing a virtual machine was as simple as typing the following command on Dom0:

koan --server=cobbler.kobj.net --virt --nogfx \\
   --system=kib0 --virt-bridge=virbr0

Koan stands for "kickstart over a network" and is the command that uses cobbler to install systems from a Cobbler server. I've still got some small kinks to work out, but I'm well on my way to having this all completely automated.

All this automation may seem like overkill, but it's essential in a small shop. Without taking the time to create these systems, a few customers would quickly eat all the time we have and then we've got to hire more people. By creating automated system, we're largely free to move onto creating new features and servicing new customers because existing customer maintenance has been automated to the greatest degree we can. Some might call that survivial--I just call it getting to sleep through the night.

Tags: virtualization kynetx virtual+appliance

Yesterday I accidentally overwrote the disk image of a running Xen server that represented a machine in production. I didn't notice for two hours because the services on the machine kept going since they didn't need the disk. In fact, the only reason I noticed was because I happened to need to do the same thing again and got the command from the shell command history and noticed the mistake--a one character typo.

Recovery from this event required rebuilding the machine. Fortunately, we've been working hard on infrastructure automation and have a set of Puppet recipes for completely building out a server and an automated deploy process that loads the required application code and tests it. In addition, since it's all on Xen and the old server was still running, we just build up another machine while the old, corrupted server trundled along.

In about an hour a new machine was up and running. We have been meaning to put up redundent copies of this particualr service. In fact, that's what I was working on when I caused the problem in the first place. We took this opportunity to put the new machine behind the load balancer and moved the DNS entry. Overnight the traffic has tailed off considerably to the damaged machine and the new machine is now handling most of the load. I'll build up another copy on another host machine this morning so that we have the redundency we need.

Lessons learned:

• Automate everything. More automation would have prevented the mistake I made. The automation we had saved us. Puppet and other tools are the only way to manage infrastructure.
• Virtualization gives your incredible flexibility. I love the ability to put up new machines as needed wthout having to manage the hardware tasks at the same time.
• Put things behind load balancers and have multiple copes of services running, even if you don't need the capacity. If we'd done this earlier, my mistake would have been a non-event. We'd have just removed the damaged server from rotation and went on with life.

The end result is that Kynetx suffered no downtime, but we did have some tense moments. Changes we're making to the infrastructure will improve our chances of achieving the same results with less stress and sweat.

Tags: puppet kynetx system+administration

Normally, I subscribe to the "YNGNI" philosophy: You're Not Gonna Need It. The idea is don't over engineer things because you'll likely spend a lot of time on things that never get used. This post is about an exception to that rule that I wish I'd followed.

When I started designing KRL, there wasn't much in the way of variable scoping and it didn't really need it. Each rule was a local scope and there were no global variables. Only one ruleset ever operated at a time. Consequently, I took the easy route and instead of designing a proper environments module that managed symbol tables through extension and recursive lookup, I just used a hash.

Time passses...

I find myself with a language whose requirements have grown to include global declarations and multiple rulesets firing at the same time. Ad hoc namespacing in the hash that manages the rule environment was getting out of control. The final straw was the need to add global variable bindings to the language so that, for example, you could make a single query against a data source and then share the results in any of a number of rules that might fire. I wasn't about to pile that on top and hope it didn't fall down.

Consequently, I've spent the last two days refactoring the language to use a proper variable environment. A few observations:

• The code is much cleaner now. Not only do I feel better about that, but I'm more likely to see how things work and not make mistakes in future revs.
• I'm extremely glad to have a test suite that I'd already written that helps me feel more confident that I haven't changed the semantics of the language with this change.
• Any change this big not only changes the code, but also the test suite (mostly calls that change and return results that change). So, that reduces my confidence in consistent semantics somewhat.

The main point of this post is simple: when it comes to building intepreters for a language, no matter how simple it looks when you start out, take the time to build and use a hierarchical environment system for managing variable scoping. I wish I had. I'd have saved myself time along the way and not needed to spend two days refactoring the code now. You're gonna need it...

Tags: krl programming+languages kynetx

Image by Adriana Lukas via Flickr

I can't believe that a week has passed since IIW8 ended. I was planning on writing a wrap-up blog post then and time has just slipped away.

When we were planning for this IIW, we were pretty worried about whether people would come and whether we'd get sufficient sponsor support. As it happened, things turned out fine. We had more people than ever--about 180. And while some sponsorships went unsold, overall we had great support for which we're grateful.

The workshop itself was one of the best ever. There was a lot of energy in the various sessions and people came prepared to solve problems. For example, I remember one session that Joseph Smarr of Plaxo ran on API keys. People from Plaxo, Google, Facebook, Microsoft, Myspace, and others were in the room throwing out ideas and working together to come up with a generalized method.

We started earlier this time (Monday morning instead of the afternoon) and the extra time seemed to be all used up. I expect we'll do that again. We didn't give away Android phones, but there was a lot of good food, good conversation, and the "thank-you" circle at the end is not to be missed. Kaliya did a great job with the program, as usual. I can't say enough good about how she creates workshops that generate community.

Drummond and I ran a session on action cards--information cards that create action and I followed that up with a demoinstration of using Kynetx to create action cards. We have a private beta of our developer tools going on. If you're interested in participating, let me know and I'll send you the signup URL.

The next (ninth) IIW will be November 3-5. Get it on your calendar now. If you're interested in identity, IIW is not to be missed.

Tags: identity iiw kynetx krl

Kynetx Network Service, or KNS, modifies a user's Web page using Javascript. The ability to customize pages in the browser is a powerful capability, but it goes well beyond that by allowing data from multiple sources, even other Web pages, to be used as part of that customization. Sure we can change change colors, fonts, and layout, but we can also mashup Web sites to produce completely new experiences.

Described as I have, KNS is not unlike Greasemonkey, a popular plug-in for Firefox that allows user scripts to modify Web pages. In fact, in a recent post Paul Madsen discussed the KNS/Greasemonkey connection. Consequently, I thought a comparison of KNS and Greasemonkey would be in order. Note that I'm not a Greasemonkey expert, but I happen to be the world's foremost authority on KNS. :-)

Like the Web, KNS is a hybrid cloud and client solution. Greasemonkey is a client only solution, although it can, of course, reference data in the cloud. This has a number of positive consequences:

• Like any good cloud-based application, Kynetx apps are available and work consistently on any machine you use.
• Referenced data sources can be easily cached and proxied. This offers opportunities to increase performance and ease of use.
• Updates happen automatically as the servers are updated, like any cloud service. For example, recently Google started experimenting with AJAX search result pages which required updating scripts that augment Google. We changed some things on the server and users saw the updates immediately.
• Increased security. As an example, a malicious KNS app can be disabled on the server saving all users from the effects.
• Auditing and analytics are possible. Kynetx can audit apps for suspicious patterns and provide easy-to-understand reports about app privacy and security policies and activities. Server-side analytics give developers usage data on their cards without compromising individual privacy.
• We can track when the actions (ruleset) associated with a particular card change and warn the user.

KNS is controlled by a domain-specific language: KRL (Kynetx Rule Language). Greasemonkey uses raw Javascript.

• KRL provides developers with an easy way to create apps that work across browsers and operating systems. All of the details about working in Firefox or IE are abstracted away.
• KRL provides developers with a powerful lever for quickly developing apps. Augmenting search result pages, for example, is a simple action that replaces dozens of lines of Javascript.
• The abstractions of KRL allow Kynetx to respond to changes in browsers, Javascript, and Web sites with updated interpreters or runtime libraries to address the changes without developers having to change their apps.
• People who would never develop Javascript programs have successfully developed Kynetx apps.
• When the going gets tough, like any good domain specific language, KRL let's you jump out into raw Javascript to make up for holes in the language. As this happens, we'll incorporate commonly used patterns into KRL, making it more powerful.

KNS is based on Information Cards. Greasemonkey scripts are not associated with any particular identity system. We call Kynetx action cards "KIX".

• Information cards are tied to a specific action. Because of the security model of information cards, the KRL ruleset associated with a particular card cannot be changed. Thus, when a user gets a KIX, they can be assured that the actions associated with it are the ones the developer intended and haven't been replaced by a malicious program.
• The converse of the last point is that if someone creates a malicious app, it has a specific identity that cannot be hidden or changed. KIX can be rated, reviewed, and analyzed based on a non-mutable identifier.
• The information card selector provide a nice user interface for installing, controlling, managing, and deleting KIX and their associated actions.
• While KIX do not yet support in-card claims (personally identifying data), when they do, access to that data will be via the well-thought out and "socially tested" information card ceremony putting users squarely in control and mindful of what personal data they're releasing to make a Kynetx app work.

For the moment, KNS is simply placing tags on pages ('ala Greasemonkey 0.3) rather than executing scripts in a sandbox ('ala Greasemonkey 0.5). The later model is clearly superior from a security standpoint and would also give KNS performance advantages over it's current execution model. That's where we're headed. Still, because of mitigating design in our architecture, the current risk is small: we do not allow tags to be inserted on wildcard domains. This allows developers control on which domains tags are placed and thus protect user privacy and security.

Web augmentation is a fascinating place to be building technology because there are so many interesting problems to solve. Our goal for the KNS model for augmenting Web sites is to create a system that is general, safe, performant, easy to use, and easy to develop for. We're very close to opening up the doors for developers to start using KNS to create augmented Web experiences of their own. Stand by!

Tags: kynetx code krl kns security privacy

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the eightteenth installment. You may find my efforts instructive. Or you may know a better way---if so, please let me know!

Yesterday, Steve had an interesting experience which proved one of our claims. We've always said that we were trying to build a Web augmentation service that even our Mom's would use. This is in contrast to, say, Greasemonkey, which is a great tool, but it's decidely aimed at hackers for their personal use. We're trying to do something similar, but allow developers to build augmentated Web experiences that they could share with anyone--technical or not.

Azigo announced their deal to provide their RemindMe service to AAA of Washington members. Kynetx powers the RemindMe service. Steve's brother Bob happens to live in Seattle and is a AAA member so Steve forwarded the press release to him. Without any help or prodding Bob downloaded the reminder and started using it. He was very excited and sent Steve a note thinking him for helping him remember about his AAA discounts.

What happened next is classic. Bob sent a note talking about RemindMe to the family mailing list. Not only did Steve's mom get it, but she also asked the key question: what else can this do for me? She immediately recognized that there was more to this than AAA reminders--as important as those might be.

We can now answer the key question, "but will your Mom use it?" with a solid "yes" because she does! That and the viral nature of the interaction are major milestones.

Tags: startup kynetx usability

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the seventeenth installment. You may find my efforts instructive. Or you may know a better way--if so, please let me know!

When you start a business you put a lot of yourself into it: time, ideas, and, of course, money for things like servers, supplies, and people.

In addition to whatever money you put into the business, you also are spending money just living. Starting a business is not for the fearful. Quitting a job and convincing your family to sacrifice for months or even years while you pursue a venture takes real faith. You might think you can get by without much personal sacrifice, but you can't. That's just not realistic.

When Steve and I started Kynetx, we sold our plane to fund the business. We also realized that we'd be living off our second mortgages for a while. Frankly I've been surprised at how low I have been able to get our burn rate and how long the money has stretched. Good thing too since I've also been surprised how long it's taken to get funding!

Sooner or later, however, most founders will need to start getting some money out of the business in the form of a salary. That is a magical moment. For me at Kynetx it happened yesterday. I can't properly describe the emotions that ran through me as I held a small check representing the first bit of money that I hope is a permanent reversal of my personal cash flow as it relates to Kynetx.

Paying yourself is tricky business in a startup. Investors don't really want to put money into a company only to have the founders transfer it--even as a small salary--to their personal checking account. The best thing to do is to pay yourself out of sales and use investment dollars to grow the business. Besides, a "no sales, no pay for the founders" policy incents everyone toward the thing that has to happen to make the business work: make sales.

Tags: startup kynetx funding

As we get ready to launch a limited beta program for developers to use the Kynetx platform, one of the things I wanted to get going was a place to talk about Kynetx code. So we stood up code.kynetx.com.

When it's all done, there will be documentation, forums, and examples. I've been making some posts on it even though the template isn't done. So far I've posted:

• New feature: CSS in Globals
• Updating the Runtime: Annotated Search Results
• KRL Templates

I've been enjoying writing these because it gives me a chance to write about changes and features in one place.

Tags: kynetx code programming krl

Image via Wikipedia

The short term lot at Salt Lake City Airport has electronic display boards that tell you how many parking spots are available on each deck. That wasn't always the case. Monday when I was parking to go to RSA, I realized that in the old days my default behavior was to head straight to the top deck because there were always more spots there. The small overhead of driving up to the third deck was far better than driving around the aisle looking for a place to park.

Now, with more info, my alrgorithm has changed. If there's a small number of spots on one of the decks, I skip it, but if the number is large (>100), then I stop at that deck and park.

Of course, the idea that data leads to better decisions isn't a great revealation. People have known that for centuries. But its interesting to see it affect simple activities in my own life.

Our goal with Kynetx is this exact kind of information empowerment. We call the process "context automation." You can get some ideas about how this works by reading our latest white paper: Creating Contextualized Web Experiences (PDF).

Tags: data information kynetx

Image via Wikipedia

KRL, the Kynetx Rule Language, is a domain specific language that operates on the Kynetx Network Service (KNS). When I first started building KNS the first priority was getting a rude dog demo working that we could use to start having customer and investor conversations. Consequently, I was less than pure in some of the implementation details.

For example, the conditionals in if statements were just single predicates and the declarations in the rule prolog were all one-offs. Later I went back and wrote a complete, generalized expression language that could be used in the conditionals and in the actions, but I'd never gone back to redo how declarations were done in the prolog.

A need to query user-defined data sources and then do something with the results, led me to determine that using generalized pattern-based selectors were the most general and powerful means of interacting with the data (think XPath, CSS selectors in jQuery, and JSONPath).

Unfortunately, when I dug into the code, I remembered I'd incurred a little programming debt and needed to clean up the declarations in the prolog rather than add yet another piece of one-off code to that part of the interpreter. Not only did this mean I only needed to deal with the selector expressions once, but it also greatly simplified the declaration code while making declarations in the language much more powerful.

To see this in action, imagine the following scenario. You've got a queriable data source at a URL:

http://www.example.com/bookdata?isbn=xxx

that returns JSON data telling you if a book is available at your local library. The JSON might look like this:

{"responseHeader":{"status":0,
                   "QTime":1,
                   "params":{"q":"0316160202","wt":"json"}},
 "response":{
     "numFound":1,
     "start":0,
     "docs":[
          {"isbn":"0316160202",
           "title":"Eclipse",
           "url":"http://library.com/search/?isbn=0316160202"
           } ] } }

You could create a KRL ruleset that queries that data set and puts a Growl-like notification at Amazon informing you of that fact for any book page you visit like so:


ruleset library {

    global {
	datasource library_search 
                  <- "http://www.example.com/bookdata/?wt=json";
    }

    rule book_notfication is active {
        select using "www.amazon.com/gp/product/(\d+)/" 
                 setting(isbn)
        pre {
      	  book_data = datasource:library_search("q="+isbn);
  	  url = book_data.pick("$..docs[0].url");
 	  title = book_data.pick("$..docs[0].title");

	  msg = <<
This book's available at your local library. 
Click here to see: <a href="#{url}">#{title}</a>
>>;
	}
	if(book_data.pick("$..numFound") > 0) then {
		notify("top-right", "#222", "#FFF", 
		       "Local Library", true, msg);
	}
    }
}

If you're interested in the details of how this works, there's some KRl documentation available online and more to come soon. The pick operation is applying a JSONPath pattern to select specific parts of the JSON data. I'm happy to explain any of it in more detail, so just ask.

As I describe here, you'd enable this functionality by installing a small Information Card (we call them KIX) in your card selector. The rule lives and executes in the cloud.

I love when there's something new someone wants to do that we can't quite do yet. I get a chance to build more on the language. There is a grand design, but I can only afford to build out the pieces we need right now and even at that, I can't really keep up. The key, I've found, is making sure I take time to refactor as I go. Otherwise, you get to the point where every task is too hard to do.

Tags: kynetx language+design krl json

I've been hearing about information cards for the last four years or so. For that entire time, they have been largely theoretical or, at best, used for logging into a blog or something else. I'm out to change that.

A few days ago Steve Fulling was on the phone with Paul Trevethick of Parity describing a pilot project we're doing for the Locals Care organization of New Mexico. Steve said "I'll just send you the card; put it in your selector and you'll see the results in Google."

That's a pretty interesting statement: "I'll just send you a card." The reason I love it is because that's a key part of the Kynetx vision: viral web experiences. Of course, if this were a managed card, with claims, you wouldn't just send them around, but when the card has no claims, this works.

If you'd like to try it yourself, you need two things: first you need the Azigo selector. This works on Mac or Windows with IE or Firefox. One warning: it will mask your CardSpace selector but it's still there--just go to the Azigo selector switch in the Control Panel to get it back. Of course, if you haven't done anything with your CardSpace selector yet, you won't care.

Second, just import this card into the Azigo selector. Now, restart your browser and search for something local to Santa Fe, like santa fe coffee or santa fe pizza.

If everything worked, you should see a box with a yellow border and the Locals Care logo at the top of the Google search results with local results. These have been percolated up through the first ten pages of results to the top. Note that this only happens for people who have opted-in to the experience by taking the Locals Care card.

We view this result as being the triangulation of three different things:

1. The page the user's looking at (e.g. Google search results)
2. The dataset delineating relationships (e.g. the Locals Care database)
3. The information card itself (giving the ruleset ID and, possibly, claims)

The triangulation of these three dimensions is performed by the ruleset and the Kynetx Network Service (KNS). Change the page, the dataset, or the card (and it's associated ruleset) and the results will be different. This makes it vastly more powerful and flexible than a special purpose add-on like SurfCanyon or even a Greasemonkey script. Just try getting your mom to install a Greasemonkey script. She'll have an easier time with the card, trust me.

If you're interested in the details of how all this works, I invite you to read our white paper (PDF), leave a comment, or contact me.

If you're interested in playing around a bit more, here are two more cards you can install.

• Better Craig's List - install this card and when you search on the Provo Craig's List, you'll get pictures on the results page so you don't have to click into every link.
• TwitterSpy - this card augments the Twitter search page to actually show you new results rather than just telling you when there are new ones. To see it in action, search on something that's trending up.

These two cards are missing the dataset dimension and thus "two dimensional," but it's still fun to just install a small card and get better functionality out of your browser.

Note that this process is particularly easy when you already have the selector. The selector and associate browser extension become a "universal" add-on that does it's work through the cloud. Just add a card and you've got a new, better experience. I'm biased, but I think that's pretty darn cool.

Tags: kynetx information+cards web+augmentation

Last week Jon Udell interviewed me on Interviews with Innovators about Kynetx and Contextual Browsing. It was fun to be the one answering questions for a change and Jon asked some good ones. If you've been wondering what Kynetx does, this podcast is a pretty good intro.

Tags: kynetx itconversations

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the sixteenth installment. You may find my efforts instructive. Or you may know a better way----if so, please let me know!

Image via Wikipedia

In fusion energy research Q is the ratio of fusion power produced in a nuclear fusion reactor to the power required to maintain the plasma in steady state and is called the fusion energy gain factor. When Q=1, the amount of power that the reaction is producing is equal to the amount of power you're putting into it to establish the fusion in the first place. Alas, fusion researchers have yet to get to this point.

You can also think of your business having a Q factor that is the ratio between the energy and activity your business is generating and the amount of energy you're having to put into it.

In the early days, your Q factor will be very low. You're putting in enormous amounts of energy (not to mention money) and seeing very little excitement or movement. Every conversation is hard because you're not sure how to explain the idea. You won't have a real product yet, so customer conversations are as difficult as they are necessary. Convincing investors, volunteers, and co-founders to jump in or keep going is hard.

Then one day something will seem different. Instead of you telling others why what you're trying to do is cool, they'll start telling you. Investors will be excited when you talk them instead of tolerant. And people will sense the energy and want to jump on board. Your Q-factor just crossed 1.

One of the hardest parts of reaching Q=1 is that you often don't realize when you're getting close. The energy requirements aren't linear. You often have to work harder than ever just when it's the easiest to give up and believe you're wasting your time.

In the same manner, getting past Q=1 doesn't mean it's easy afterwards. You still have to put in a lot of work. Probably even more than before. But psychologically, you've reached an important milestone that makes it easier to do because you're getting positive feedback that reinforces your beliefs and efforts.

Tags: kynetx startup

This morning Steve gave a presentation on context automation and Kynetx at the Utah Technology Council's CTO P2P forum. The presentation was great and the audience asked a lot of good questions. One thing that came up (I don't even remember why) was the subject of continuous deployment. I decided I'd pull a few URLs out of my head and put them in a blog post for people to mull over.

The first URL I think of when I consider continuous deployment is code.flickr.com. If you've never been there, the bottom of the page lists when the last deployment of Flickr was, how many deployments have happened in the last week, and who was involved (pictures). Here's a screenshot:

When I first saw that I was astounded. Sometimes there are a dozen or more deployments in a single day. The questions that spring to mind: why? and how? Both are answered in a few posts by Timothy Fitz.

In Continuous Deployment, Timothy discusses the concept. Basically, it comes down to "fail fast." Deploying a few small changes is less likely to break something and when it does, you'll know more quickly what caused the problem and be able to correct it.

In Continuous Deployment at IMVU: Doing the impossible fifty times a day Timothy goes into more detail about how they do this. The idea comes down to

• Commit early and often
• Automatically test on commit
• Automatically roll the code out if the tests pass

There are, of course, some problems to solve to get that done. First, you need a good, thorough test suite. Timothy points out that you also need tests that

• run fast and
• execute reliably.

The test suite Timothy is describing takes 4.4 machine hours to execute. That's a lot of testing. To make it run fast enough to deploy continuously, they have a buildbot that runs tests across 36 machines in parallel.

The point about test reliability is important too. Intermitently failing tests will ruin this process. Timothy says:

When I say reliable, I don't mean "they can fail once in a thousand test runs." I mean "they must not fail more often than once in a million test runs." We have around 15k test cases, and they're run around 70 times a day. That's a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day. It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it's just one part of my day job.

From Continuous Deployment at IMVU: Doing the impossible fifty times a day. « Timothy Fitz
Referenced Fri Mar 13 2009 10:54:22 GMT-0600 (MDT)

I love this whole idea. I've lived the life of infrequent deployments and it will suck the soul right out of your engineering and ops teams. That's why when we started up Kynetx, I was determined to not repeat those mistakes. Our system is not as sophisticated as the one Timothy describes, but my goals is to get there and we make specific goals about things that need to happen to get there.

You may not be able to get to 50 deployments a day overnight, but you can increase the frequency of deployment and prioritize the development efforts necessary to increase that frequency. Set some goals and take your life back.

Tags: operations deployment kynetx

Image by Thomas Hawk via Flickr

Guy Kawasaki is speaking at the Infopia ecommerce conference in Salt Lake City today. (You might also enjoy reading my notes from the last time Guy was in Utah.) His topic is how to drive the competition crazy. In standard Guy style, he gives his talk as a list of ten things:

1. Find a mighty opposite - find a great enemy who is trying to do something in direct opposition to what you're doing. Portray them in ways that emphasize how you're different. It doesn't have to be a company. It could be something like "ignorance." Finding an opposite allows you to make meaning instead of just making money.
2. Know thyself - you can't fight the competition unless you know what you're about and where you stand. Guy recommends creating a mantra (not a mission statement) as the first step. Here's Kynetx' mantra: automate context.
3. Know thy customer - everyone says this, but the key is the degree to which you do it. Don't farm it out; do it yourself. Making sense of who your customers are and what they want will drive you crazy--not just your competition.
4. Know thy enemy - the best way is to become your competition's customer. What's the shipping policy, their credit policy, etc.? How do they do it? Get first hand knowledge. Go to their conferences.
5. Focus on the customer/create good shitake - too many companies focus on their competition. People don't care about your battle with the competition; they only care about whether or not you serve their interests. The most important aspect of focusing on your customer, you have to build a good product that your customers want to buy. He takes a swipe at SEO: Google is in the business of finding good stuff. If you build good stuff, Google will find it because their interests are aligned with yours. Great products aren't a little better; they're 10 times better. Your product ought to be emotive and be polarizing. He references his DICEE advice.
6. Turn customers into evangelists - if customers are emotional about your product they will talk about it--for free. Guy uses Nike as an example: they're not selling cotton and leather stiched together as shoes. They're selling ideas and emotion.
7. Create your own day - Levi Strauss commissioned a study on the effect of casual dress on workplace morale and productivity. They made this available to the press and created a "casual dress implementation kit." Lots of papers picked it up. They created their own opportunity.
8. Make good by doing good - align yourself with good causes.
9. Turn your competition into allies - find ways to work with, rather than against your competition. Where can you be collborative to engage new markets? He gives the example of J.B. Hunt trucking teaming with railroads to put semi trailers on flatbeds.
10. Play with the minds - Once you've done everything else, play games with the competition. He tells of a pizza company that offers 2 for 1 pizzas if you tear out and bring in the yellow pages ad of their competition. He talks of a small business person sitting next to a Home Depot who renamed his business "Main Entrance" to get people coming to Home Depot to come into his store.
11. Use Twitter - A bonus 11th tip. What good is a cell phone that allows you to listen to random snippets of conversation? Use search to find out what people are saying anything about you or your company? When you see people talking about things your sell they are potential customers. Guy gives a demo of Twitter to a room of people who aren't Tweeple (for the most part). He demos Tweetdeck (FTW). He recommends using TwitterHawk but staying away from the autopost features.

Good marketing pisses people off. If people aren't getting mad at you, then you're boring and bland. You can't let a vocal minority tell you what to do.

I enjoyed this talk a lot. First of all Guy is an entertaining speaker. Second, and more important, even though he's using some material from older talks I've heard, he's constantly refreshing and rearranging his content. There's new stuff here and it adds new interest.

Tags: utah events business kynetx

Joe Andrieu has written a blog post on Netizen Developers wherein he makes this assertion:

As netizen developers, we have an obligation not just to do what makes us money, or even what makes users happy, but to build systems that work at Internet scale, when everyone does it. If the systems we build don't work when everyone tries to get into the game, then we are just being selfish, hording value just because we are first-to-market.

From joeandrieu.com » Blog Archive » Netizen Developer
Referenced Tue Mar 03 2009 09:07:56 GMT-0700 (MST)

He mentions systems like the one we're building at Kynetx, others like ad blockers, as well as his own toolbar-based SwitchBook. You might view his point as altruistic given the quote I use above, but his real point is "how to we make this all work?"

It comes down to a question of open systems. Open systems that work, work when everyone does it, because that's where you get game-changing economies of scale. The network effect only happens if the value of the system increases when more and more people use it and open systems are all about the network effect.

• What happens if everyone uses TCP/IP? WhoohoO! Seamless interconnected networks.
• What if everyone uses SMTP, POP, and IMAP? Yes! You can email anyone, anywhere, anytime!
• What if every company, government agency, and organization uses HTML and http to build online services for their users? Mega efficiency. 24 hour engagement. Low-cost quick answers. Happier people and happier organizations.

Those are good open systems.

From joeandrieu.com » Blog Archive » Netizen Developer
Referenced Tue Mar 03 2009 09:10:59 GMT-0700 (MST)

I don't know that I or anyone else can answer Joe's questions completely at this point. The ultimate answer will include standards involving the execution environments that augmentation services operate in and how they interact with each other. This is a tough problem. There's not a single, elegant answer. To this point standards that have been written have mostly been security related and been something that makes mashup writing more difficult, not more reliable.

I don't see this as an "ethical" issue the way that Joe seems to--in terms of being a good "netizen." I see it through a practical lens. If we don't solve this, then users will solve it for us by just not using our stuff. That said, the problem with simply looking at this as an issue that vendors must solve is that it means that no one may build a true "system" that works in the way email, the Web, and the 'Net itself do.

Such a system is the solution that Joe proposes:

Ultimately, what we need is an open system that allows all of these types of augmentations from Adaptive Blue, SwitchBook, Kynetx, Azigo, Google, Skype, and others, to mingle smoothly in the same interface.

From joeandrieu.com » Blog Archive » Netizen Developer
Referenced Tue Mar 03 2009 09:44:04 GMT-0700 (MST)

Kynetx sees itself as that system. Azigo is built on top of Kynetx and ultimately so could most of the others.

Joe says we need an "open system" and I agree. The word "open," however, comes with a lot of baggage that may not be helpful. Kynetx is not open in the sense that it's not open source. What it does provide, however, is an open API and a language specification that anyone could implement (note that this is still in development). We may open more in the future as we determine how this works and discover business models.

I look forward to discussions about the model and requirements for adoption. I'm especially interested in figuring our how what Kynetx is building can better serve as a platform for creating web augmentation services that work well for users and don't conflict.

Tags: kynetx web+augmentation

Judith Burton has written a good piece on context automation and what we're doing at Kynetx. She says:

When browsing the Internet, each web site encounter throughout the browsing episode is a separate or ad hoc experience. There is no thread or connection from one site to another except in the user's mind. The user must hold the context of the browsing episode outside of each web site encounter.

From Context Automation
Referenced Tue Feb 17 2009 09:30:53 GMT-0700 (MST)

And then later:

To create a context across multiple sites, an abstraction layer may be employed that is capable of gathering information from the user and from each site throughout the entire browsing experience. This interactive cross-site abstraction layer is called Structured Browsing.

From Context Automation
Referenced Tue Feb 17 2009 09:31:45 GMT-0700 (MST)

Having an abstraction layer that is easily programmed on one hand and easy to interact with on the other is the key. Such an abstraction layer can be programmed to manage different kinds of context while giving the user a consistent experience.

We're working hard to get the Kynetx abstraction layer, which we call KNS, ready for a more general deployment. We hope to offer limited access to developers interested in creating contexual Web experiences in the next several months. In the mean time, I'll be blogging thoughts here. Also, if you haven't read the white paper (PDF) it will explain much of this in more detail.

Tags: context+automation kynetx structured+browsing

Joe Andrieu posted a response to the white paper I released last week. I'm grateful that Joe would take the time necessary to read the paper in depth and offer a long, well-thought out, and helpful set of questions and critique. From his article it's clear to me that Joe understands the problem space well and has a firm grasp on what Kynetx is doing there.

Joe raises a number of questions and points that I'd like to respond to.

First, Joe asks who the target developers are: Web sites or third party services. Or both? Our primary offering is aimed at third party services, but we also recognize the value that Web sites can add by responding to people more appropriately. For example, we have an OEM arrangement with Parity for their RemindMe service. Their customers (i.e. people with RemindMe cards) benefit, however, if Web sites work well within this larger context. Incidentally, those Web sites benefit as well. We believe this dual-use strategy is the right way to go since we want to see the kind of silo-spanning context-aware services that third parties are likely to build and yet know that there has to be a way for the silos to play the game too.

Joe's second question foresees a user experience nightmare with users managing A cards on B sites and getting caught in a crush of A x B "identity ceremonies." Something that isn't clear from the white paper is how this user experience is managed. In most cases (certainly the kind of casual-browsing, context-management activities mentioned by Joe), Kynetx is the relying party--not the individual Web sites. Before data from cards is sent to the site, of course, a separate ceremony establishing a identity session between the site and the person would have to be performed. But this happens only when the user is intent on dealing with that site, not when just cruising around.

This might raise some questions about privacy and security, so it's important to understand in these scenarios that the user data isn't ever leaving the browser. Some session data is sent back to Kynetx servers, but not the data out of the cards themselves.

Joe also mentions in his discussion the idea of user data stores. Yeah. That's one reason I'm interested in first class namespaces in programming languages. We'd love to collaborate with Joe on the idea's behind Search Map and user-driven search since user controlled data stores are important to us as well. I foresee the day when a KRL rule can use and respond to the data in a person's SearchMap. KNS has the ability to link to data stores on the Web and on the user's machine (permissioned, of course) but accessing those in a coherent way within KRL requires more advanced linguistic leverage than we now have. It's on my list...

Joe says that user's do want to manage their context, but they haven't been given the right tools. Fair enough--in fact, I don't think we're actually saying different things. We don't anticipate that people would have no part to play in managing context. We see KNS as a tool for managing context and using it effectively. Right now, Web users mostly manage context in our heads--there is a dearth of tools for helping with that task.

Joe also raises questions about privacy and data rights management of data inside information cards (or OpenID attributes, for that matter). That's a bigger issue that Kynetx alone can solve, but I do think we're in good shape in that regard. As I mentioned above, most of the data stays in the browser and Web sites never see it until the point that the person is ready to take action. Joe uses the AAA and Hertz example. When you use your AAA card at Hertz, Hertz knows you're a member and can tells others. We don't solve that problem, but we don't make it any worse. In fact Kynetx decreases the frequency that you'd have to reveal you're a AAA member while still letting you receive the benefit of knowing what it will get you as you cruise the Web. We allow merchants to respond to you without you having to reveal data to those merchants.

Perhaps's Joe's most important discussion from my perspective is on business model. He's right: CPM charges for ruleset evaluations increases friction at the adoption point and for smaller players. That's a problem and we're open to fixing it. I'm not opposed to more open models--in fact I see great value there.

That said, Kynetx also has to survive and right now that means getting funding. The because of model is great as far as it goes, but I don't find the idea of selling consulting and IDEs to be very compelling. I frankly can't imagine myself sitting with a VC and pitching it. Maybe I'm gun shy or lack vision, but I'm unsure how it would play.

At any rate, I'm anxious to collaborate with Joe and others on this. Our vision is similar and our methods aren't that far apart either. This is a fun time to be working in the Web.

Update: I forgot to comment on Joe's point about centralization. His idea of using a reputation network with strong identity in place of centralized certification is brilliant. We're definitely not looking for ways to make this more centralized. Quite the opposite.

Tags: kynetx vrm structured+browsing krl white+papers

Today I'm releasing the first of a series of white papers on Kynetx and what it's trying to accomplish: The Advent of Next Generation Browsing (PDF). I introduce the problem in the first few paragraphs like so:

We are mired in a tangle of architectural legacies that make today's Web browsing experience uncomfortable, confusing, and tiresome for many users. In particular, the lack of Web site independent identity has hampered the ability of the browser to effectively intermediate the Web on the user's behalf. But change is coming and we are about to witness a significant improvement in the nature of Web browsing---indeed, the nature of the browsing experience is about to change forever.

In the early days of the Internet, companies sought to give users the benefit of a consistent experience by building portals that integrated multiple activities. Portals are now mostly a thing of the past; a few large examples such as Yahoo! and MSN still exist, but by and large they have fallen victim to what must be the most important law of the Web: a different site is only a click away. As users sought out the best sites for any given purpose, the browsing experience fractured and became ad hoc. As a result, disaggregation of Web sites and services is now the norm.

An example from the world of ecommerce illustrates this. Shoppers use search engines like Google or Yahoo! to find a product and choose an online retailer from the search results. Before buying they might research products at independent review sites like Epinions.com and Viewpoint.com. They might discuss those same products on myriad blogs, Twitter, and social networking sites like Facebook.

As we'll see, disaggregation causes users to manage too much of the experience themselves. This situation is untenable and must change. Fortunately, three major technology trends are creating the needed opening for improving the browsing experience.

Certainly I can't hope, in a short blog post, to lay out the vision that the white paper carefully develops in many pages. You'll have to read it to get the full story; but, we structure the browsing experience using explicit episode context to the benefit of users and Web-site operators. This paper explains why that's desirable and explains the system we've developed to make it happen. I hope you'll take a few minutes to review the paper and let me know your thoughts.

I should mention that we owe a great debt to Craig Burton for patiently working with us to clarify and strengthen our vision. He worked magic and has our gratitude. I'd recommend him without reservation to anyone developing a product vision and strategy.

Tags: kynetx structured+browsing context information+cards

I've had a handful of people ask if they could stop by Kynetx and see what we do. Steve has had similar requests. In an effort to not miss anyone who would like to visit Kynetx and get a demo of our fledgling product, we're hosting a Kynetx Demo Lunch on Friday, Feb 6th at 11:30 at Kynetx World Headquarters in Thanksgiving Point. This map will give you directions and we're in Suite 275 (metal doors). We'll supply the pizza, you bring your curiosity. Please RSVP so we know how much pizza to buy.

Tags: kynetx utah events

I'm going to be speaking at GlueCon in Denver on May 12-13. The overall theme of the conference is that there is a lot of interesting stuff happening in what we might have thought of as "glue" before--all the code that holds things together. Turns out that there's plenty of value you can add in the glue that makes the resulting mash-up better.

Glue is a new conference (the best kind) and is being organized by Eric Norlin, Seth Levine, and Phil Becker. These guys do good conferences. Eric and Phil were the founders of DIDW. More recently Eric's been doing Defrag. Follow the Gluecon blog for more info.

My session is titled "Building Context-aware Applications using Identity as a Foundation." The gist of the talk is about how Web-site independent identity allows us to create context that spans Web-sites. Of course the most obvious context is an authentication context that we refer to as "single sign on" but it goes well beyond that. This is exactly the problem we're tackling at Kynetx. More on these ideas in this blog between now and then.

Tags: gluecon travel events kynetx

Image by chascar via Flickr

One of the components of Kynetx Network Service (KNS) is a 30K (compressed) static Javascript library. This is mostly a slightly modified jQuery along with some other components. We set the Expires header so that it is cached in the browser for 24 hours. Even still, it's a significant load on our network bandwidth and, consequently, our budget.

When Amazon's CloudFront (CF) was announced, we realized that we could move these kinds of static files to CF as a way to reduce our bandwidth and maybe get a performance improvement. If you aren't familiar with it, CF is an Amazon service that fronts their S3 storage system to provide worldwide access & distribution over the 'Net. It's not quite a content delivery network (CDN) but is a substitute for one for many companies.

CloudFront works by fronting S3. Thus, to get something on CloudFront, you upload it to an S3 bucket. You configure CloudFront to point to an S3 bucket and then anything you put in that bucket is available through CloudFront.

Because CloudFront is a simple pointer, if you will, to an S3 bucket, there's no way to tell CF when that item has changed and naturally Amazon doesn't want to check it each time. CF just happily serves up whatever is in it's cache until the object expires. The shortest expiration time you can set on a CF object is 24 hours.

Many items you might serve from CloudFront never, or rarely, change--pictures, for example. In Kynetx' case, the files do change from time to time as we update the library. While you could decide to just live with that, it's not acceptable if you accidentally push a faulty library out. Can you imagine having this conversation? "Gee we're sorry Mr. Customer. We know it's broken. If you'll wait 24 hours, we'll try again."

The common solution seems to be timestamping the filenames so that you just create a new file to serve from CF each time. When you also control the reference to that file, that's a great solution. We don't. The reference is controlled by our customer and telling them all to update their systems each time we want to push a new file is untenable.

You know the saying "there's no Computer Science problem that can't be solved with a layer of indirection." In homage to that, our solution is to redirect from a static URL to the timestamped filename. That's not ideal, but it works. The downside is the other half of that homily: "there's no performance problem that can't be solved by eliminating a layer of indirection." Still, our measurements show that the price we're paying isn't too high.

We put this change in place and tested it for many weeks and then rolled it out to all our customers. So what do we get? Look at this graph showing the dramatic drop in our outbound traffic when we put the change in place:

I did a workup comparing what we'll pay through Amazon and what we were paying for our network traffic and I'm convinced the savings will be nearly as dramatic as the preceding graph.

Tags: amazon aws web+servics cloudfront kynetx operations

Last week Jeff Atwood wrote a piece about why throwing hardware at a problem makes sense in some situations. He's careful to avoid the trap of thinking that throwing hardware at a poorly designed program will do more than buy you a little time, but makes a good point:

Clearly, hardware is cheap, and programmers are expensive. Whenever you're provided an opportunity to leverage that imbalance, it would be incredibly foolish not to.

From Coding Horror: Hardware is Cheap, Programmers are Expensive
Referenced Sat Dec 27 2008 15:42:36 GMT-0700 (MST)

Riffing on that, Gordon Weakliem makes a related and somewhat contradictory argument: Programmers are Expensive, But Operations are More Expensive. Gordon's point is that hardware has to be managed:

It's pretty simple to manage a single server, but each additional server starts to increase operational complexity. Before long, you're hiring people to manage the additional servers, and these people have a different skill set. They're not programmers, they're experts at running networks. A good engineer is a godsend to a programming team, relieving programmers of all kinds of tasks that shouldn't be distracting them, and that programmers probably aren't that good at anyway. But operations experts aren't cheap either, maybe not the price of a programmer, but not cheap by any stretch. Furthermore, horizontal scale introduces new problems: power consumption, heat dissipation, etc.

From The Eighty-Twenty Solution: Programmers are Expensive, But Operations are More Expensive
Referenced Sat Dec 27 2008 15:44:33 GMT-0700 (MST)

Gordon later states that large, complex systems of servers have to be carefully deigned to be manageable.

It's true that services like Amazon and Google are examples of massive horizontal scale, but these systems didn't happen by accident. They were the result of careful planning and good architectural principles. The danger in the "throw hardware at the problem" aphorism is that it's really easy to ignore true architectural problems in the process. As always, there's a balance between thinking an acting that's at play. Like all silver bullets, Moore's Law comes with caveats.

From The Eighty-Twenty Solution: Programmers are Expensive, But Operations are More Expensive
Referenced Sat Dec 27 2008 15:46:11 GMT-0700 (MST)

Most modern as-a-service offerings rely on multiple servers with myriad functions. It's silly, and I'd sure Jeff would agree wholeheartedly with this, to ignore the architecture of this system. Further, not using infrastructure automation to take as much operational expense out of the system is foolish.

I've been singing the praises of Puppet for months. Kynetx is a long way from achieving our operational goals, but what we're done using Puppet and Nagios allows two very part time people to manage and monitor more than a dozen servers. If you're not using Puppet to manage your servers, you're spending more time and money on operations than you need to.

Tags: kynetx operations programming engineering puppet nagios automation cs462

Yesterday Azigo (Parity) launched it's RemindMe service. RemindMe uses Information Cards to represent relationships that are meaningful to the shopper and then overlays Google and Yahoo! with reminders to indicate when those relationships might be valuable. For example, if you're a AAA member, you get a discount at Hertz. RemindMe will alert you to that fact.

The reason I'm jazzed about it is that Kynetx Network Service (KNS) is being used to provide the overlays. In our partnership, Azigo is supplying the cards and selector (including the browser add-on). The add-on calls our service depending on what cards are installed and we update the page.

The selector only works on Windows for now, but the service works on both Internet Explorer and Firefox. KNS is browser and system independent, so it's a good combination. If you're on Windows, go install the selector and a card or two and try it out. As an example, install the AAA card and then do a search for "hertz" or "car rental."

Kynetx is convinced that Information Cards are a key technology that will be necessary to transform the current ad hoc Web experience. A firm foundation of identity allows context-aware services to be built that are both general and powerful. Combining cloud based services with an Information Card-based browser add-on yields real transformative power. Without it, Information Cards are all about data. With KNS, Information Cards are about action.

Tags: kynetx azigo infocards kns

Image via Wikipedia

I've spent the last several days immersed in debugging Javascript in Internet Explorer. It's not as bad as it could be. After all, I'm using jQuery and that cuts down on the cross-browser incompatibilities.

In it's normal usage, Kynetx Network Services (KNS) compiles our domain specific language, called KRL, into Javascript. Since we test that the Javascript is compatible with multiple browsers, normal rule evaluation is pretty safe.

However, like any good DSL, KRL has a way of dropping out of the DSL and into the base system--in this case Javascript. That's when things get dicey. There are so many ways of creating browser problems when you're working with Javascript and CSS.

In Firefox, the venerable Firebug add-on is the tool for debugging CSS and Javascript. I use it's console all the time to execute Javascript against the page and see what works and what doesn't.

Alas, Firebug isn't available for IE. There are a few "for pay" tools that are supposed to be similar, but I haven't been able to see to my satisfaction what I'd get for my money. I also understand that Visual Studio has good tools.

When I was at Add On Con last week, Joshua Allen from Microsoft said that you should develop in IE8 because it has the best developer tools. I found that to be true--although they still leave a lot to be desired compared the Firefox.

The first thing you have to do is turn on debugging in IE by making sure that the "Disable script debugging" options are unchecked.

The error messages you get are pretty cryptic, but they at least give you a clue about where to look. The worst part is that they don't give you the name of the file where the error occurred. Ugh. @whitmer suggested Companion JS (not free for commercial use). I haven't tried it, but it looks like its a step above the stock debug messages.

Another good tool is the IE Developer Toolbar (free). This gives you the ability to browse the DOM, change CSS properties on the fly, clear the browser cache, etc. As far as I can tell there's no actual Javascript debugging and certainly no console. Still, being able to see the DOM, especially as its modified by Javascript components has been a great help.

I found this tutorial on using Web Developer Express Edition to debug Javascript, but I haven't tried it.

So far, that's what I've discovered. I'm sure there are better ways and tools. I look forward to someone educating me.

Tags: debugging internet+explorer kynetx firefox

Image via Wikipedia

The Business Technology blog at the Wall Street Journal reported last night that several ecommerce merchants experienced slowdowns and in some cases service disruptions due to higher than expected demand.

It shouldn't come as a surprise that people are shopping online Friday. This year, retailers have been promoting online sales more heavily than in the past. In the case of Sears.com, the promotions seem to have worked too well: The site was unavailable for many visitors between 10 am and 12:40 pm EST Friday, according to Keynote. (It was also down when we tried to access to access the site at 3:30 EST.) A spokesman for Sears said that traffic was "higher than anticipated" and that the company was taking steps to ensure the site would be available on Monday---another popular online-shopping day.

From Business Technology : Retail Sites Crash as Shopping Season Opens
Referenced Sat Nov 29 2008 10:56:56 GMT-0700 (MST)

One of the realities of the online retail world is that most online retailers lock down their sites sometime in late October and don't make code changes and sometimes even template changes until after Christmas. There's just no point taking a chance during the time you collect a disproportionate chunk of your revenue. For ecommerce tool vendors, that means that the fourth quarter can be slim.

I have a feeling that most retailers aren't taking advantage of cloud resources to scale with demand. I'd love to see more data on that. With this years failings and the increasing popularity of cloud computing, maybe next year will be different.

Tags: online+shopping capacity+planning ecommerce kynetx

Image via Wikipedia

Pat Eyler sent me a note asking if we were still using puppet. In the spirit of trying to leverage my response to him for the widest possible audience, I thought I'd blog the answer.

Kynetx has been using Puppet (and a little Capistrano) to automate our infrastructure since this summer. While we haven't gotten everything as automated as we'd like, we're quite a ways down the road and it's making life as a small startup bearable.

My ultimate goal is to register a node in the system that delivers Kynetx Network Services (KNS) to the machine database (iClassify) and have it come up, configure itself, and be ready to put in production in less than 15 minutes. While there are still a number of manual steps in that process we are remarkably close and getting closer all the time.

I haven't found learning puppet to be difficult. There's plenty of documentation, including recipes that if not usable as is, are great resources for figuring out best practices. Also, there's a very helpful group on IRC (#puppet on irc.freenode.net) that I've turned from time to time.

The best thing about Puppet is that I can count on nodes under puppet management to be consistently configured and the puppet recipes represent an unambiguous record of what should be happening.

A recent example, we needed to ensure that data files from an OEM partner were on all of our initialization servers. I used Puppet in two ways:

1. Puppet is used to distribute the data files from the ops server to each of the initialization servers.
2. Puppet is used to manage the cron job that grabs updated data from the partner and put it in the right directory for distribution.

If I add another initialization node puppet will make sure it has the right data files just because I declare it to be an initialization node. Nothing else need occur.

I recently released Kynetx's operational procedures. For a small company with no fulltime engineering staff other than me, I think we do a pretty good job operationally. Much of that we owe to Puppet.

Needless to say, I remain, more than ever, a Puppet fan.

Tags: puppet kynetx

In the spirit of transparency, Kynetx has published our operational procedures and practices online. As we gain more control over our infrastructure automation, we'll also have real-time operational information and statistics.

I'd appreciate feedback on what's there. Keep in mind, this isn't out "dream" but rather we're trying to present a true picture of where we're at. We obviously have goals to make this better and have quite a bit of experience in mature operations so we know where we need to be and where we currently fall short. I don't believe that there's anything there that would compromise operational security, but i you think there is, please let me know.

Obviously we're taking some risks in being open and honest about the true nature of our operational capability as a young start-up. But I believe that in the long run transparency will win out and our customers will appreciate the openness, recognize that we're a start-up with lots of competing priorities, and not punish us for places where we may fall short compared to more operationally mature companies.

Tags: kynetx operations transparency

Dave Winer

Dave Winer wrote yesterday about the death of online advertising. He says:

I've been saying it for as long as people have been building businesses on advertising on the web, it's not a longterm thing. Now we're at the end of the road.

Assuming the economy comes back from the recession-depression thing that it's in now, when it does, we will have completely moved on from advertising.

The web will still be used for commercial purposes, people will still buy things from Amazon and Amazon-like sites, but they will find information for products as they do now, by searching for it, and finding out what other people think, not by clicking on ads and buying things on the pages they link to.

No one needs advertising, and there are much better ways to sell products.

From Online advertising is now dead (Scripting News)
Referenced Fri Nov 14 2008 08:02:18 GMT-0700 (MST)

I don't know that I'd go as far as saying it will not come back at all, but merchants will find new, innovative ways to reach customers without advertising when their budgets get tight.

Of course, we might call these new things "advertising." I get that frequently when I explain what Kynetx does. People call any message from a merchant an advertisement even though, in many cases, it's not widely broadcast.

There were plenty of discussions relevant to Dave's post at IIW this week. The VRM crowd was there in force with grounded discussions about how the implicit contract between merchant and shopper can be rewritten in ways that are better for both.

Most exciting to me, the discussions about r-buttons were getting down to the level where you could see real protocols and standards developing behind the talk.

I was troubled by some VRM discussions that still see to border on being hostile to merchants and even commercial efforts in general. I think that only hurts the opportunity to have to redefine what commerce is and how it takes place. I also think that some of the discussion gets lost in relationships in general, not just those that exist between the merchant and shopper. Nothing wrong with figuring out infrastructure for relationships, but it's nice to keep things focused on the task at hand.

The next steps we take can be as important as the final end state. Dave's insight that the economic downturn opens up opportunity is a good one. At Kynetx we're working on new ways for shoppers and merchants to relate. We're creating new channels for relevant messages--both from shopper to merchant and from merchant to shopper. Not advertising--just plain old communication.

Tags: ads google ecommerce iiw vrm kynetx

Here's a few good articles by Eric Ries I ran across today on startups.

In A Hierarchy of Pitches, Eric talks about aligning your pitch with what stage your company is in. This is helpful analysis and acknowledges that companies are at--and get funding at--a variety of levels in their maturity. Eric gives ideas about what investor questions are relevant at each level and what slide you have to focus on. If you're getting questions that seem misaligned with where you think you're at, then you should re-evaluate something.

In Lean Startups vs Lean Companies Eric defines what it means to be a lean startup in a very helpful way:

I suggest [startups] define waste as "every activity that does not contribute to learning about customers." (aka "how you get to product/market fit.") This is why I find the concept of the Ideas-Code-Data feedback loop so helpful. Any activity that actively promotes us getting through each iteration (including the learning phase) faster, is value-creating. Everything else (including a lot of traditional "agile" or "lean" tactics) is waste.

That's clarity around the key idea of what it takes to make a startup work.

Tags: startup kynetx customers

I've been creating screencasts to show what Kynetx does for VCs and business development. Using screencasts is a quick, easy way to do a demo that doesn't fail and is self contained. I can email the link to a screencast to someone and they get a good idea what I want them to know.

I've been using SnapZPro to record browser sessions and then using iMovie to create the actual movie. SnapZPro works great, but I'm not as impressed with iMovie. Maybe it's just me, but I find it inflexible and hard to use. If you've got suggestions on better programs, I'd love to hear them. I'm interested in

• Cheap -- we're a startup, after all.
• Ability to easiliy lay in audio after the fact. I like to script and record the browser interaction and then add the explanation.
• Good title slide capability. This is one place iMovie left me cold. I wanted to add what amounted to slides, and the internal capabilities were scare. I had to create slides, export them, and then use those pictures.
• Easy to understand and use--I'm not a video editor.

Tags: osx screencasting imovie kynetx

Paul Graham has a new essay on Why to start a startup in a bad economy. Actually, he doesn't necessarily think starting in bad times is better than good times, just that there's nothing special about good times when it comes to starting high-tech businesses. He says that the state of the economy is a rounding error compared to the effects the founders have.

The economic situation is apparently so grim that some experts fear we may be in for a stretch as bad as the mid seventies.

When Microsoft and Apple were founded.

As those examples suggest, a recession may not be such a bad time to start a startup. I'm not claiming it's a particularly good time either. The truth is more boring: the state of the economy doesn't matter much either way.

If we've learned one thing from funding so many startups, it's that they succeed or fail based on the qualities of the founders. The economy has some effect, certainly, but as a predictor of success it's rounding error compared to the founders.

From Why to Start a Startup in a Bad Economy
Referenced Sat Oct 18 2008 07:02:32 GMT-0600 (MDT)

I enjoyed reading it since I'm in the middle of starting a business now. I would disagree with Paul only in that I think bad times are better for starting businesses in some ways. Here's one: it's easier to find good talent.

Tags: startup kynetx entrepreneurship

I'm looking for someone to build a simple plugin for Internet Explorer. We use the plugin for testing Kynetx. I already have a Firefox version, but need to test against IE as well. If you've got expertise and would like to know more, contact me.

Tags: internet+explorer kynetx ie

The code that makes Kynetx work is a couple of custom Apache modules written in Perl. So, configuring machines via puppet, naturally requires ensuring that a set of Perl modules are loaded.

For a long time, I was using a private bundle, but I found that was unreliable. For the most part it worked fine, but then sometimes I'd get an error that Perl couldn't find the bundle, even though it was clearly in the path.

I didn't want to spend a lot of time debugging it because I figured it was a dead end. Eventually, I've got to be able to control more of what happens. So, I decided to define a custom Puppet type that used CPAN directly to load Perl modules. Here's what I came up with:

define cpan_load() {
  exec{"cpan_load_${name}":
     command => "perl -MCPAN -e 
             '\$ENV{PERL_MM_USE_DEFAULT}=1; 
               CPAN::Shell->install(\"${name}\")'",
     onlyif => 
          "test `perl -M${name} -e 'print 1' 2>/dev/null || \
                 echo 0` == '0'",
     require => [Package["perl-CPAN"],
                    File["/root/.cpan/CPAN/MyConfig.pm"]],
     }
 }

The hardest part to get right was the onlyif clause. The purpose of this clause is to ensure that the command is idempotent--that is that it only runs if it needs to. The truth is that CPAN itself is idempotent, so I could have skipped it, but I saved 80 seconds of processing time for a "do nothing" run of puppet by installing it.

I got something more too. CPAN will report things are installed when they aren't really. I don't know how it gets in this state, but it does. My onlyif clause only reports a module is present if a perl program can load it.

This still isn't perfect since when the load fails and you rerun the puppet file, you see it trying to install them over and over and it's not going to happen. Then manual intervention is required. I'd rather have the system back out and fix the error but that's beyond me right now.

I think the answer may be to move to Tasks, but I'm not sure. If you know better ways of managing Perl module installs with Puppet, please let me know.

Tags: puppet cpan perl kynetx

I recently ran into a problem with Prototype (the Javascript framework) that exposes a big difference between the design philosophy of Prototype and other frameworks like JQuery.

Consider this code:

var aryOmnDiv = [
   'header_tracking',
   'left_nav_tracking',
   'footer',
   'homepagecategories',
   'subcats',
   'bread_crumb',
   'subcat_wrapper',
   'prod_group_wrapper']; //container divs

for(var outside_i in aryOmnDiv) {
   var divname = aryOmnDiv[outside_i];
   var div = $(divname);
   if(div) {
	var anchors = div.getElementsByTagName('a');
   }
}

Running this with Prototype installed produces and error that says "getElementsByTagName is not defined." The culprit is that the code is using a for...in loop as a shortcut for looping through the array. The official ECMAScript standard says (§12.6.4) that the for...in construct exists to enumerate the properties of the object appearing on the right side of the in keyword. It's only an accident that it also worked for looping through arrays because stock arrays have a sparse property set in Javascript.

The problem is that Prototype changes that adding dozens of new properties (functions mostly) to the Array object. This means that the loop above gets past the Array indices and keeps on going, setting outside_i to each of the Array properties in turn. Of course, those aren't valid array indices and so the code eventually errors out.

As explained on the Prototype site, this can easily be fixed by changing this line:

for(var outside_i in aryOmnDiv) {

to this line:

for (var outside_i = 0; outside_i < aryOmnDiv.length; ++outside_i) {

Alternately, you could make use of the nifty new iterators that Prototype adds to Array like each.

I think I remember reading about this on the Prototype site before but it didn't mean much to me until I ran into it and spent some time looking at it.

This problem points out a fundamental difference in the design philosophy of Prototype and other frameworks like JQuery. Prototype extends base objects (like we've see with Array) whereas JQuery doesn't. Put another way, almost all of the new functionality in JQuery is neatly sandboxed in the JQuery namespace whereas with Prototype that's impossible.

The extensions that Prototype makes to Arrays are nice. I've used them and like the coding style, but be aware that this isn't a neatly packaged library that plays nicely with other Javascript you might already have. It's a change to Javascript itself yielding a new thing. Very cool that Javascript allows that, but be careful not to shoot yourself in the foot.

Tags: kynetx prototype javascript

This Wall Street Journal article talks about how geolocation data can aid online commerce. I find it interesting because Kynetx takes this data and layers other services on top of it and then makes the whole thing incredibly easy to integrate with any Web site.

This story about ACE Hardware is exactly what Kynetx does today:

Consider Ace Hardware, a cooperative of more than 4,600 dealer-owned hardware stores that overhauled its Web site in February 2007. Using geolocation software from Digital Element, a unit of Norcross, Va.-based Digital Envoy Inc., Ace now gives Web-site visitors a list of stores within a 30-mile radius of their location. A click takes visitors to a page showing the stores plotted on a Google map, and from there, they can drill down to the various stores' Web sites for information such as hours of operation and driving directions.

Dana Kevish, Ace's e-commerce marketing manager, says geolocation is important to Ace because some 75% of online sales go through local stores -- where Ace will ship products free -- and 30% of customers who pick up an online order make an additional purchase in the store. Having geolocation capabilities "helped us create a connection between the consumers and the local store," she says.

Ace plans to expand its use of the technology this winter by directing Web visitors to one of five or six different home pages, depending on their location. People in cold climates will see a home page featuring snowblowers, for example, while Floridians might see patio furniture.

From We Know Where You Are - WSJ.com
Referenced Mon Sep 29 2008 10:11:53 GMT-0600 (MDT)

Tags: wsj kynetx geoip

A few weeks ago Yammer walked away with top honors at TechCrunch 50. I'd been hearing about it, so last week I went over and signed up. I'm really liking it.

Yammer is Twitter for work. The first person to sign up using an email from a particular domain establishes a sandbox for yammers from people in that domain. Since I signed up using my kynetx.com email, Yammer automatically created a domain for Kynetx and made me an administrator. Very low friction.

I sent out invites and soon had the whole Kynetx gang yammering away about work. Only people within your domain (i.e. those with emails you control) can see what the others are saying. Yammer has a Adobe Air client that makes keeping up with what folks are saying easy (you can also use the Web). In addition, Yammer has a number of features that are geared for business like the ability to input and display detailed contact info and organizational information as a crude social network.

We've been using Yammer for a week now and everyone feels it's been a good addition to our communications patterns. This is especially true with a distributed workforce like we have now.

So what of Twitter? Most of us still use Twitter too. Twitter is my public space. Yammer is my work space. That separation has real value and enables conversations and uses for Yammer that just wouldn't happen on Twitter.

We've made real decisions over Yammer that would have previously taken a phone call or email chain. Generally people are pleased with the connectivity it provides. For us, Yammer is a keeper.

Tags: yammer twitter microblogging work kynetx

The system we're building at Kynetx includes a domain specific language that uses rules to create JavaScript programs that get sent down to the browser. I've documented our decision to use a domain specific langauge and our choice of Perl in other posts.

When I started this project, I was reading Mark Dominus' book Higher Order Perl and started using his HOP parser to play around with. One thing led to another an before you know it I had a full blown language parser in HOP without giving much thought as to whether or not I'd made the right choice.

I found the HOP parser to be pretty flexible, but it has it's quirks. More importantly, I didn't like the BNF specification format and so I was constantly trying to keep the spec and the implementation in sync. Better if I could just use the spec as the implementation 'ala Bison. Don't get me wrong, this is a great book with lots of wonderful ideas, but I wanted something else for the parser.

As I added more and more features to the language, it got to where I'd dread making the parser changes. Recently, I decided I had to significantly beef up the predicate expressions and thought it would be a good time to change out the parser as well.

A few months ago I picked up Christopher Frenz's Pro Perl Parsing in anticipation of just this day. Reading through it illuminated my choices and ultimately, I picked Damian Conway's Parse::RecDescent, a recursive descent parser over the other contender, Parse::Yapp. The reasons for my choice were partly esthetic and partly a trust in Damian. The main thing I was after was a parse spec that I could read and compile and RecDescent seemed great in that regard.

The biggest downside of RecDescent is that there's no associated Lexer. For most things that's not a big deal since terminals can be specified as regular expressions. The place where it really bit me was comments. Removing comments is trickier than you'd think because you don't want to process "start of comment markers" inside any quotes. With a lexer, that's easy; without one, it's more problematic. Writing the regexp to remove comments took me a while to get right. I ended up using a modified version of the solution given in this FAQ. The problem with most solutions, including Regexp::Common, which has a language comment module, is that they don't account for comment markers in quotes.

All in all, rewriting the parser was a good exercise and I'm happy with the choice of RecDescent. Here's a sample production from my file:

decl: VAR '=' VAR ':' VAR '(' expr(s? /,/) ')'
      {$return =
       {'lhs' => $item[1],
        'type' => 'data_source',
        'source' => $item[3],
        'function' => $item[5],
        'args' => $item[7]
       }
      }
    | VAR '=' 'counter' '.' VAR
      {$return =
       {'lhs' => $item[1],
        'type' => $item[3],
        'name' => $item[5]
       }
      }
    | VAR '=' HTML
      {$return =
       {'lhs' => $item[1],
        'type' => 'here_doc',
        'value' => $item[3]
       }
      }
    | 

This production for decl has three alternates. Each has a separate return value (a hash) that represents the portion of the abstract syntax tree created for that part of the input.

If you decide to give Parse::RecDescent a try, here are some resources:

• The Parse::RecDescent documentation
• The Parse::RecDescent tutorial
• The Parse::RecDescent FAQ
• An O'Reilly getting started-style tutorial
• An example showing SQL CREATE TABLE statements

Reading the documentation and the FAQ thoroughly is highly recommended. There's lots of little tricks that can make your job easier.

My job, replacing an existing parser, was made easier by the fact that I'd previously built a pretty thorough test suite in Perl for the parser and some related modules. So once I got the language spec pretty much complete, I started running the tests and correcting errors as they cropped up. In a few hours, I'd solved all the problems and was confident my parser was ready to go. Definitely the way to go.

At any rate, now I've for a shiny new parser that I can go modify. Fun!

Tags: kynetx perl programming+languages parsing

If your Web service does anything that sets cookies, you'll probably bump up against the fact that Internet Explorer--since version 6--has implemented a fairly strict privacy policy regarding cookies. In a nutshell, if the site does not have the right P3P privacy policy, first-party cookies (i.e. from the site itself) are downgraded to session cookies and not stored in between browser sessions and third party cookies (i.e. from another site) are rejected completely. Here's what to do to solve this problem.

P3P, or the Platform for Privacy Preferences is a W3C "protocol allowing websites to declare their intended use of information they collect about browsing users." In IE 6 and 7, users can use a slider bar to set their desired degree of privacy and then IE will automatically check the privacy policy of the sites they visit and "protect" them according to their preferences. The default setting (medium-high, which most people never change) gives the behavior I describe above.

Deploying a P3P policy actually isn't very hard. There are some great tools for creating the policy itself. But it can be difficult to know exactly what to do. I followed these instructions but still have a few questions, so I'll document exactly what I did below.

The first step is to create the policy. I used IBM's P3P policy editor. It's a Java program, so it will run most anywhere. Using the tool takes a little work since it's not clear at first what you're editing. Create your policy from a template if you can since that will save a lot of decisions later. Once you've done that, select Policy->Policy Properties and fill in the information about your service and organization. If you look at the errors, you see that you have to fill just about everything in. Make sure you add a "privacy seal" even if it's just a notice that your customer service department can answer questions.

The policy itself is in the "groups" on the right. Double click each one and make sure you agree with what it says. Clicking on "Errors" will show you things left undone and clicking on "HTML Policy" will show you the human readable version of what you're creating. At the bottom it provides an analysis of how this policy will play in IE. Very helpful.

When you're done and there are no errors, you need to save four things:

1. The policy itself as name.xml where name is the name you selected under "Web Sites" in the Policy Properties pane. You will likely have just one, but you can have many covering different parts of your site.
2. A policy reference file as p3p.xml. This file provides discovery services for the policies. Whether you have one or many policies for your site, this file tells programs which policy applies where and how to find them
3. A human readable policy
4. A compact policy. This is a string of three and four letter acronyms that specify the policy in a compact manner.

Put the first two in http://yoursite.com/w3c/... Put the third in whatever URL you specified the human readable policy would be referenced by.

The compact policy is used in the HTTP headers that your server returns for ant HTTP request. This gets rid of one or more round trips to the server to request the XML version of the policy. In my experience, this was a necessary step to get IE to recognize the policy.

Having Apache return the compact policy in the header requires building and installing the mod_header module. I'd already done that so I simply added this line to my HTTP configuration file:

Header append P3P "CP=\"NOI DSP ADMo DEVo TAIo ... DEM STA\""

Once you've got all this installed, you should be able to open IE, double click on the eyeball with the red slash through it in the status bar and confirm that your cookies are no longer blocked. If there are no blocked cookies, the eyeball is not there at all.

That's it from a technology standpoint. The trickier part is deciding whether you can actually live with the restrictions you'll need to put in place to let IE store your cookies.

The whole thing feels like a waste of time. Your product won't be better and most people won't be any more protected when your done. But you need to do it in an IE world.

Tags: kynetx privacy internet+explorer

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the fifteenth installment. You may find my efforts instructive. Or you may know a better way---if so, please let me know!

One of the things that ought to strike fear into any technologist's heart is getting your first customer. You've spent months building a product you hope people will buy and find useful and suddenly someone actually is putting their faith in you and your baby.

You're happy, of course. But most of all, you're scared. What if something doesn't work? What if it doesn't scale? What about feature A, B, and Z that we haven't built yet? What about the shortcuts you took here and there to get it done? What about...

I've done this a couple of times now and I'm convinced of two things: (1) you're going to sell your service before you're ready and (2) you should have sold it even earlier.

We're at that stage with Kynetx. We've signed up several good pilot customers and we're actively courting the indirect sales channel. These are sharp, demanding folk with good ideas and little time to waste. That scares me.

Don't get me wrong. I've never built a product or service that has been more ready to scale and that I'm as confident works as promised. We've spent months automating infrastructure. I've written thousands of tests. Still, it's scary. If you asked me, I'd take another couple of months. Of course, at that point, I'd still take another couple of months. You're never ready.

That leads nicely to my second point. Since you're never ready, you might as well start selling even earlier. There are big advantages to having real people kick the tires and give you feedback. The product becomes better faster through that process than anything else you can do. And you invariably plug some of the holes that have crept into things.

But wait. There's more. The best part of selling your product early is that it helps you understand what you do and how you're different from your competitors. These are two key pieces of information when you're going out to raise money.

You may think that it's weird that you could start a company and not know what you do, but in fact it's quite common. Oh, you know what you do on many levels, but telling other people about it can present some challenges. In an earlier post in this series I wrote about finding your story. Same idea.

When you start to tell others about your product or service, they invariably try to understand you by comparison. "Oh, so you're just like [insert company name here]" or "Oh, so you're a [insert product category here]." Often these are the very companies or categories you're trying to distinguish yourself from. You need to understand how your different in ways that you can easily explain.

Going through some VC meetings and pitching events helped us refine our story, but we're reached a whole different level talking about what we do with clients. They're usually more willing share, explore, and engage in a dialogue than VCs who typically hold their cards very close to their vest.

Clients aren't as interested in comparing as VCs because they don't care as much who you're like. They're not trying to find the next category killer. They just want to solve a problem and if your nail will secure their board, then they're game.

Besides, I don't know about you, but in client meetings I'm a whole lot more relaxed than I am in a VC pitch. I'm in my element--showing off the things I'm most passionate about right now. Clients usually want to see what you have and our demo usually wows people. VCs never want to see a demo.

As a consequence of all this, I wish we'd been positioned to start selling well before we started raising money. Unfortunately, because of various timing issues and our own understanding, things didn't work out that way. If I do this again, I'll start selling much earlier.

Tags: kynetx startup selling

Dave Cross has a piece on why corporations hate Perl. He's being a little hyperbolic (as he admits)--not everyone hates Perl, but he's right in noting that there is a backlash against it. He says:

I was talking to people from one such company last night. The Powers That Be at this company have announced that Perl is no longer their language of choice for web systems and that over time (probably a lot of time) systems will be rewritten in a combination of Java and PHP. Management have started to refer to Perl-based systems as "legacy" and to generally disparage it. This attitude has seeped through to non-technical business users who have started to worry if developers mention a system that is written in Perl. Business users, of course, don't want nasty old, broken Perl code. They want the shiny new technologies.

And so, in a matter of months, the technical managers at this company have create a business environment where Perl is seen as the cause of most of the problems with the current systems. It's an impressive piece of social engineering

From Why Corporates Hate Perl - O'Reilly ONLamp Blog
Referenced Wed Aug 20 2008 10:43:00 GMT-0600 (MDT)

He labels this kind of things "anti-Perl social engineering." I've run into similar feelings.

At Kynetx, our systems are built in Perl and Ruby. We use Rails to create the user facing pieces, but the engine, that part that needs to run fast, is written as an Apache module in Perl.

I've noticed that when I explain the language choice to people I almost always have to answer the spoken or unspoken question: "Perl?? Really? Why Perl?" After I explain the architecture and the reasons behind the decision, they're almost always satisfied, but there's a definite stigma.

I'm doing a talk next week at the Utah Open Source Conference (Friday morning at 10am) on using Apache as an application server. There are some significant advantages and Apache offers some real great application support that is largely untapped. But to talk advantage of it, you're pretty much stuck with C...or Perl. Given that choice the decision is a no-brainer. I'll be putting my thoughts together for the talk later this week or next week and promise to blog the ideas.

Still, the idea that Perl is an old crusty language couldn't be more wrong. I've been programming in Perl for nigh on 15 years now and I don't think it's ever been more interesting. Perl sports the best collection of libraries of any dynamic language and supports modern engineering practices like modules, objects, and testing. I've seen ugly Perl code--heck I've written it--but that's not Perl's fault.

I spoke with a local company last week that has been a Perl shop but is moving to Java. I smiled because I know that means there will be a lot of unhappy Perl programmers there who I can hire later! :-)

I'll continue to proudly carry the torch for Perl.

Tags: kynetx perl programming+languages

We were notified that Kynetx will not be one of the TechCrunch 50. Ah well... The rejection notice said this:

We are sorry to let you know that your company was not selected as a finalist for the TechCrunch50 conference. As you know, we are only able to select a very, very small percentage of the more than 1,000 outstanding applications we receive.

Your company was among a select set of candidates that we considered, and it was a difficult decision driven purely by the limited number of presentation slots. Since we regarded your business so highly, we want to make sure you still get the opportunity to participate in the conference in our DemoPit.

As a DemoPit company, you will have the opportunity to be nominated for the People's Choice award and win the 50th spot on the TechCrunch50 main stage. As the 50th company to present, the People's Choice award winner will be able to compete for the $50,000 TechCrunch50 award. Act fast, as spaces are very limited and first come, first served.

Additionally, all DemoPit companies will benefit from the exposure generated by media attending the event. We do anticipate having approximately 300 members of the international press in attendance.

Participating in the DemoPit costs $3000, a not insignificant sum for our early stage start up. My question is "is it worth it?" Any wisdom out there?

Tags: techcrunch kynetx

The whole idea of Puppet is to put your machine configuration scripts in a versioned repository. This is good because I've found that a syntax error on a manifest not even used by the current machine will stop the puppet updates from running. One error anywhere kills the whole thing everywhere. So, being able to back out of a change is good.

We can go one better however and keep people from checking in files that aren't syntactically correct using the pre-commit hook in SVN. If you don't use SVN, your repository probably has something similar.

I followed these instructions for the hook and these instructions for installing it.

I ran into a problem however. The pre-commit script wasn't reporting an error condition correctly. I'm not sure what was wrong. Running the puppet command on the command line and then checking $? gave a "1", but inside the script it wasn't working that way. So, I punted and wrote the error output to a file and checked whether or not the file existed and then just cat'd that out to standard err. Messy, but it worked and I need to go home. Here's the file as modified by me:

#!/bin/bash
# SVN pre-commit hook to check Puppet syntax for .pp files
PATH="/usr/bin:/bin"
REPOS="$1"
TXN="$2"
tmpfile=`mktemp`
errfile=`mktemp`
export HOME=/
SVNLOOK=/usr/bin/svnlook
$SVNLOOK changed -t "$TXN" "$REPOS" | awk '{print $2}' \
   | grep '\.pp$' | while read line
do
  $SVNLOOK cat -t "$TXN" "$REPOS" "$line" > $tmpfile
  if [ $? -ne 0 ]
  then
    echo "Warning: Failed to checkout $line" >&2
  fi
  /usr/bin/puppet --color=false --confdir=/tmp \
     --vardir=/tmp --parseonly --ignoreimport $tmpfile \
     >$errfile 2>/dev/null
  if [  -s $errfile ]
  then
    echo "Puppet syntax error in $line." >&2
    cat $errfile >&2
    exit 2
  fi
done
res=$?
rm -f $tmpfile
rm -f $errfile
if [ $res -ne 0 ]
then
  exit $res
fi

Note: you may need to combine lines separated by "\\" into a single line.

This works great. Syntax errors are caught before they make it into the repository. I'm still looking for a good way to thoroughly test puppet scripts before they get into the production servers. If you have any best practices in that area, I'd love to hear about them.

Tags: puppet kynetx svn

I'm continuing to explore the use of puppet for managing systems. I'm convinced that it's the best system for managing large groups of interacting servers and keeping them all in sync. For example, today I was trying to figure out why a cronjob that processes log files wasn't working. Turns out that it was a system issue and there was also a problem with mail aliases, so we weren't even finding out.

I create a simple puppet recipe for all our machines that simply ensures crond is running. That's basic and now it's ensured on all our machines. This was simple:

class cron {
      service { crond:
	      ensure => running,
	      enable => true
      }		
}

Next I created a manifest to installed the correct crontab entry on the right machines to process the logs along with setting the right environment variables for the processing daemon. There's a cron "type" in puppet so it's pretty easy:

class kobj_log_daemon {

      cron { kobj_log_daemon:
           command => "bin/kobj_log_daemon -v",
	   user => www,
	   hour => 0,
	   minute => 5,
	   environment => ["WEBROOT=/www", "MAILTO=root"],
      }
}

Finally, I took care of mail aliases by putting /etc/aliases under the control of puppet. Here's the class definition that does that:

class aliases {
 file { "/etc/aliases":
   mode  => 640,
   source => 
     "puppet://puppet.kobj.net/dist/apps/sendmail/aliases",
   owner => "root",
   group => "root",
   before  => Exec["create aliases db"]
 }

 exec { "new_aliases":
   command => "/usr/bin/newaliases",
   alias => "create aliases db",
   subscribe => File["/etc/aliases"],
   refreshonly => true,
 }
}

Now there's one place--under version control--where all of this is controlled for every machine we own. Managing this any other way would be a recipe for disaster.

Tags: system+configuration kynetx puppet

Ross Mayfield notes that in an Apple retails store "50% of the space is for retail sales and 50% for service and support." He goes on to contrast that with places like Fry's or Best Buy. I'm always amazed when I go into an Apple store: they're happening places. If you're in retail, visit an Apple store and then go back to your place. Seem kinda quiet and dead. Yeah, I thought so.

Ross goes on:

What Best Buy is missing is the fact that they provide no after market value add with their retail -- in comparison to buying and servicing with an e-commerce vendor. If I buy something in person I expect a person to be able to help me when things go wrong. At least during the manufactures warranty, and I might pay to extend that period with the retailer.

But I think Apple gets something more than the value of customer experience. According to the Consortium of Service Innovation, there is an iceberg effect for product knowledge. 90% of conversations about supporting products never touch the company. Only 10% touch the call center. And 1% of this service and product quality knowledge are assimilated.

In other words, Apple's trying to capture more of the product knowledge conversations. That goes beyond mere "customer experience" and gets to building relationship.

Finally Ross gets to the key question for online retailers:

For your business online, what porportion is dedicated to retail vs. support? When not constricted by the boundaries of physical space, and can be empowered through community, where do you draw that line? What crosses that line is a process not unlike osmosis, where energy is released with the right balance.

When I was at Internet Retailer it was clear that one of the hot features for ecommerce Web sites was customer reviews. More and more places are following Amazon's lead and adding places for customers to talk to other customers (and inform the retailer in the process). This is a great way to capture more of the customer product conversation and capitalize on it in order to keep shoppers coming back for more.

Tags: ecommerce kynetx customer vrm

I’m starting a new business called Kynetx. As I go through some of the things I do, I’m planning to blog them. The whole series will be here. This is the fourteenth installment. You may find my efforts instructive. Or you may know a better way—if so, please let me know!

Most days lately, I've been getting dressed up and talking to people about Kynetx. I'd rather be in jeans writing code, but when you're raising money you're going to dress up more and code less. Raising money is a distraction from running a business and so should be avoided unless it's absolutely necessary. Unfortunately for the kind of business I'm interested in building, there are times when it's necessary. I spoke with the CEO of one venture backed late-stage startup yesterday who said he spends 25% of his time raising money. Ugh.

Going into this process (which I've been doing seriously for several months) I knew some things about raising money, but hadn't ever done it myself. There was always someone else to do that while I made stuff. I've learned a lot and likely still have a lot to learn.

One of the things I've learned is that you're in great peril of losing the sense of magic--the very thing that made this exciting and launched you on the journey in the first place. When you say something over and over again, you get bored with it--even when it's the first time for your audience.

Another thing, like software, the second version of your pitch will be overbuilt. Last summer, based on what we wanted to do, I put together a slide deck. Based on some information we got this spring we completely rewrote it. Once we got good feedback on the second version, we ended up with a third version that was--mostly--just right and much more like the first than the second.

Steve and I went to a FundingUniverse speed pitching event last Wednesday and it was probably one of the most useful things we've done in this whole process as far as getting our story down and getting the pitch smoothed out.

If you haven't seen one, it's like speed dating, but there are angel investors sitting at the tables instead of women (or men, depending). The entrepreneurs move from table to table. You have 7 minutes at each table: 4 minutes to pitch your idea and 3 minutes for questions.

The investors expect the full pitch--everything--in those four minutes. When we first heard this, we thought it was impossible. Turns out you can do it. And when you're done refining your pitch to four minutes and then giving it a dozen times (after practicing it dozens more) you'll have be very good at explaining what you do.

That last point is important. Explain what you do. That's the most important thing you can do in any pitch--long or short; Guy Kawasaki is right: the most important thing you can do with any audience is tell them what you do. When we first started practicing the four minute pitch people would say "I'm still not sure what you do" when we were done.

Of course we told them in the first slide--or thought we did. But what you think you said and what people hear when they're unfamiliar with the idea and bringing their own assumptions to bear are two different things. You need to them over and over again in many different ways before they'll really understand. All in four minutes.

Of course, the primary idea is to generate interest and get a chance to sit down in a more relaxed atmosphere to really get into the details. We were pleased to get 5 or 6 people express interest in another meeting from the group and we'll be following up in the coming weeks. If you get a chance to speed pitch do it. It's a great exercise--even if it doesn't lead to funding.

Tags: startup kynetx venture+capital

Someone asked me for a one-paragraph description of Kynetx today. Here's what I sent them:

Kynetx is an early stage ecommerce company focused on turning shoppers into customers. We provide online merchants with easy to use tools and services that give them the ability to merchandise in real-time to shoppers according to the shopper's individual context without changing their existing ecommerce toolset. Shoppers get a better shopping experience without sacrificing their privacy. Merchants sell more. The management team includes experienced ecommerce veterans Steve Fulling (CEO) and Phil Windley (CTO) who created the iMALL product strategy that successfully led to a $450M exit in 1999.

The last sentence isn't so important for someone wanting to know what we do, but it's good information for investors. Comforting, I hope.

Tags: kynetx ecommerce venture+capital

I've been trying to use Capistrano for application deployment over the last few days, writing rules to do some common tasks, figuring out how it works, etc. One problem I ran into is that I have a private CPAN bundle that I use to ensure a machine has all the right Perl libraries when I deploy to it.

The problem is that CPAN is often run interactively and so module writers often assume the user will be present. That means that it stops in the middle and asks questions about skipping tests, etc. I searched for a while to figure out how to get a default answer to questions. It's not Capistrano's job and CPAN didn't seem to have a configuration option that worked. Turns out it's in MakeMaker.

MakeMaker is the Perl library that the CPAN modules use to automate the build process. There's an environment variable called PERL_MM_USE_DEFAULT that when true causes the MakeMaker prompt function to assume the default answer.

So, here's the task from the capfile I came up with.

task :load_bundle, roles => :local do
     run "cd /web/lib/perl/etc/kynetx-private-bundle; 
          sudo perl -MCPAN -e 
             '$ENV{PERL_MM_USE_DEFAULT}=1;
              install Bundle::kobj_modules'"
end

This works fine. Of course, you also need to make sure the account you're using for installs can sudo without a password or this will fail as well. Maybe there's a better way to do sudo inside Capistrano? I'd like to know about it.

Tags: kynetx sysadmin ruby perl

I'm going to be in the Bay Area next Thursday (May 15) with the day free and would love to get a few meetings with venture firms who might be interested in hearing about what we're doing with Kynetx. I don't know many VC's in the Bay area well, so if you wouldn't mind doing an intro to your favorite Bay Area VC, contact me.

Tags: kynetx

I’m starting a new business called Kynetx. As I go through some of the things I do, I’m planning to blog them. The whole series will be here. This is the thirteenth installment. You may find my efforts instructive. Or you may know a better way—if so, please let me know!

Sooner or later when you're starting a new venture, you need more money than you can raise with your credit card, a second mortgage, or selling your plane. Maybe you're tapped out personally and need to start taking a salary or maybe you need to hire help.

Whatever the reason, raising money is not very fun. Let's face it: you didn't start a business to raise money. You started a business because you saw a problem, figured out a solution, and you're passionate that your solution is going to change the world. Nothing is worse than weeks or months of meetings and legal work when you'd rather be creating.

I see three solutions to getting more money:

1. Get more clients: the is "customer funded" development
2. Extend your runway with friends and family or angel money
3. Go to a traditional venture capital firm

I'll admit upfront: I'm no expert. I've been around a lot of enterprises raising money, but never done it myself. I've always had the luxury having someone else do that part while I kept coding and solving client problems. Not this time.

Clearly option (1) is the best--if you can do it. But most startups reach a point where an infusion of cash can lead to more growth and better client service than you can reach with just customer money. If they last long enough, most end up at option (3) sooner or later.

So, the real question is when's the right time. Everything I know tells me that the right time is after you have traction, including at least a few good clients. You have to work to that point, either self funded or with a healthy dose of money from option (2). In parallel, you have to plan for option (1) in case the funding never happens. Pursuing options (1) and (3) in parallel--with all the energy you have--is all you can do. Option (2) is the safety value to extend runway.

At Kynetx we've focused our thinking with one simple question: what do we have to do to get five clients. Why five? No special reason. But thinking about the answer to that question--in everything from sales to infrastructure to customer service--gives special clarity in most prioritization questions. That's vital in a startup where there's always more to do than you have time for.

The good news is that if you work toward that goal, then you'll be aggressively pursuing option (1) while preparing in the best way for option (3). Heck, if you can establish that kind of traction, the VC's might start calling you and wouldn't that be a great position to be in?

Tags: startup kynetx funding

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the twelfth installment. You may find my efforts instructive. Or you may know a better way--if so, please let me know!

Every business makes a choice, often implicitly or by default, about what kind of relationship they want to have with their customers: will their customers be active participants or passively resentful? We all know business in the latter category. Cell phone companies spring to mind with almost no effort. You never hear someone say "Wow! I just love how my cell provider treats me!" Most of us are resentful of them.

On there other hand, there are businesses that have made their mark through the participation of their customers and users. Amazon and Google are both great examples. Neither would exist in their current form without the active participation of people--often without their even being aware--in providing the business with better information. Google relies on links. We don't link for Google's benefit, but whenever you do, you're helping Google make sense of the world. Amazon also relies on implicit participation, but also uses explicit requests of the customers to help make their service better: reviews, list, and wish lists are all examples.

With a nod to Doc Searls who taught me this, some of this has to do with language. At Kynetx, for example, we don't "target consumers." Rather we "serve customers." Even if the action you take is largely the same in either case, the words make it different. The former is impersonal, intrusive, and demeaning. The latter is uplifting and makes us think about our customers as people.

The word "consumer" brings to mind a stupid cow, carelessly ingesting whatever happens to be put in front of it. And who wants to be "targeted?" Not me. People are happy to be served, they want to be understood, they crave relationships.

This focus on nomenclature may seem silly and soft headed, but I think it's vital to developing the right culture in our business. Ultimately, I believe that culture can be a bigger differentiator than anything else you do.

I'm putting a jar at the office. Anyone who uses the word "consumer" or "target" when talking about our clients or their customers is putting a quarter in it. Maybe a buck. Along with a share of stock. Just kidding about the stock--but only because it's too hard to implement.

Tags: startup kynetx vrm

I'm starting a new business called Kynetx. As I go through some of the things I do, I'm planning to blog them. The whole series will be here. This is the eleventh installment. You may find my efforts instructive. Or you may know a better way--if so, please let me know!

When you talk to someone who's starting a business, ask them what they do--you can tell how far along they are by the answer you get. Every business has a story and it takes time to get that story straight. If they stumble around, say "it's hard to explain," or something like that, you're talking to someone with a brand new idea. This doesn't mean it's not a great idea--it just means that the story hasn't developed yet. I imagine that if you'd talked to Tim Berners--Lee in the first few weeks after he initially thought up the Web, you'd have walked away totally confused.

You may be at that stage. You might feel like you can't explain your idea easily. This isn't just because you can't explain it. It's also a sign that your idea is still developing. There are a few things you can do to get your ideas down and get your story straight.

The most important thing you can do is to keep talking. Talk about what you're doing with everyone you can. The more you explain your idea, the better chance you have of discovering good ways to describe it to others. You'll find a lot of people just nod and say "Wow! That sounds great." Valuable because you got some practice, but the most important conversations you'll have will be with people who challenge your idea and ask questions.

If someone challenges your idea and you feel completely discouraged and ready to give up, question your ability to pull this off. Being successful at a startup requires a good blend of passion combined with enough humility to take advice and good suggestions. You have to be able to take the hits, learn from them, change where necessary and prudent and yet keep the excitement about what you're doing. The most important and valuable conversations I've had are those where someone told me something I didn't want to hear.

I've been working on the idea behind Kynetx for almost a year. In that time, I've had meetings with dozens of people from VC's to friends who I trust. While the core of what Kynetx is hasn't changed, the way we talk about it, the ideas we have for where it can be used, and the way we have thought about funding it have changed dramatically. It's amazing to me how much our thinking has changed over that time. Just last week we had an epiphany on how we explain Kynetx to others that has us all very excited.

You might find it a little discouraging to think that you're going to be a year into something and still discovering new ways to think about it. Actually, I find that to be a lot of fun. The best part, for me, is the discovery.

Let me emphasize again, that the core idea is still largely the same. We've got over 10,000 lines of code at this point--I'd be pretty bummed if we were going to have to throw out our code every time we had a new idea. Once you start cutting code, you're committing yourself in important ways. On the other hand, putting that stake in the ground is also a great way to generate new ideas as well. Since I started writing code in October our progress has accelerated in amazing ways--and the way we thinking about what that code does has changed and become clearer.

I'm lucky in knowing a lot of people in the high-tech industry who graciously agree to talk to me and give me their advice. The secret is that many people like hearing about ideas and giving advice. Don't ask any of these folks to sign an NDA. It's a waste of time and paper and just puts people off. Be respectful of their time and don't be a pest.

Knowing your story will help a lot as you design products, raise money, and explain to your friends and neighbors what you're so passionate about.

Tags: startup kynetx funding

I caught a little flack in response to my post calling attention to Steve Yegge's recent essay "Portrait of a Noob." In particular Levi thought I was out of line for endorsing something so inflammatory:

"People who approach programming differently than I do are insecure n00bs"

That's a great attitude for a professor to endorse in a public forum. Steve's rant is nothing more than hot air attempting to justify his personal preferences at the expense of others. It makes you feel good because his preferences are the same as yours. The fact remains, however, that there's no real meat; it's just the evisceration of a straw man.

Dynamic languages and agile methodologies (whatever those vague terms mean today) are great, but so are modeling tools and static program analysis. Different tools for different jobs, etc. No one needs to be called insecure or a 'n00b'.

From Phil Windley's Technometria | Types as Comments
Referenced Tue Feb 12 2008 08:37:20 GMT-0700 (MST)

We're not talking about your taste in clothes or furniture. We're talking about a way of working that can be right or wrong. This isn't simply a matter of personal preference. We like to wave it all away with "depends on the situation" kinds of comments, but that's a cop out for the most part.

I believe that there are styles of programing and of organizing large programming projects that are better than others--not simply a matter of style. Businesses, particularly small ones can gain significant advantages by choosing one language or style over another. This is not simple a matter of taste.

That said, circumstances often dictate choices. I'm in the middle of a project right now, for example, where the right architectural choice was to use the Apache Web server as the application server. That requires writing Apache modules and that means C or Perl. No other languages offer complete solutions in that space. (I chose Perl.)

A few specific comments on the Yegge article

1. You need to read Yegge's entire article--or at least skim it--not just read the pull quote I used which is the most inflammatory part of an otherwise long article. Maybe you did...
2. Yegge makes some interesting and useful points about static analysis and it's sometimes dubious value (yes, I'm now stating my opinion).
3. Static type analysis is taken by so many people as "the right way" and dynamic type analysis as "the lazy way." I see part of my role as an educator to challenge that assumption.
4. No matter what kind of type system you use, you need to think about types. That's part of programming. What level of tool support is appropriate depends in large part on what the programmer is comfortable with.
5. That said, for large programming projects (which I am also leery of), static modeling allows the automation of large parts of the refactoring code and other tasks. More power to them.

So, in summary, I do think that Yegge's post (and many of his others) is delightful read that challenges many preconceived notions about types and forces exactly the discussion we're having now. "It worked!", I think Steve would say.

Tags: programming programming+languages kynetx

I’m starting a new business called Kynetx. As I go through some of the things I do, I’m planning to blog them. The whole series will be here. This is the ninth installment. You may find my efforts instructive. Or you may know a better way—if so, please let me know!

Today I got a cold call at my office that essentially went like this: "I read you blog and would like to tell you my business idea and get your opinion on it." I said that I had a few minutes to listen and so the guy launched into a description.

Actually, that's not quite accurate. The first thing he did was say that he didn't think that I'd want to sign an NDA and that he wasn't inclined to ask for one. Good thing. When people ask me to sign an NDA, I always say "no." Like I'm going to keep track of dozens, even hundreds of NDAs and who said what when. There's not enough time in the world. So, he was right and scored points with me right off.

His idea was an interesting one, if I'm being honest and I told him a few places where it intersected with some other trends I saw and thought were important. He then asked "how do I get started?"

My first instinct was to tell him to build a demo, but then I remembered something even more fundamental than that.

I asked what his background was. "Media," he says. Uh oh. Not that there's anything with media types, but how's he going to build a demo. I told him he needed a CTO and why.

"Can you recommend some people who might be my CTO?" he asked. Nope. I can't. I asked him if he thought he'd see much action from a list of folks I recommended he call and asked for money. He understood that this wasn't the best way to raise money; when you're asking people for an investment, they need to trust you--especially at this stage. That's why friends and family are the most likely investors in your early stage start up (if you don't have a plane to sell).

I explained that anyone he asks to work with him on his idea will be making a huge investment of time and commitment--especially on the margin. They need to trust you. They need to feel your passion for the idea. They need to believe that spending the next three to six years with you 14 hours a day will be fun.

This is why successful founding teams have usually worked with each in the past, gone to school with each other, or are otherwise friends. That's not always the case, but it often is. Collaborations among strangers just aren't likely to happen without a pretty significant bonding first.

Every founding team needs to have a technical member. I've seen more ideas killed by ignorance of this fact than I care to remember. I told my newfound friend to look to his friends, co-workers, and old classmates to see if he couldn't find someone to be the technical yang to his marketing yin.

Once he's got the technical founder, he can get a demo working--that will make people believe it's real. After that, you're on your way.

Tags: kynetx startup

I’m starting a new business called Kynetx. As I go through some of the things I do, I’m planning to blog them. The whole series will be here. This is the eighth installment. You may find my efforts instructive. Or you may know a better way—if so, please let me know!

I have a friend who has a way with words and has started his share of high-tech businesses. I asked him his advice on getting started and Dan said "Welp, you gotta get yourself a rude dog demo!"

What he meant is that you can't just start a business with an idea. You've got to have something to show people. The demo doesn't have to be too pretty (that's the "rude dog" part) but it does have to demonstrate your idea and your ability to execute it.

For some applications, you might get by with a PowerPoint mock-up of your UI, but I believe that to raise serious money for a high-tech company, you need to have working code. Nothing else will do.

Paul Alstrom, a friend of mine, who's also one of the managing partners of vSpring Capital talks about nailing and scaling. VCs don't won't give you money to "nail" your idea (although angels might). VCs want to put money into proven ideas that need capital to scale. The more you have that shows your idea is solid, the closer you are to securing capital.

Working code has heft. It turns ideas into action. Code makes ideas come alive. Try telling someone about your idea. Then show them a demo. Watch what happens to their eyes. I love how the lights go on when someone can see something work.

Working code also instills discipline in the founding team and forces you to "get real." Once you sit down, start cutting code, and making things work, you suddenly start finding holes in your original idea and ways to improve it even more.

This was written about open source projects, but I think it applies to start-ups as well:

The best way to start an open source project is with code. Working code. Hack away at home on weekends, maybe get a couple of friends to help you out, and don't go public until you have something to show people that does something interesting, and that other people can use to build more interesting stuff on top of. You need this for a bunch of different reasons: it establishes the original contributor's bona fides in the open-source meritocracy, it shortcuts all sorts of damaging debates about coding styles and architecture that can stop a project before it starts, and so on.

Most importantly, though: working code attracts people who want to code. Design documents attract people who want to talk about coding. I've seen what happens on projects that start with no code and a commitment to produce a design. Some of the procession of UML diagrams were really well put together, but that's about the extent of it.

From The Fishbowl: Finding Discord in Harmony
Referenced Thu Jan 31 2008 21:52:29 GMT-0700 (MST)

Working code gives the rest of the people on your team something to use for leverage in their thinking. Seeing things come to life is a sure way to spark ideas.

What if you don't know how to code? Then you need a founder who does. See my earlier post on why you need a CTO. If you don't have founders who can cut code, you have no business starting a high-tech business. That may sound harsh, but I believe it's true.

The hard part of producing "working code" is defining "working." How close to production does it need to be? For a "rude dog demo" not very. It's probably more important that the UI be pretty than the guts be complete. It may be running on your laptop and need a month of work to get ready for production. That's OK for the demo part.

Eventually you have two choices: throw it away and build the real thing or morph what you've got into the real thing. For Kynetx, I've designed the engine so that all the pieces are there, including a plug-in architecture, and the difference between the production version and the demo version is filling in the holes rather than rewriting what's already done. That took more work, but we wanted to progress from "rude dog demo" to "alpha customer ready" pretty quickly.

I'm here to tell you: nothing made Kynetx seem more real than having code we could call our own. Ideas are not assets, but code is. And assets are ultimately what you leverage to make any high-tech business work.

Tags: startup kynetx programming cto

I didn't grow up in an era where testing was as well though of as it is now. When I learned to program, you ran a few tests after the fact and threw it over the wall to the QA department. Not very politically correct in today's software engineering world.

As a result, I understand the value of testing and support the idea intellectually, but I don't have the discipline.

Recently I was faced with the problem of building code that translated a domain specific language (KRL) back and forth between three different representations:

The textual representation is what people read and write. The JSON representation is what machines (other programs) will read and write. The abstract syntax (a Perl data structure) is what will be interpreted.

I quickly realized that getting this right--and convincing myself of that--would require something more than some ad hoc testing. So, I decided to mechanize.

I knew that Perl had testing facilities--anyone who's built a library has seen that. So, I had to figure out how they worked. I found these resources handy:

• Test::Tutorial - the basics
• An Introduction to Testing - some more advanced things, like Test::More that helped specifically with what I needed to do
• Automated testing with Perl
• Large Project Testing with Perl (PDF) - for the project stuff

Here's what I did:

• Wrote a collection of test programs in KRL. These can be expanded as needed, as I think of things to test.
• Built a test script for running each of those programs through a round trip through the parser and pretty printer. If the text matches (modulo whitespace) it's OK
• Built another test script for running each of these programs through a round trip all the way to the JSON representation and back.

I also added test scripts for each module in the system as well as a few scripts to check for developer issues--thanks to petdance (PDF).

I used the petdance smoke script to run all the tests in Test::Harness. I modified the petdance rule.t script to check for Perl programming best practices like always having use strict at the front of every Perl program in the collection.

I also wrote a use.t script that goes through the entire collection of Perl modules and programs, gathers a unique list of every library they use, and then checks for availability on the current machine.

Now, I can flesh out this framework with other tests as they come to me or I run into problems. It feels good.

Tags: testing programming perl kynetx