Utah Open Source Planet

I’m in the process of making PLT Scheme multiprocessor/multi os thread aware.
Everyone knows that globals are the arch enemy of multiprocessor code. Needless to say, PLT Scheme has plenty of globals from is origin as a single processor/single os thread app.

One such global is called kernel_modidx ( the plt scheme kernel module index )
There is some init code that sets these globals.


REGISTER_SO(kernel_symbol);
REGISTER_SO(kernel_modname);
REGISTER_SO(kernel_modidx);
kernel_symbol = scheme_intern_symbol("#%kernel");
kernel_modname = scheme_intern_resolved_module_path(kernel_symbol);
kernel_modidx = scheme_make_modidx(scheme_make_pair(quote_symbol, scheme_make_pair(kernel_symbol, scheme_null)), scheme_false, kernel_modname);

As you can see kernel_modname includes the kernel_symbol, kernel_modidx includes the kernel_symbol and the kernel_modname. This is very important for things to work later.

You would think that calling this init code twice would work the same way the second time as it did the first. It took me two days to hunt down why it doesn’t.

Taking a look at scheme_make_modidx you will see the special case that if kernel_modidx is defined that scheme_make_modidx will not make a new modidx, but will return the old one.

Its not that obfuscated, I just missed it the first thousand times I stepped through the code.


Scheme_Object *scheme_make_modidx(Scheme_Object *path,
Scheme_Object *base_modidx,
Scheme_Object *resolved)
{
Scheme_Modidx *modidx;

if (SCHEME_MODNAMEP(path))
return path;

if (SCHEME_PAIRP(path)
&& SAME_OBJ(SCHEME_CAR(path), quote_symbol)
&& SCHEME_PAIRP(SCHEME_CDR(path))
&& SAME_OBJ(SCHEME_CADR(path), kernel_symbol)
&& SCHEME_NULLP(SCHEME_CDR(SCHEME_CDR(path)))
&& kernel_modidx)
return kernel_modidx;


I live close to the University of Utah.

Life Flight and AirMed helicopters pass over my apartment complex all day long ferrying people to and from the several hospitals that are part of the University Medical Complex. They often pass low enough that they literally shake the apartment. I can feel the shaking, especially when I’m lying down in bed. It must be the air pressure generated by the rotors.

There is also a large construction project next door that has a bulldozer that constantly pushes dirt around. When the dozer is hard at work the apartment building likewise shakes. I’ve experienced real earthquakes in California and Chile. My current environment feels like one never ending earthquake.

Organick Lecture 2008
High Performance Programs and Programmers: A Personal Perspective
Fran Allen

IBM Fellow Emerita
IBM T. J. Watson Research Center

University of Utah School of Computing
Wednesday, February 20, 2008 7:30 p.m.
Turing Award Winner

Fran calls supercomputers one of the miracles of our time.

“In the beginning there was Fortran.” Jim Gray

Fran’s first assignment was to teach the scientist of IBM Research Fortran which had just been released.
Up till then the scientists had written their code by hand in assembly.

John Backus set goals, stuck with them, and achieved them.

Fortran had two objectives:
Increase the productivity of programmers.
Be almost as efficient and hand crafted code.

Fran’s next project was the IBM Stretch 1956-1961
Fred Brooks
The stretch was announced to be 100x faster than any existing machine
Memory Access Time was the main performance limitation
The stretch had up to 6 overlapping storage references at one time.
Had up to 11 instructions in flight at a time in the processor.
The stretch was built for solving engineering and scientific problems

HARVEST
built for the NSA, delivered in 1962
Used for code breaking.
The HARVEST was a streaming data computation model.
It had eight instructions.
IO, memory, and computational where all balanced and synced to a clock.
Only machine that maintained this balanced.
The machines language ALPHA was designed for the NSA problem and the machine.

The stretch, when deliver to Los Alamos, missed its performance target by 50%.
The stretch was an attempt to do what was beyond the capabilities of the people and the technology of the time.
The stretch had huge influence on IBM future systems, particularly the IBM 360.
We don’t take risks developing new systems like we use too. The strech was taking a risk and eventually winning because of it.

The IBM 360 unified the business and scientific product roles.
This was a goal, particularly a unified instruction set, was set by Fred Brooks.
There were competitions of competing ideas to find the best solutions.

John Cocke was a engineer who “never wrote a “paper” or “gave a talk”".
John Cocke build the most difficult part of the machine, the look ahead.
John Cocke was in charge of the ACS project that was eventually cancelled in 1968.

The ACS compiler did many pioneering things in the area of compilers.

John Cocke decided to stop trying to create the worlds fastest computer, (the ACS machine), and decided to create the best performance/cost machine which resulted in the Power PC.

Ideas in our field come from problems.

Good an useful results come from problems that result from building something.

It takes twenty years from the appearance of a problem to it being totally solved in a product.

The new thing will be simpler, slower, cooler multicore processors.

The real challenge is to organize tasks to execute in parallel.
How do we use these multicores and get performance out of them?
Fran says we are not ready to do parallel yet. Maybe after 20 years we will be ready.

Everyone will have multicores, not just the high end scientific community.

How these cores will be used is a big question for the software and application people.
We now have a parallel hammer, how will we use it?

Parallelism
“The biggest problem Computer Science has ever faced.” John Hennessy
“The best opportunity Computer Science has to improve user productivity, application performance, and system integrity.” Fran Allen

Coming back to Elliot Organick
The things that succeed best are those that have complete clarity about what the problem is and what the solutions are.
Writing things down and engaging communication is the key to success.

Questions:
What have we learned from the transputer.
The CSP work done for the transputer I think is interesting.

Is the problem in conceptualization or in finding a language to describe our conceptualizations?
Tony Hoard wrote a paper called “Toward the understanding of parallel computers”.
John Nash had some ideas about a parallel machine while working for RAND.

Could you expand on your ideas system integrity?
Language integrity could be a second order effect of better languages.

Users still have to know about the raw details of the machine to get great performance.

Parallelism will increase and the hardware will get easier.
The complexity of hardware will move to parallelism in software.

We have a lot of work in front us and it will be interesting to see

You mentioned that you would like to see caches disappear.
Scratchpad memories may be a better solution than caches.
But being responsible for the memory hierarchy whether you want to or not can be as much a curse as a blessing.

Yes, I’m a alpha geek and and early adopter.

Hardy Heron is working great, but the Firefox 3.0 packaging is still undergoing some revision.
I use three applications extensively, Firefox, Thunderbird, and Konsole.
I can’t deal with the extra large fonts that display by default in Firefox 3.0 in Hardy Heron.
So I used this set of instructions to temporarily downgrade the firefox package.

http://www.nowhere.dk/archives/2008/02/14/firefox_3_0_is_now_the_default_browser_in_ubuntu/index.php

I run Linux on my laptop most of the time.
I like to run the latest version, but I don’t want to compile it myself and try to package it up in a deb every time a new kernel comes out.

I just found KernelCheck.

KernelCheck is a project that is designed to easily build the latest kernel for your distribution

The latest BETA even automatically builds in support for the latest NVIDIA proprietary driver.

So far its simple and seems to be working flawlessly.

I’ll let you know how it goes.

On Linked-In there are “those people” that mark their connections as private, so that the only connections you can see are people you are already linked-in with.

Linked-In is a community of sharing. Those who mark their connections as private are saying loud and clear, I’m attempting to freeload off others’ willingness to share openly.

It’s no surprise that the common private makers are head hunters, technical recruiters, or other individuals attempting to monetize their network. That being said, I have one technical recruiter in my Linked-In connections that openly shares their connections. Guess who I recommend and go to first when I need to work with a technical recruiter.

I’ve been seriously thinking about dropping people from my Linked-In profile that mark their connections private. So listen up Linked-In, I have a feature request: I want my connections to appear private to other users who mark their connections private. But wait a second, that feature doesn’t follow the golden rule though. Do unto others as you would have others do unto you. On second thought Linked-In, just drop private connections all together. They don’t help build a community.

So, If you find someone who looks interesting in my linked-in connections, that has their connections marked private, send me an email. I probably have some else I would recommend first.

I’ve been working on a new web application. Initially I started out with Ruby on Rails and HTML/Ajax. I also tried out Jifty, the latest hip Perl5 web framework from Best Practical.

I love puzzles. HTML/CSS/Ajax are always a puzzle. I’ve spend significant time debugging and solving HTML/CSS/Ajax problems. But when I am trying to get something done, and done quickly, the endless puzzles of HTML/CSS/Ajax just serve to frustrate me and leave me with a bitter taste in my mouth.

Complaints
1) Rendering and general application behavior is still inconsistent across browsers vendors.
2) Graphical layout is a hacked up mess compared to the gui toolkits such as SWING, .NET Winforms, or even XUL.
3) (Sarcastically) Wireshark (Ethereal) is my number one debugging tool for the “Browser Platform”.

Well I’ve spent the last couple of days sitting down and working with Flex and ActionScript 3.0. What a welcome relief. I’m starting to get things done and making progress.
Flex isn’t perfect, but Adobe is in process of making the Flex SDK open source.

I’d really prefer to be using the new Silverlight platform from Microsoft, but unfortunately is proprietary, doesn’t have a linux port, it doesn’t currently ship with any gui controls, and its barely been released as an alpha preview.

Bruce Eckel seems to be having similar frustrations.

Update:
Now, I’m not saying that HTML/CSS/Ajax doesn’t have its place. Flex isn’t indexable and searchable by search engines, but if you are building a web application that is username/password protected anyway why waste your time.

I’m using ruby on rails, but this video gives you an idea of what you can do:
Adobe - Developer Center : Video tutorial: Creating a Flex application using the TurboGears framework:

John Taber has been trying to use SVG in his Ruby on Rails app.
Unfortunately, Webrick and Rails have been uncooperative.

From railties/lib/webrick_server.rb we see that rails attempts to serve requests from the public directory before dispatching to controllers.

def service(req, res) #:nodoc:
    unless handle_file(req, res)
      begin
        REQUEST_MUTEX.lock unless ActionController::Base.allow_concurrency
        unless handle_dispatch(req, res)
          raise WEBrick::HTTPStatus::NotFound, "`#{req.path}' not found."
        end
      ensure
        unless ActionController::Base.allow_concurrency
          REQUEST_MUTEX.unlock if REQUEST_MUTEX.locked?
        end
      end
    end
end

If you put a svg file in the public directory of a rails app, webrick serves it up, but with the wrong mime type.

$ wget localhost:3000/rect1.svg
--15:54:54--  http://localhost:3000/rect1.svg
           => `rect1.svg.1'
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 333 [application/octet-stream]

100%[====================================>] 333           --.--K/s

15:54:54 (12.92 MB/s) - `rect1.svg.1' saved [333/333]

Firefox displays a friendly description based on file extension, but doesn’t display the svg because the mime type is wrong.

So add a svg mime type entry to webrick

--- /usr/lib/ruby/1.8/webrick/httputils.rb.orig 2007-04-21 16:00:42.000000000 -0600
+++ /usr/lib/ruby/1.8/webrick/httputils.rb      2007-04-21 16:01:03.000000000 -0600
@@ -86,6 +86,7 @@
       "rtf"   => "application/rtf",
       "sgm"   => "text/sgml",
       "sgml"  => "text/sgml",
+      "svg"   => "image/svg+xml",
       "tif"   => "image/tiff",
       "tiff"  => "image/tiff",
       "txt"   => "text/plain",

And presto

$ wget localhost:3000/rect1.svg
--16:17:22--  http://localhost:3000/rect1.svg
           => `rect1.svg.2'
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 333 [image/svg+xml]

100%[======================================>] 333           --.--K/s

16:17:22 (22.73 MB/s) - `rect1.svg.2' saved [333/333]

SVG content delivered through a controller works just fine, you just have to set the content disposition to inline.

def index
  data = <<END_D
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

<svg width="100%" height="100%" version="1.1"
xmlns="http://www.w3.org/2000/svg">

<rect width="300" height="100"
style="fill:rgb(0,0,255);stroke-width:1;
stroke:rgb(0,0,0)"/>

</svg>
END_D
send_data data, :type => 'image/svg+xml', :disposition => 'inline'
end

Now a DSL for SVG creation would be cool, but that is a completely different project.

I just read on Lambda the Ultimate that John Backus passed away this past Saturday.

---

In his speech accepting the Turing Award, John Backus admonishes developers and computer scientists to look towards functional programing. He states that the composability and mathematical rigor of functional programming empowers programmers to scale and best describe the ever increasing complexity of problems. Interestingly, Backus describes the von Neumann architecture and its associate imperative programming language as obese. Backus continues saying that these obesities encourage micro-managed word at a time computing. Instead of moving data between named locations, we should focus on the larger conceptual units of the problem says Backus.

I see this debilitating behavior every day, both in developers and managers. Most IS/IT managers get caught up in the word at a time or task at a time work flow that cripples teams and organizations. Senior developers and managers must be instructors and mentors in the enterprise. They must be experience practitioners of abstraction and modeling. Their first and most important priority must be to impart the skills of scale, abstraction, modeling, and reuse to those they mentor.

Ruby Binary Lottery - Mike

WAX (Web Applications X) - Dan Kirkpatrick eparklabs.com

Managing SSH keys with Capistrano or How Jamis Buck made my life easier./- Jay ???

Goldberg Ruby on Rails Generator - Coby Randquist

CruiseControl.rb - ThoughtWorks Continuous Integration Tool written in Ruby.

Why would you want to use JRuby on Rails - Charles Nutter
RailsIntegration part of JRuby Extras Project on RubyForge

LogWatchR - Pat Eyler aggregates logging for 3,000 boxes to a central machine for operation notification.
Initially it handled 600 log entries per second. Right now it notifies via email and a log file. Looking to support nagios, jabber, etc.
Now supports 2250 log entries per second.

Background

Hey! Why not build everything yourself?
That’s insane. No, really. It’s a mouse hole.
Why not use a Ruby app?
Not a good selection of apps.
Why not use what works best?
P* Languages are not the work of Satan.
Big Motivation: “Hey, I wonder if …” “Wow. It never made that sound before. Neat!”

Use case: Trac Project tracking tool

But way too much mousing.
Hey, I wounder if you could use Trac from the command line?
Sure: Use Tracula. I wrote it.
Tracula uses Hpricot & Mechanize to prentend to be a browser.
tracula.neurogami.com

Brittleness gets introduced when going against web pages.
Version updates break things. Trac Plugins break things.
Yes it’s fragile, but it’s easy and it’s easy to get started.

The Django book has a cool commenting system.
Go look at beginningruby.com

Techniques

• Screen scraping
• DOM munging
• Proxies, proxies, proxies

Observations

Most sites offering a real Web API
Proxies offer buffering and separation of concerns and can help avoid tool lock-in.
Robust exception handling.

RubyStuff.com - front end for CafePress
CloudPanel - EC2 admin tool

www.neurogami.com
www.jamesbritt.com

Jeff heard Dave Thomas talk at Java - No Fluff Just Stuff and became converted to Ruby. Beyond lightweight frameworks like Spring, Ruby was the answer being preached at NFJS. The famous 10x programmer efficiency increase with Ruby proved true.

WYSIWYG for web development. Rails Erb looks to much like the old ASP/JSP stuff. View stuff was spread across too many files in an extreme attempt to by DRY(Don’t Repeat Yourself).

Goals

• WYSUWYG (x)html
• Keep it simple DRY Somple Syntax iwht Ruby Flavor
• Designed for RoR
• Utilize Rails layouts, partials, and helpers.
• Reduce complexity, no extra view (presentation) objects or hashes
• Production ready scaffold templates or work from html prototype..
• Redcude number of files. Preview in browser

MasterView has a richer CRUD scaffolding than plain old Rails.

See also

• Amrita2
• Kwartz
• PageTemplate
• IOWA, Liquid
• Components, WebObjects JSF
• PHP/Zope, Tal, Tapestry

Problems with causable Agents

1. Impotence
2. Theoretical Ignorance
3. Pratical Ignorance

• How to handle uncertainty?
• Enter Probability Theory
• Probability provides a way to of summarizing the uncertainty that comes form our laziness and ignorance

Basic Probability Theory

• Propositions always between 0 or 1
• Random variables, discrete or continuous
• Sum of Probabilities, product rule, chain rule, etc
• Mutually exclusive events, Marginalization, Full Joint Probability
• Bayesian Probability
• Prior Probability or Marginal Probability, Likelihood, Posterior Probability or Conditional Probability

Bayesian Networks

Inference Methods

1. Exact (potentially exponential)
2. Approximate (less Accurate) - stochastic or Monte Carlo algorithms

Simple Bayesian Networks with Ruby

SBN - gem install sbn

• 2003 - Class project
• 2005 - C++ version
• 2007 Ruby version

“Premature optimization ois the root of all evil.” Donald Knuth, paraphrasing C.A.R. Hoare

• 86 Slides
• Rolling the Dice with Ruport
• Reporting Sucks
• #mwrc irc.freenode.net
• Ruby is a community, we should love Ruby and need Ruby
• Are we dealing with ideologically problems, Firefox logo vs IceWeasel Logo
• Good software is produced by involving a community
• Lets analyze Ruport
• Ideals:
• Ruport is GPL. License Choices do Matter. It is important to poll the communities opinions of licenses.
• Licenses are hard to change, Ideology isn’t
• Don’t write your own license. It’s a matter of comprise.
• Free Software is a gift
• BSD/MIT favors individuals.
• GPL favors communities.
• RubyLang is dual licensed
• Go with RubyLangs license and you will probably be ok with the free software crowd
• Communities are warehouses of ideas
• You can’t please everyone in four weeks, working part-time
• You can only really work passionately on your problems.
• Learn when to say no to feature bloat or -O nofun.
• Sometimes Less is More. Tens vs Thousands. Much easier to read tens of emails vs thousands of emails per week.
• Mailing List != Bug Tracker
• GForge is nice. But is it what you need? Ruport needed SVN/Trac
• Trac blurs the lines between developer/contributor/user
• svk + cron allowed use to mirror our Trac SVN repo to the RubyForge SVN repo.

• Tools Matter:
  

• Trac has improved participation in Ruport
• Friction Effects Contribution
• Technical decisions and input shouldn’t occur on RubyTalk
• Every patch, even if unapplied is valuable. Don’t dismiss them. Examples and tests can come from patches.

• What’s Relevent
  

• Create a roadmap.
• Telling yourself what your software is, helps you build it.
• Design Goals need scoping as well as features.
• Use cases can drive development. We invite people

• No Cruise-Control

• Don’t introduce things that are only meant for you, the developer, or one single user
• Move non-essential features or customizations for single users out to plugins.

• Unique Poject Identity is good

• Be Bold about your decisions

• Take risks
• Come up with crazy ideas
• Start finding your community early.

• One Last Thing - One Last Good Idea

• Ruport 1.0 2007.05.15

Questions/Comments:

• Look around us at other projects and how they use their community building skills.
• Make it easy
• Meta-documentation that describes basic processes such as howto submit but reports, patches, etc
• Make meta-documentation stupidly easy to find
• We need to create a collection of best practices for managing ruby projects.
• Go read the book “Producing OpenSource Software“. It’s free online. Go Read it.

RubyQueue is an embarrassing parallel work queue system that simply uses NFS directories as a shared queue.

• NFS lock sharing isn’t defined to be fair. Ruby Queue uses a exponential back off algorithm to try to make NFS locking more fair.
• NFS uses arbitrary ports, because of portmap, which causes problems for firewalls. It is possible, however, to pin those ports which makes firewall traversal possible.