Utah Open Source Planet

Imagine you have a repository. At time t₀ someone copies the code from the repository, starts another repository from there, and commences development. You also continue development. At time t you want to pull in the changes the other guy has made.

One approach could be you simply merge other guys code into yours with one big commit calculated from time t₀. Now suppose you want to keep other guy’s history from t₀ on, or even worse, you want to pull only some changes and you want to push some of your changes to other guy.

This is a true story. It all started with CVS. A webapp was developed, cloned and branched. Now there are 4 of them, mostly the same, but subtly different, and each with it’s own CVS repo. Each one at some point in time is identical to another, but the history behind those points in time are different, thanks to copy and commit cloning.

So after converting the CVS to Darcs, I came up with a way of tricking darcs into treating time t₀ in each repository as the same t₀, thus enabling cross-pollination, and here I describe that method.

First you find t₀, and tag liberally. By liberally I mean twice. I’ll explain why later. Of course you put the same two tags on both repos, both at the same point in time. Then you do a darcs opt --checkpoint on each so you can do a partial get on them.

Now you have to decide which repo you’re going to keep, and which one you are bringing in. You probably want to keep the one with past history, and bring in the one that did copy and commit cloning. Do a darcs get --partial on each repo, keeping track of which is what. I like to use a for the destination, and b for the bad repo.

Now go into b/_darcs and we’ll play around with these guts to make it think it’s a, but we want it to have it’s future patches first. So we take a snapshot before pulling the other stuff. I just made a temp directory and did cp -r inv* patches temp, these are the three things in the _darcs directory we care about. Here’s where the liberal tagging comes in, the partial at the later tag only depends on the first tag, so that makes your inventory and patches much simpler. Now you’re ready to pull in all the subsequent patches from the cloned branch.

Once that’s done, you go back into b/_darcs, this time you take anything in temp/patches and temp/inventories and remove the corresponding files from patches and inventories. Then you replace those files with the files from a/_darcs/{inventories,patches}. These two steps I accomplish like so:

ls temp/inventories | while read i; do rm inventories/$i; done

ls temp/patches | while read i; do rm patches/$i; done

for f in inventories patches; do cp ../../a/_darcs/$f/* $f; done

One last step remains. Now you edit inventory and replace that very first patch number, the one that’s a tag, with the one that’s in the inventory file from a. Now b effectively has the exact same past history as a, and has all it’s future patches, and now you can share between your repo and other guy’s repo freely.

One might think this is an uncommon situation, but I can’t help but think it’s more common than I think… Hopefully this guide will save others the headache of trying to wrap their minds around the darcs inner guts, as well as save me some headache should I ever encounter this again (which I’m pretty sure I will, in only a few weeks!).

this script checks to see if a program is running and starts it if it is not. if a certain time has passed the script will kill the running program

#!/bin/bash
 
#location of script to monitor
PROGRAM=‘myscript.php’
COUNTER=0
 
#!/bin/bash
 
date2stamp () {
    date –utc –date "$1" +%s
}
 
# convert a date into a UNIX timestamp
# time afterwhich script should not run
finishtime=$(date2stamp "Thu Nov  1 15:22:00 MDT 2007")
#echo $finishtime
 
while [ 1 ];
do
 
nowtime=`date –utc +%s`
#echo $nowtime
diffTime=$((finishtime-nowtime))
if ((diffTime > 0));
then
echo "still time to go"
else
echo "TIME UP!: checking if running:"
PROG_CHECK=`ps aux|grep $PROGRAM|wc -l`
if [ $PROG_CHECK -gt 1 ];
then
echo "running after time up"
PROG_ID=`ps uax|grep $PROGRAM |head -n 1|awk ‘{print $2}’`
#echo "KILLING"
#kill -9 $PROG_ID
else
echo "not running after time up"
fi
exit 65
fi
 
 
PROG_CHECK=`ps aux|grep $PROGRAM|wc -l`
if [ $PROG_CHECK -gt 1 ];
then
echo "running"
else
let COUNTER=COUNTER+1
echo "not running: starting: new log file: $COUNTER"
/usr/local/bin/php /root/$PROGRAM &
fi
 
#time in seconds to sleep
sleep 5
 
done

forward a single IP

• ssh into a server
• right click on toolbar, change settings
• ssh->tunnels
  • source: 80
  • destination: someip:80
  • CLICK Add
  • click Apply
• in a browser: goto localhost and you’ll see the contents of someip

forward all addresses

• ssh into a server
• right click on putty toolbar, change settings
• ssh->tunnels
  • source: 5432
  • select (*) Dynamic
  • Click Add
  • Click Apply
• in firefox:
  • tools->options, advanced, network
  • (*) manual proxy configuration
  • SOCKS host: 127.0.0.1, port: 5432

To make it more secure, set the user shell on the ssh server to /bin/false and in ssh settings click [x] don’t start a shell or command at all
don’t forget to change the network settings back when you want to surf normally. Also once the ssh session closes your internet access will not work

I recently stumbled upon Javascript/Canvas Graph library which allows you to create cool network graphs with nodes and interconnections. Now that I had a graphic library, I needed something with lots of connections to graph. That’s when I thought of mapping health IT blog sites. Now all I needed was a spider…

Spiders, robots, crawlers all work on the same principle: start with a seed url(s), request the urls and then parse out the resultant html for all link tags which are then pushed into the queue for crawling.

time for some code:

1. <?
2. $tldList=file("tlds.txt");
3. $masterList=file("masterList.txt");
4. $acceptableNodes=array();
5. foreach($masterList as $node)
6. {
7. $node="http://".trim($node);
8. $acceptableNodes[]=parse_url_domain ($node);
9. }
10. //print_r($acceptableNodes);
11. $connections=array();
12. $nodeCounter=array();
13. foreach ($masterList as $webpage)
14. {
15. $url ="http://".trim($webpage);
16. $strippedWebpage=parse_url_domain ($url);
17. $input = file_get_contents($url); //or die("Could not access file: $url");
18. if (!$input) continue;
19. $regexp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
20. $URLs=array();
21. if(preg_match_all("/$regexp/siU", $input, $matches, PREG_SET_ORDER))
22. {
23. foreach($matches as $match)
24. {
25. //echo($match[2]."\t");
26. $currentURL=parse_url_domain ($match[2]);
27. //echo($currentURL);
28. //echo("\n");
29. if ($currentURL!=$webpage && !in_array($currentURL,$URLs)&&check_domain($currentURL)&&check_tld($currentURL))
30. {
31. $URLs[]=$currentURL;
32. if (in_array($currentURL,$acceptableNodes)&& ($strippedWebpage!=$currentURL))
33. {
34. $connections[]="$strippedWebpage -> $currentURL";
35. $nodeCounter[$currentURL]++;
36. }
37. }
38.  
39. }
40. }
41. echo("$webpage : \n");
42. print_r($URLs);
43. }
44. echo("connections: ");
45. print_r($connections);
46. echo("nodeCounter: ");
47. print_r($nodeCounter);
48. echo("javascript edges: ");
49. $multiplier=3;
50. foreach($connections as $connection)
51. {
52. list($from,$to)=explode(" -> ",$connection);
53. $fromCtr=3*$nodeCounter[$from];
54. $toCtr=3*$nodeCounter[$to];
55. echo("g.addEdge($('".$from."'), $('".$to."'),".$fromCtr.",".$toCtr.");\n");
56. }
57. function parse_url_domain ($url)
58. {
59.  
60. $raw_url= parse_url($url);
61. if ($raw_url['host'] == '')
62. {
63. $raw_url['host'] = $raw_url['path'];
64. }
65. $domain_only[1] = $raw_url['host'];
66. return strtolower($domain_only[1]);
67. }
68. function check_domain ($url)
69. {
70. if (!ereg("^.*\..*$", $url))
71. {
72. return false;
73. }
74. $local_array = explode(".", $url);
75. for ($i = 0; $i < sizeof($local_array); $i++)
76. {
77. if (!ereg("^(([A-Za-z0-9!#$%&'*+/=?^_`{|}~-][A-Za-z0-9!#$%&'*+/=?^_`{|}~\.-]{0,63})|(\"[^(\\|\")]{0,62}\"))$", $local_array[$i]))
78. {
79. return false;
80. }
81. }
82. return true;
83. }
84. function check_tld($url)
85. {
86. global $tldList;
87. $parts=explode(".",$url);
88. $lastpart=trim($parts[count($parts)-1]);
89. foreach($tldList as $item)
90. {
91. if (trim($item)==$lastpart)
92. return true;
93. }
94. return false;
95.  
96. }
97. ?>
98. Download this code: linkSpider.phps

and no, I’m not talking about the Utah Summer Games Arm Wrestling competition.

I’m talking about Python, the programming language. And to encourage adoption of that language some clever people created a little Python programming game (back in 2005.) You’re presented with a riddle of sorts and you’re supposed to code up a python script to solve it. Only, I don’t like people telling me what to do, so I intentionally did NOT code up python to solve them. I used PHP mostly. Here are my notes:

QUESTION 1: what’s 2^38?
ANSWER: I asked google: http://www.google.com/search?hl=en&q=2%5E38&btnG=Google+Search
and then I used the answer to get to the next step: 274877906944.html

QUESTION 2: given: K->M, O->Q, E->G, decode: g fmnc wms bgblr rpylqjyrc gr zw fylb. rfyrq ufyr amknsrcpq ypc dmp. bmgle gr gl zw fylb gq glcddgagclr ylb rfyr’q ufw rfgq rcvr gq qm jmle. sqgle qrpgle.kyicrpylq() gq pcamkkclbcb. lmu ynnjw ml rfc spj.
ANSWER: I used an online cryptogram solver (though i could have written a simple script to shift up two letters…)
“map” then, shifts to “ocr”, which leads to http://www.pythonchallenge.com/pc/def/ocr.html

QUESTION 3: Find rare characters in the mess below (mess omitted)
ANSWER: php:
$challenge=file_get_contents('phychall2.txt');
echo(ereg_replace("[^a-z]","",$challenge));

which produces: http://www.pythonchallenge.com/pc/def/equality.html

QUESTION 4: “One small letter, surrounded by EXACTLY three big bodyguards on each of its sides.”
ANSWER: I used textpad’s regex ([a-z][A-Z][A-Z][A-Z][a-z][A-Z][A-Z][A-Z][a-z]) and found “linkedlist”, which means: http://www.pythonchallenge.com/pc/def/linkedlist.html

QUESTION 5: “urllib may help. DON’T TRY ALL NOTHINGS, since it will never end. 400 times is more than enough”
ANSWER:
$linked="12345";
for($x=0;$x<400;$x++)
{
$page=file_get_contents("http://www.pythonchallenge.com/pc/def/linkedlist.php?nothing=$linked");
echo($page."\n");
$previouslinked=$linked;
$linked='';
list($blank,$linked)=@split('and the next nothing is ',$page);
$pos = strpos($page, "Divide");
if ($pos === false)
{
if ($linked=='') die();
}
else
{
$linked=$previouslinked/2;
echo("divide found!!");
}
echo($linked."\n");
}

which produces: http://www.pythonchallenge.com/pc/def/peak.html

QUESTION 6: peak hell
ANSWER:
arghh. have to use python…need to load the pickle dump:

import pickle
unpicklefile = open('banner.p', 'r')
unpickledlist = pickle.load(unpicklefile)
unpicklefile.close()
for item in unpickledlist:
print item

python unpickle.py > nextstep.txt

lines look like: [(’ ‘, 14), (’#', 5), (’ ‘, 70), (’#', 5), (’ ‘, 1)]

parse them with:

$problem=file("nextstep.txt");
$parseout=array('(',']',')','[',"'");
function printit($value,$times)
{
for($loop=0;$loop<$times;$loop++)
echo($value);
}
foreach($problem as $line)
{
$tuples=split('),',$line);
foreach($tuples as $tuple)
{
list($first,$second)=split(',',$tuple);
$first=trim(str_replace($parseout,'',$first));
if ($first=='') $first=' ';
$second=trim(str_replace($parseout,'',$second));
printit($first, $second);
}
echo("\n");
}


which produces a nifty ascii art version of: http://www.pythonchallenge.com/pc/def/channel.html

QUESTION 7: : now there are pairs (in http://www.pythonchallenge.com/pc/def/channel.zip)
ANSWER:

$linked="90052";
for($x=0;$x<909;$x++)
{
$bigArray[]=system("zipinfo -v channel.zip {$linked}.txt|tail -3|head -1",$retval);
$page=file_get_contents("{$linked}.txt");
$previouslinked=$linked;
$linked='';
list($blank,$linked)=@split('Next nothing is ',$page);
$pos = strpos($page, "Collect");
if ($pos === false);
else
{
$x=10000;
}
}
$counter=2;
foreach($bigArray as $dot)
{
if ($dot=='') $dot=' ';
echo($dot);
if ($counter++%65==1) echo("\n");
}

Which produces more ASCII art and leads to: http://www.pythonchallenge.com/pc/def/oxygen.html

QUESTION 8: the next level is [105, 110, 116, 101, 103, 114, 105, 116, 121]
ANSWER:

$im = imagecreatefrompng("oxygen.png");
for($x=3;$x<629;$x+=7)
{
$rgb = imagecolorat($im, $x, 46);
$r = ($rgb >> 16) & 0xFF;
$g = ($rgb >> & 0xFF;
$b = $rgb & 0xFF;
echo(chr($r));
}

more code:

$thelist=array(105, 110, 116, 101, 103, 114, 105, 116, 121);
foreach($thelist as $entry)
{
echo(chr($entry));
}


http://www.pythonchallenge.com/pc/def/integrity.html

QUESTION 9:
un: ‘BZh91AY&SYA\xaf\x82\r\x00\x00\x01\x01\x80\x02\xc0\x02\x00 \x00!\x9ah3M\x07<]\xc9\x14\xe1BA\x06\xbe\x084'
pw: 'BZh91AY&SY\x94$|\x0e\x00\x00\x00\x81\x00\x03$ \x00!\x9ah3M\x13<]\xc9\x14\xe1BBP\x91\xf08'
ANSWER: missing link : bee
use commandline python, arghh…

s = "BZh91AY&SYA\xaf\x82\r\x00\x00\x01\x01\x80\x02\xc0\x02\x00 \x00!\x9ah3M\x07<]\xc9\x14\xe1BA\x06\xbe\x084"
import bz2
bz2.decompress(s)
'huge'
t = "BZh91AY&SY\x94$|\x0e\x00\x00\x00\x81\x00\x03$ \x00!\x9ah3M\x13<]\xc9\x14\xe1BBP\x91\xf08"
bz2.decompress(t)
'file'