There's a whole lot of web out there. Who's
gonna watch it? Even the
smartest search engines and their
spiders can't keep up. dmoz
does them one better by distributing the indexing, but no editor
can expect to monitor each site on the list for updates.
The truth is, just about nobody can handle it alone.
The Solution
Good old fashion helping out. distributed.net
showed us the power of internet-wide distributed computing for
complex number crunching. Why not apply that same power
to managing the internet itself?
The Linkwatcher Distributed Bot Project aims
to build a web portal as flexible and dynamic as the web
itself. Anybody with a computer will be able to join the
project just by installing the
python "sandbox",
where bots can safely run without harming the host machine.
The main linkwatcher site sends out bots - little programs
that do useful things - to play in the sandboxes. They
can do things like compare a web page for changes,
look for broken links.
The Possibilities
Imagine being notified within the hour when your favorite
sites have changed.
If you're a webmaster, imagine submitting a page to a search engine,
and having your site spidered within minutes.
Future systems could use this same software to search
for prime numbers, test genetic algorithms, or do anything
else that takes too long or is too big a task for
a single computer to handle.
The Source
Open source projects just work better. So
this is an open source project. CVS access,
bug tracking, and other fun stuff can therefore be
found at our sourceforge
home page.
It would be nice to use Python for programs involving huge numbers of very light-weight threads. This would make it easy to simulate large swarms of interacting agents, which sounds like fun.
And, who could forget Knowbots, invented by the same people that bring us python?
-Michal Wallace
The python sandbox is all done through the magic of the rexec module (restricted execution)
-Michal Wallace
I'm thinking about calling the system that runs all this "snakepit".... :)
-Michal Wallace
Not sure this applies, but... Hector is a system that sits on top of stuff like CORBA and DCOM and whatnot to help with distributed stuff.. (it's written in python)
-Michal Wallace
