This new LookSmart initiative is an exciting and highly intelligent response to the currently massive (and growing) size of the web. Search engines are overwhelmed by the sheer number of sites applying to be indexed and consequently the web site information they reflect may in fact be several weeks out of date. To address this problem, LookSmart has released the Grub Distributed Crawling Client . The idea is to distribute the task of crawling sites to participating users. The software (now in beta) is easy to use and configure and does not affect computer performance since it operates only when the processor is otherwise idle.
LookSmart hopes not only to grow its own database but eventually to crawl every site on the web daily ! The success of the scheme obviously depends on the number of participating users. They will, I hope, be many since the benefits of it are obvious to webmasters and web users alike. Webmasters however have the additional incentive that the Grub agrees to crawl their own sites on a regular basis.
