List:General Discussion« Previous MessageNext Message »
From:Chris Trown Date:September 13 1999 8:53pm
Subject:Re: [mySQL] Spiders and Robots
View as plain text  
On Mon, Sep 13, 1999 at 04:37:32PM -0700, d e l   d h a n o a scribbled:
> Hello,
> 
> I have a question concerning spiders and robots.  Is a spider/robot a little
> program written in PHP which searches the web for sites and then enters them
> into a databse automatically?
> 

     Writing a robot using PHP would be a neat trick.

     Spiders are written in many languages, usually C/C++ or even Perl.

> If so I would really like to learn how to program one.  Does anyone have any
> tutorials/articles on this subject so I can learn?
> 
> Also, does anyone know of any open source code for spiders/robots?
> 

     There are enough spiders in the world, please don't create another.

     Yes, there are spiders out there that are open source.  HT://Dig
(www.htdig.org) is one that comes to mind.

     You can do a search on Yahoo for interesting web pages.  One that I used
to visit often when I was researching these things is:

http://info.webcrawler.com/mak/projects/robots/robots.html

Chris...

-- 
What's the point?					NT is not the solution.
							     NT is the problem.
Computers make very fast, very accurate mistakes.
PGP fingerprint: 063FCE320681C336  78C164FC9B2F91EA
Thread
Access to MySQLRobert Mullen13 Sep
  • Access to MySQLJani Tolonen13 Sep
  • [mySQL] Spiders and Robotsd e l   d h a n o a14 Sep
    • Re: [mySQL] Spiders and RobotsChris Trown14 Sep
  • [mySQL] accuracy for searchesd e l   d h a n o a14 Sep