No announcement yet.

What’s a ’spider’?

  • Filter
  • Time
  • Show
Clear All
new posts

    What’s a ’spider’?

    From the frontpage of the web-site:

    Total Users Online:

    We have 44 member(s), <b style="color: rgb(255, 0, 0);">13[/b]@@@@SPAN style="color: rgb(255, 0, 0);"> spiders,@@@@/SPAN></font> 31 guest(s), and 1 anonymous users online.

    Any ideas?

    Trust is good; control is better. V I Lenin.

    <h1 ="firsting">Web crawler</h1>

    <h3 id="siteSub">From Wikipedia, the free encyclopedia</h3>

    <div id="jump-to-nav">Jump to: navigation, search</div> <!- start c&#111;ntent ->
    <dl><dd>See WebCrawler for the specific search engine of that name.</dd></dl>

    A web crawler (also known as a Web spider</font> or Web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Other less frequently used names for Web crawlers are ants, automatic indexers, bots, and worms (Kobayashi and Takeda, 2000).

    This process is called Web crawling or spidering. Many legitimate sites, in particular search engines,
    use spidering as a means of providing up-to-date data. Web crawlers are
    mainly used to create a copy of all the visited pages for later
    processing by a search engine, that will index
    the downloaded pages to provide fast searches. Crawlers can also be
    used for automating maintenance tasks on a Web site, such as checking
    links or validating HTML
    code. Also, crawlers can be used to gather specific types of
    information from Web pages, such as harvesting e-mail addresses
    (usually for spam).

    A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.


      Thanks Robbie!
      Trust is good; control is better. V I Lenin.