SPIDERS

When search engines (like Yahoo or Alta Vista, for example) are used to find information on the Internet, the results one receives normally come from giant indexes or databases, instead of from the actual Internet in real time. Because the Internet changes constantly, a search engine's index must be continually updated. Spiders are the tools used for accomplishing this critical task. They work in tandem with indexes and search software to comb the Web for information. Without spiders, it would be difficult to find new Web sites, or current content on existing ones. Also called crawlers, ants, or wanderers, spiders technically are a member of the bot family—software programs that operate unattended, usually on the Internet. Therefore, spiders often are referred to as bots. However, it's important to know that spiders are not the same as intelligent agents—another kind of bot that has a wider range of capabilities, including interactivity.

Spiders travel from server to server, visiting different areas of the Internet—normally sites on the World Wide Web but also File Transfer Protocol (FTP) sites and Gopher archives. This process, known as discovery, can be performed blindly or in a more directed manner. When done blindly, a spider attempts to visit every possible Internet Protocol (IP) address—unique numbers assigned to every machine on the Internet. This approach takes longer than a directed approach, which involves searching registered domain names, or the names used to identify a site (such as Intel.com). Both approaches have advantages and drawbacks. Large search engines often employ many spiders at once, working in parallel on many different machines or servers to archive the online world in one database. After spiders report back to search engines with new information, it often takes additional time before the information is updated in the engine's index and made available for end-users to see in search results.

Some spiders record every word on a Web site for their respective indexes, while others only report certain keywords listed in meta tags. Although they usually aren't visible to someone using a Web browser, meta tags are special codes that provide keywords or Web site descriptions to spiders. Sometimes, the information listed in meta tags is incorrect or misleading, which causes spiders to deliver inaccurate descriptions of Web sites to indexes. In any case, the issue of keywords and how they are placed, either within actual Web site content or in meta tags, is important to online marketers. The majority of consumers reach e-commerce sites through search engines, and the right keywords increase the odds a company's site will be included in search results.

While spiders are critical elements of the online world, they also were a source of aggravation and controversy in the early 2000s. On the technical side, spiders sometimes slow down the performance of Web servers—the computers or applications that host Web sites—by visiting them over and over in a short period of time, sometimes as often as 100 times in a single minute. An example of this type of behavior includes spiders that search for up-to-the-minute news, or product or stock-market information. For companies without strong technical systems, spiders that exhibit this kind of behavior can cause major problems.

Another concern centered around how information collected by spiders was gathered, redistributed to other parties, and ultimately used. Part of this concern, which created related legal issues, involved security, because spiders sometimes uncover information a site's owner considers private or off-limits to the public. Another issue was misrepresentation, especially for distributors who risked having old, incorrect information about product availability or inventory displayed on other Web sites.

Because of these concerns, Web site administrators took measures to deny spiders access to their Web sites, or to certain areas within them. Administrators post specific rules about what spiders are allowed to access on their sites in an exclusion file called robots.txt, which spiders normally find and read. These rules can also be seen by the naked eye if one adds robots.txt to the end of a site's address or uniform resource locator (URL). By looking at an access log, site administrators are able to determine which spiders have visited their sites and what information they recorded. In this situation, spiders can be identified individually by name, which are given to them by their creators. This gives administrators the ability to exclude certain bots from visiting in the future, should they present a problem.

FURTHER READING:

Baljko, Shah. "Web Crawling: Sticky Issue For Distributors." Planet IT, November 1, 2000. Available from www.PlanetIT.com.

Champlin, Leslie. "E-firms Lure Search Spiders to Their Corner of the Web." The Business Journal of Kansas City, March 31, 2000. Available from www.kansascity.bcentral.com.

"Crawler." Tech Encyclopedia, February 1, 2001. Available from www.techweb.com/encyclopedia.

"How Search Engines Work." Search Engine Watch, February 1, 2001. Available from www.searchenginewatch.com.

Pallmann, David. Programming Bots, Spiders, and Intelligent Agents in Microsoft Visual C++. Redmond, Washington: Microsoft Press. 1999.

"Spider." Ecommerce Webopedia, February 1, 2001. Available from e-comm.webopedia.com.

"Spider." Netlingo, January 31, 2001. Available from www.netlingo.com/inframes.

"SpiderSpotting." Search Engine Watch, February 1, 2001. Available from searchenginewatch.internet.com.

"What's a Bot?" Internet.com , February 6, 2001. Available from www.bots.internet.com.

SEE ALSO: Intelligent Agents; Results Ranking; Search Engine Strategy; Webcrawler

spiders
1. See ARACHNIDA; ARANEAE; LIPHISTIIDAE; MIMETIDAE; SPARASSIDAE; TRAPDOOR SPIDERS.

2. cobweb (comb-footed) spiders See THERIDIIDAE.

3. crab spiders See THOMISIDAE.

4. funnel-web spiders, tarantulas See MYGALOMORPHAE; THERAPHOSIDAE.

5. hunting (wolf) spiders See LYCOSIDAE.

6. jumping spiders See SALTICIDAE.

7. money spiders See LINYPHIIDAE.

8. nursery-web spiders See PISAURIDAE.

9. orb-web spiders See ARANEIDAE.

10. purse-web spiders See ATYPIDAE.

11. sheet-web spiders See AGELENIDAE.

12. spitting spiders See SCYTODIDAE.

Spider

Various folklore beliefs surround the spider. In England, spiders were known as "money makers." If found on clothing, they were a sign that money was on the way, provided that the spider was not killed. A similar idea prevailed in Polynesia, where a spider dropping down in front of a person was a sign of a present. An American belief is that killing a spider will bring rain.

In folk medicine, a spider was rolled in butter or molasses and swallowed. As a cure for ague, it was tied up and secured on the left arm. A spider was also traditionally used as an amulet. The insect was baked and worn around the neck.

The British antiquary Elias Ashmole stated in his Memoirs (1717): "I took early in the morning a good dose of elixir, and hung three spiders around my neck, and they drove my ague away. Deo Gratias!"

Robert Burton (1577-1640) stated:

"Being in the country in the vacation time, not many years since, at Lindly in Leicestershire, my father's house, I first observed this amulet of a spider in a nut-shell, wrapped in silk, so applied for an ague by my mother…. This I thought most absurd and ridiculous, and I could see no warrant in it … till at length, rambling amongst authors, I found this very medicine in Dioscorides, approved by Matthiolus, repeated by Aldrovandus…. I began to have a better opinion of it, and to give more credit to amulets, when I saw it in some parties answer to experience."

Spiders were sacred to the ancient Egyptian goddess Maat and are used today as symbolism of a Maatian (feminist) form of ceremonial magic.

spi·der / ˈspīdər/ • n. an eight-legged predatory arachnid (order Araneae, class Arachnida) with an unsegmented body consisting of a fused head and thorax and a rounded abdomen. Spiders have fangs that inject poison into their prey, and most kinds spin webs in which to capture insects. ∎  used in names of similar or related arachnids, e.g., sea spider. ∎  any object resembling a spider, esp. one having numerous or prominent legs or radiating spokes. ∎  a cast-iron iron frying pan, originally made with legs for cooking on coals in a hearth.• v. [intr.] move in a scuttling manner suggestive of a spider: a nuthatch spidered head first down the tree trunk. ∎  form a pattern suggestive of a spider or its web.DERIVATIVES: spi·der·ish adj.

spider in proverbial and traditional allusion, references are made to the cunning, skill, and industry of the spider, as well as its power of secreting or emitting poison. In a traditional story of the Scottish nationalist leader Robert the Bruce (1274–1329, from 1301 Robert I of Scotland) encouraged by watching a spider in a cave attempting to spin a thread long enough to reach another piece of rock, it becomes a type of perseverance.

A spider is also said to have helped Muhammad. According to the story, the Prophet and his friend Abu-Bakr were in flight from the men of Mecca and took shelter in a cave; to protect them, pigeons built their nests and a spider spun its web across the mouth of the cave, so that the pursuers assumed that it was undisturbed.

In Greek mythology, the weaver Arachne who challenged Athene to a contest of skill was changed into a spider by the angry goddess.

See also if you want to live and thrive, let the spider run alive.