Agents, Bots, and Spiders

Posted by ljmacphee on January 29, 2007 under bots, source code, topics in artificial intelligence |

‘Bot’ is short for ‘robot’ and refers to s software robot. A bot goes out into the Internet and pulls back data. Bots are used to handle repetitive tasks, to index the web for search engines, by games to be your avatar and by any program that sends out ‘robots’ to perform tasks. Java and Python are the preferred language for them.

A spider is a specialized bot. A spider starts on a given web-page and is restricted to a given domain or set of domains. The spider traverses the area by collecting links on the page and going from link to link.

Information gathering agents are all over the Internet now. They are the spiders that index web pages for you to find. There are bots that traverse databases and pullout information to serve up to users. Since computers do not understand language teaching bots to gather and sort information is quite a challenge. Several techniques are discussed here.

WebMate uses multiple TF-idF vectors each one in a different domain of interest to the user, as well as ‘Trigger Pair’ word searches in documents, and WebMate learns from watching the user. It runs between the browser and the HTTP server monitoring transactions. WebMate learns the classifications rather than have the user select and feed them to the program. The program learns incrementally and changes as the users interests change. The learning algorithm is run whenever the user flags a document as useful.

Information gathering agents may use the SIMS architecture. Each agent is a specialist in a different subject. These agents use KQML as the communication language between them, and LOOM as the knowledge representation language.

One agent is created, then others are instantiated to become experts in different areas of knowledge (flight schedules, hotel locations and rates, etc) and an area of domain (type of database, physical location, etc). A network of these agents is then put together in an acyclic graph.

Link Checker Spider (JAVA)

More information:
Alice
Alpha Works
BotSpot

See also:
Cognitive code develops software personal assistant using Lua
FBI’s bot roast finds compromised computers
AI promoting ‘Flatland’
Personal agents may reach your phone before they reach your computer