Web - Robots (Wanderers | Crawlers | Spiders)

> (World Wide) Web - (W3|WWW)

1 - About

Bot in a web context.

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are crawler program that scan the web generally

More than half of all web traffic is made up of bots.


3 - Example of report

Example of bots agent report on this website.

where: user-agent is the user agent given by the robot.

3.1 - Implementation

bot are generally implemented with a headless browser library.

4 - Management

4.1 - Configuration

4.2 - Inspect

See how google Bot see you website at: URL inspection of Search Console