Robots.txt file controls Web spiders (HTTP_RobotsTxt)

About this signature or vulnerability

Proventia Server IPS for Linux technology, RealSecure Desktop, RealSecure Desktop Protector 3.6, Proventia Network IPS, Proventia Desktop, Proventia Network IDS, Proventia-G 1.1 and earlier, IBM Security Server Protection for Windows, Proventia Network MFS, BlackICE Server Protection, BlackICE PC Protection, RealSecure Sentry, RealSecure Guard, BlackICE Agent for Server, RealSecure Server Sensor, RealSecure Network, Virtual Server Protection for Vmware:

This signature detects an HTTP URL that contains the string '*/robots.txt'.

Looks for HTTP URL matching '*/robots.txt'.

Default risk level

Low risk vulnerability  Low

Sensors that have this signature

Proventia Server IPS for Linux technology: 1.0, RealSecure Desktop: baseline, RealSecure Desktop Protector 3.6: baseline, Proventia Network IPS: 2.0, Proventia Desktop: 8.0.614.1, Proventia Network IDS: A Series, Proventia-G 1.1 and earlier: G Series, IBM Security Server Protection for Windows: 1.0.914.0, IBM Security Server Protection for Windows: 2.1.14.2400, Proventia Network MFS: 1.0, BlackICE Server Protection: 3.6.cbd, BlackICE PC Protection: 3.6.cbd, RealSecure Sentry: 3.6, RealSecure Guard: 3.6, BlackICE Agent for Server: 3.6, RealSecure Desktop Protector: 3.6, RealSecure Server Sensor: 7.0, RealSecure Server Sensor: 5.5.2, RealSecure Network: 3.2, RealSecure Network: 7.0, BlackICE: 3.5ebo, Virtual Server Protection for Vmware: 1.0

Systems affected

Various vendors Any application, IETF HTTP/1.1

Type

Pre-attack Probe

Vulnerability description

The robots.txt file is commonly placed in the root directory of a system's Web server to control the actions of Web robots (often called crawlers or spiders). All robots that adhere to the Robots Exclusion Standard (see References) will check this file on your server before proceeding to index or search your site. A user who is able to modify the contents of the robots.txt file could control the actions of Web robots on your server.

How to remove this vulnerability

This is not a vulnerability. Administrators should review the contents of the robots.txt file to check if the information is consistent with the policies of their organization.

References

The Web Robots Pages Web site
A Standard for Robot Exclusion
http://www.robotstxt.org/wc/norobots.html

ISS X-Force
Robots.txt file controls Web spiders
http://www.iss.net/security_center/static/1533.php