GoAdmio

bots

Report bot visits from the access log, add newly detected bots, and flag likely bad bots for blocking

On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also


Overview

/go/bots shows which bots have visited your server, good and bad. It:

This command changes the bot list. It is not a read-only report.


Quick start

/go/bots

Syntax

/go/bots

The command takes no options or arguments.


Options

None. Any arguments you pass are ignored.


Output

The output is a table with three columns, sorted by visits from lowest to highest, so the most active bots are at the bottom.

Column Meaning
BOT Bot name, as found in the user-agent field of the access log
Visits Number of times the name appears in the access log
Blocked 1 if the bot is flagged for blocking, 0 if not

Example (names and numbers are illustrative):

BOT          Visits  Blocked
------       ------  -------
ExampleBot   12      1
SomeCrawler  240     0
OtherSpider  1850    0

Bots with no visits in the access log are not shown. Each run also adds a line to /var/log/go/bots.log.


Examples

# Show bot visits
/go/bots
# List every bot currently flagged for blocking
sqlite3 /var/lib/sqlite/bot.db "select Name from Bot where Block=1"
# Stop flagging a bot that was marked by mistake
sqlite3 /var/lib/sqlite/bot.db "update Bot set Block=0 where Name='ExampleBot'"

Troubleshooting

Symptom Likely cause Fix
The table only has the header The access log has no matching bot visits. Check that the access log is being written.
A legitimate bot is marked 1 Its name matched the bad-bot rules (see Detection rules). Set Block=0 for it in bot.db.
A bad bot is listed with 0 Its name did not match the rules, or it contains google, yandex, adsbot, or bing. Set Block=1 for it in bot.db, or use bottop.
A flagged bot still visits Bot blocking is off for the domain, or the configuration was not rebuilt. Enable it with bot, then run conf or reload.
The command is slow The access log is large and is searched once for every bot. Run it at a quiet time, or rotate the log with logrotate.

Details

What it does, in order

  1. Logs the run to /var/log/go/bots.log.
  2. Loads the access log path and starts the output table.
  3. Reads every bot name from the Bot table in /var/lib/sqlite/bot.db.
  4. For each known bot, counts its occurrences in the access log and adds it to the table if it appears.
  5. Extracts the user-agent field from the access log and keeps the words that contain a bot-like term (bot, spider, agent, client, crawl, http, index, search, scraper, fetcher, scan, crawler). The list is saved to the temporary text file /usr/local/goadmio/tmp/bot.db.
  6. For each new word, applies the detection rules, adds it to the bot list, and adds it to the table if it appears in the log.
  7. Sorts and prints the table.

Detection rules

A word from the log is ignored when it is https, bots, robot, robots, or Client, when it contains a dot, or when it is already in the bot list (case-insensitive).

A new word is added to the bot list with Block=1 only when all of these are true:

Rule Condition
Not a search engine The name does not contain google, yandex, adsbot, or bing.
Bot-like name The name contains bot, crawl, spider, scan, or activity.
Simple name It has only letters, numbers, and hyphens.
Length It has 8 or more characters.

Every other new word is added with Block=0.

Known issues


See also

Command Description
bot Block malicious bots from accessing Apache or Nginx
bottop Block bad bots from visiting your server
blacklist Block blacklisted IPs (bad bots and SSH brute-force attacks)
iptop Track the most visited IP addresses on a server
sizetop Show the largest responses in the access log
conf Rebuild the Apache/Nginx configuration in GoAdmio