bots
Report bot visits from the access log, add newly detected bots, and flag likely bad bots for blocking
- Command:
/go/bots - Category: Security
- Required option: None
- Output: A table of bots with their visit counts and blocked status
- Last updated: 2026-11-20
On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also
Overview
/go/bots shows which bots have visited your server, good and bad. It:
- counts the visits of every bot already in the GoAdmio bot list
- scans the access log for new bot names
- adds new bots to the bot list, and marks those that look malicious as blocked
- prints a table sorted by visits
This command changes the bot list. It is not a read-only report.
Quick start
/go/bots
Syntax
/go/bots
The command takes no options or arguments.
Options
None. Any arguments you pass are ignored.
Output
The output is a table with three columns, sorted by visits from lowest to highest, so the most active bots are at the bottom.
| Column | Meaning |
|---|---|
BOT |
Bot name, as found in the user-agent field of the access log |
Visits |
Number of times the name appears in the access log |
Blocked |
1 if the bot is flagged for blocking, 0 if not |
Example (names and numbers are illustrative):
BOT Visits Blocked
------ ------ -------
ExampleBot 12 1
SomeCrawler 240 0
OtherSpider 1850 0
Bots with no visits in the access log are not shown. Each run also adds a line to /var/log/go/bots.log.
Examples
# Show bot visits
/go/bots
# List every bot currently flagged for blocking
sqlite3 /var/lib/sqlite/bot.db "select Name from Bot where Block=1"
# Stop flagging a bot that was marked by mistake
sqlite3 /var/lib/sqlite/bot.db "update Bot set Block=0 where Name='ExampleBot'"
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| The table only has the header | The access log has no matching bot visits. | Check that the access log is being written. |
A legitimate bot is marked 1 |
Its name matched the bad-bot rules (see Detection rules). | Set Block=0 for it in bot.db. |
A bad bot is listed with 0 |
Its name did not match the rules, or it contains google, yandex, adsbot, or bing. |
Set Block=1 for it in bot.db, or use bottop. |
| A flagged bot still visits | Bot blocking is off for the domain, or the configuration was not rebuilt. | Enable it with bot, then run conf or reload. |
| The command is slow | The access log is large and is searched once for every bot. | Run it at a quiet time, or rotate the log with logrotate. |
Details
What it does, in order
- Logs the run to
/var/log/go/bots.log. - Loads the access log path and starts the output table.
- Reads every bot name from the
Bottable in/var/lib/sqlite/bot.db. - For each known bot, counts its occurrences in the access log and adds it to the table if it appears.
- Extracts the user-agent field from the access log and keeps the words that contain a bot-like term (
bot,spider,agent,client,crawl,http,index,search,scraper,fetcher,scan,crawler). The list is saved to the temporary text file/usr/local/goadmio/tmp/bot.db. - For each new word, applies the detection rules, adds it to the bot list, and adds it to the table if it appears in the log.
- Sorts and prints the table.
Detection rules
A word from the log is ignored when it is https, bots, robot, robots, or Client, when it contains a dot, or when it is already in the bot list (case-insensitive).
A new word is added to the bot list with Block=1 only when all of these are true:
| Rule | Condition |
|---|---|
| Not a search engine | The name does not contain google, yandex, adsbot, or bing. |
| Bot-like name | The name contains bot, crawl, spider, scan, or activity. |
| Simple name | It has only letters, numbers, and hyphens. |
| Length | It has 8 or more characters. |
Every other new word is added with Block=0.
Known issues
- Names such as
Applebot,Slackbot, orTwitterbotmatch the rules and are flaggedBlock=1. Review the flagged bots after a run. - Visits are counted as text matches anywhere in the access log, not only in the user-agent field, and the match is case-sensitive.
- The bot description is always empty.
- The command does not rebuild the web server configuration, so a new flag may need
conforreloadbefore it takes effect.
See also
| Command | Description |
|---|---|
bot |
Block malicious bots from accessing Apache or Nginx |
bottop |
Block bad bots from visiting your server |
blacklist |
Block blacklisted IPs (bad bots and SSH brute-force attacks) |
iptop |
Track the most visited IP addresses on a server |
sizetop |
Show the largest responses in the access log |
conf |
Rebuild the Apache/Nginx configuration in GoAdmio |