GoAdmio

404top

Report IPs causing the most 404 errors and block scanners and probes

On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also


Overview

/go/404top finds the IP addresses that caused the most 404 Not Found responses in the last hour. Scanners and vulnerability probes request many pages that don't exist, so a high 404 count is a strong sign of one. The command:

No setup is needed. GoAdmio's default cron jobs already run 404top block every hour.


Quick start

# Show the report only. Nothing is blocked.
/go/404top

# Show the report and block offenders
/go/404top block

Syntax

/go/404top [block]
Form Behavior
/go/404top Prints the table. Nothing is blocked and no notification is sent.
/go/404top block Prints the table, blocks offenders, and sends notifications. This is what cron runs.

The command has no -- options. The first argument is the only one it reads.

Any other first argument is treated like no argument. There is no "unknown parameter" error. A typo such as /go/404top blok runs in report-only mode and blocks nothing.


Options

Item Required Default Notes
block (argument) No Report only Turns on blocking and notifications.
/var/go/etc/404top (file) No 300 A whole number: the 404 count at which an IP is blocked. See Threshold.

Output

Outcome What you get
Report A table of up to 25 IPs with visits, country, city, and company.
block mode, IPs blocked The same table, then an email and a WhatsApp message listing the blocked IPs.
block mode, nothing over the threshold The table only. No message is sent.
block mode, another instance running Instance (<name>) already running. and exit code 1.

Report

The columns are aligned with column. This sample is illustrative.

IP            VISITS  COUNTRY  CITY     COMPANY
--            ------  -------  ----     -------
203.0.113.7   412     US       Chicago  Example Hosting LLC
198.51.100.22 87      DE       Berlin   Example Telecom

The server's own IPv4 address, its IPv6 address, and 127.0.0.1 are never listed.

Notification

Sent only in block mode, and only if at least one IP was blocked in that run.

Channel Subject or title
Email, through sendmail High 404 errors detected (<N> IPs)
WhatsApp, through sendwhatsapp *High 404 errors detected (<N> IPs)*

The body explains what happened, then lists one line per blocked IP:

- 404 Block 203.0.113.7 (412 Visits) - Country: US, Company: Example Hosting LLC.

[!WARNING] Blocking is automatic and unattended. A threshold set too low can block real visitors, especially many people sharing one IP (an office, a school, or a mobile carrier). Keep the default of 300 unless you have a reason to change it, and read the notification emails after any change.


Examples

# Report only: see who is generating 404s right now
/go/404top
# Report and block, as cron does every hour
/go/404top block
# Block at 500 404s per hour instead of 300
echo 500 | sudo tee /var/go/etc/404top
# Go back to the default threshold
sudo rm /var/go/etc/404top
# Read the history of runs and their arguments
sudo tail /var/log/go/404top.log

Troubleshooting

Symptom Likely cause Fix
Instance (<name>) already running. A block run started less than 60 seconds ago. Wait, then run again. A lock older than 60 seconds is ignored.
Nothing was blocked Ran without block, no IP reached the threshold, or the IP was exempt. Run /go/404top block and check the Threshold and exemptions.
An IP over the threshold wasn't blocked A file /var/log/go/block/<IP>.* already exists, or the company is exempt. See Already-blocked IPs and exempt visitors.
No notification arrived No IP was blocked in that run, or email and WhatsApp aren't configured. Check the table for blocks, then test sendmail and sendwhatsapp.
Country or city is blank in cron output Cron runs use the whois record, which may lack those fields. Run the command in a terminal for fuller lookups.
A legitimate IP was blocked It sent many requests for missing pages. Run unblock for that IP, fix the broken links on the site, and consider raising the threshold.
The threshold change had no effect The file contains something other than digits. Put only a whole number in /var/go/etc/404top. Anything else falls back to 300.
The server's own IP shows in the table The lookup of the server's public IP failed. Check that the server can reach checkip.amazonaws.com and icanhazip.com.

Details

Threshold

Condition Limit used
/var/go/etc/404top doesn't exist 300
It exists and contains only digits That number
It exists but isn't a whole number 300

An IP is blocked when its 404 count is greater than or equal to the limit.

What counts as a 404

Check Rule
Time window The last 3600 seconds (one hour), judged by the timestamp in the log line.
Matching Log lines containing the text 404 (with spaces on both sides).
Visitor IP The forwarded client IP if the log has one. Otherwise the connecting IP.
Counting Case-insensitive per IP, highest first.
Rows kept The top 25.

The match is on the text 404 anywhere in the line, not only in the status-code field.

Exempt visitors

An IP is never blocked if the company name found for it contains:

Text Case-sensitive
google No
bing Yes (lowercase)
cloudflare Yes (lowercase)

Exempt visitors still appear in the table. Because the bing and cloudflare checks are case-sensitive, a company name written as Cloudflare, Inc. would not match them.

Already-blocked IPs

An IP is skipped if any file named /var/log/go/block/<IP>.* exists. Such a file is the record left by block. An IP that already has one is not blocked again and is not included in the notification.

Lookups

Data Terminal run Cron or other non-terminal run
Company whois record whois record
Country Geo lookup country: field of the whois record
City Geo lookup city: field of the whois record

Locking

Only block mode takes a lock, so report-only runs can overlap freely.

Item Value
Lock file /usr/local/goadmio/lock/block.lock
Contents The name of the command holding the lock
Considered stale after 60 seconds without an update
Refreshed Once per IP processed, so a long run stays locked
Released On exit, including Ctrl+C

What it does

In short: take the lock (block mode), read the log, rank the visitors, look each one up, block those over the limit, print the table, then notify.

flowchart TD
    A["404top [block]"] --> B{"First argument is block?"}
    B -- "Yes" --> L{"Lock under 60 s old?"}
    L -- "Yes" --> X["Instance already running, exit 1"]
    L -- "No" --> C["Take the lock"]
    B -- "No" --> D
    C --> D["Read threshold, default 300"]
    D --> E["Find top 25 IPs by 404s in the last hour"]
    E --> F["For each IP: skip own and local IPs, look up whois"]
    F --> G{"block mode, over limit, not exempt, not already blocked?"}
    G -- "Yes" --> H["/go/block IP 404 country company"]
    G -- "No" --> I["Add row to the table"]
    H --> I
    I --> J["Print table"]
    J --> K{"Any IPs blocked?"}
    K -- "Yes" --> M["Send email and WhatsApp"]
    K -- "No" --> N["Release the lock"]
    M --> N
  1. Appends the date and arguments to /var/log/go/404top.log.
  2. Loads shared functions from /go/function and the access log path.
  3. In block mode, exits with an error if block.lock is under 60 seconds old. Otherwise writes the lock.
  4. Works out the cutoff time, one hour ago.
  5. Reads the hostname and finds the server's public IPv4 and IPv6 addresses.
  6. Reads the threshold from /var/go/etc/404top, falling back to 300.
  7. Counts 404 log lines per visitor IP within the last hour and keeps the top 25.
  8. Skips the server's own IPs and 127.0.0.1.
  9. Looks up whois, company, country, and city for each remaining IP.
  10. In block mode, for each IP at or above the limit that isn't exempt and has no record in /var/log/go/block/, runs /go/block <IP> 404 <country> <company> and adds it to the notification list.
  11. Adds each IP to the output table.
  12. Prints the table.
  13. If any IP was blocked, sends the email and the WhatsApp message.
  14. Removes the lock file.

See also

Command Description
block Block an IP address on a GoAdmio server
unblock Unblock an IP address on a GoAdmio server
white Add an IP to the whitelist
bottop Block bad bots from visiting your server
bot Block malicious bots from accessing Apache or Nginx
bots Track all bot visits to your server (good and bad bots)
ddos Track and block DDoS attack IPs in GoAdmio
iptop Report the busiest IPs by request count and block floods and bot networks
posttop Track and check IPs making POST requests
cron Set up main cron jobs in GoAdmio