GoAdmio
GoAdmio Documentation

Bot Manager

From the GoAdmio knowledge base.

Bot Manager

This page explains the Bot Manager screen in GoAdmio. It provides a server-wide catalog of known bots and lets you block them by name / user agent — across every domain on the server.

Location: Admin Panel → Security → Bot Manager
Location (User Panel): Security → Bot Manager (if enabled for the account)


Overview

Every HTTP request carries a User-Agent header that identifies the client. Legitimate clients — browsers, search engines, monitoring tools — send recognizable user agents. Bad bots disguise themselves as search engines, scrapers, or generic clients to harvest content, probe for vulnerabilities, or generate spam traffic.

The Bot Manager lists 382 known bots (as of this writing) and lets you mark each one as Blocked (✔) or Allowed (✖). The blocked list is enforced at the web server level for every domain on the server, so one block applies everywhere.

Use Bot Manager to:

  • Block content scrapers (AhrefsBot, 360spider, etc.)
  • Block AI training crawlers (Amazonbot, various research bots)
  • Block known malicious bots (ALittle, AliyunSecBot, etc.)
  • Allow legitimate crawlers (Googlebot, Bingbot, AdsBot-Google, etc.) while blocking everything else
  • Reduce bandwidth and server load from unwanted traffic
  • Protect content from unauthorized harvesting
> Note: Bot Manager differs from Bot Blocker, which blocks bots based on heuristics and live traffic (rate, behavior). Bot Manager blocks by name from a curated list. Use both together: Bot Manager for known bots, Bot Blocker for unknown or misbehaving ones. See IP Blocker for IP-based blocking.

What the Page Shows

| Column / Field | Description |
|---|---|
| Name | The bot's name (matched against the User-Agent header) |
| Blocked | ✔ = blocked, ✖ = allowed |
| Action | Toggle the block state |
| Search | Filter the list by bot name |
| Pagination | Showing 20 of 382 records — navigate pages with the numbered links |

> Tip: Use the Search box to find a specific bot (e.g., Ahrefs, Google, Amazon) instead of scrolling through 382 entries.


The Toggle — Blocked vs. Allowed

| State | Icon | Effect |
|---|---|---|
| Blocked | ✔ | Requests with this User-Agent are rejected (typically with 403 Forbidden) |
| Allowed | ✖ | Requests with this User-Agent are not blocked by Bot Manager |

> Note: The ✖ icon here does not mean "blocked" — it means "not blocked by Bot Manager." This is the opposite of the checkbox convention in some panels. Read the state carefully before toggling.

> Warning: Allowing a bot in Bot Manager does not guarantee it will be served — it may still be blocked by IP Blocker, Bot Blocker, firewall rules, or the site's own .htaccess. Bot Manager only controls the name-based filter.


How to Block a Bot

  1. Log in to the GoAdmio Admin Panel.
  2. Navigate to Security → Bot Manager.
  3. Use the Search box to find the bot (e.g., AhrefsBot).
  4. Click the Blocked toggle (or the ✔/✖ icon) to set it to ✔ Blocked.
  5. Confirm if prompted.
The block is applied immediately across all domains.

> Tip: Changes are server-wide. Blocking AhrefsBot here affects every customer on the server — coordinate with account owners if any of them rely on that bot for SEO analysis.


How to Allow a Bot

If a bot is currently blocked but you want to allow it (e.g., a customer needs a specific crawler for their SEO):

  1. Open Security → Bot Manager.
  2. Search for the bot.
  3. Set its state to ✖ Allowed.
  4. Save.
> Tip: Allowing a bot does not automatically override an explicit .htaccess deny rule in a customer's directory. Verify the bot works for the affected domain after the change.

How the Block Is Enforced

GoAdmio writes the blocked bot list into the web server configuration:

  • Apache — a RewriteCond / SetEnvIf block or a shared include that rejects matching User-Agents
  • NGINX — a map or if block that returns 403 for matching User-Agents
Changes take effect immediately. No service restart is required in most versions.

> Warning: User-Agent blocking is trivially bypassable — a bot can simply change its User-Agent to anything. Bot Manager is effective against known bots that do not adapt, but not against sophisticated attackers. Combine it with:
> - IP Blocker for source-IP blocks
> - Bot Blocker for behavioral heuristics
> - A WAF or CDN for advanced filtering


Categories of Bots in the List

The 382 bots fall into several categories:

| Category | Examples | Recommended action |
|---|---|---|
| Search engines | Googlebot, Bingbot, DuckDuckBot, YandexBot | Keep allowed (✖) — blocking hurts SEO |
| Search engine ads | AdsBot-Google, AdsBot-Google-Mobile | Keep allowed — blocking can break Google Ads verification |
| SEO crawlers | AhrefsBot, SemrushBot, MJ12bot, DotBot | Block if you do not use the tool; allow if customers rely on it |
| AI / research crawlers | Amazonbot, AIProductRegistryBot, amilabs-interleaved-research | Block if you do not want content used for AI training |
| Content scrapers | 360spider, adstxtcrawl, agentcensus-crawler | Block |
| Security scanners | AliyunSecBot | Block if not needed |
| Monitors / uptime checks | UptimeRobot, Pingdom, StatusCake | Allow if you use them for monitoring |
| Social / preview | FacebookBot, Twitterbot, LinkedInBot, Slackbot | Allow — blocking breaks link previews |
| Generic clients | aiohttp, AndroidHttpClient, agents | Block if they are causing abuse; allow if they are used by legitimate apps |
| Unknown / niche | ALittle, AgentTrustBot, AionBot | Investigate before blocking; default to block if not needed |

> Tip: The Search engine and Ads bots are the safest to allow. The SEO and AI crawlers are a business decision — some customers want them, others do not.


Recommended Baseline

A sensible starting point:

| Category | Default |
|---|---|
| Search engines (Googlebot, Bingbot, etc.) | Allowed |
| Search engine ads | Allowed |
| Social preview bots | Allowed |
| Monitoring bots (if you use them) | Allowed |
| SEO crawlers (Ahrefs, Semrush, MJ12) | Blocked unless a customer requests otherwise |
| AI crawlers (Amazonbot, research bots) | Blocked unless the customer wants to be indexed by AI |
| Content scrapers | Blocked |
| Security scanners | Blocked unless you run them yourself |
| Unknown / niche bots | Blocked by default; allow on request |

> Warning: Before blocking, check whether any customer's site depends on the bot. For example, blocking AhrefsBot will remove their site from Ahrefs' index; blocking AdsBot-Google can cause Google Ads to pause ads for that site.


Finding Block Candidates

To decide what to block, look at what is actually hitting the server:

| Command | What it shows |
|---|---|
| /go/httptop | Live Apache requests with User-Agents |
| /go/bottop block | Top bots by request volume — block them in bulk |
| /go/bots | Track bot activity across the server |
| /go/404top | Bots generating many 404s (scanners) |

> Tip: Use /go/bottop block to bulk-block the highest-volume bots — it is often faster than toggling individual entries in Bot Manager.


Verifying a Block

Test with curl

```bash

Simulate a blocked bot


curl -A "AhrefsBot/7.0" -I https://example.com/

↑