GoAdmio

robots

Create (and overwrite) a robots.txt file in the web root of every hosted domain, optionally blocking several SEO and search-tool crawlers

On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also


Overview

A robots.txt file tells search engines and other crawlers which parts of a site they may visit. /go/robots writes one for every hosting account on the server in a single run. For each domain with a /home/<domain>/www folder, it creates /home/<domain>/www/robots.txt.

The command doesn't need sudo. If you aren't root, it runs itself again with sudo.

[!WARNING] The command overwrites any robots.txt that already exists, including ones that your customers or their CMS created. There is no way to skip a domain.


Quick start

# Create the standard robots.txt for all domains
/go/robots

# Also block several SEO crawlers
/go/robots block

Syntax

/go/robots [block]

Options

Argument Required Notes
block No Adds rules that block six crawlers from the whole site.

Any other argument is ignored.


Output

Creating a robots.txt File for example.com...
Creating a robots.txt File for example.org...

There is one line for each domain. The command prints nothing else, and its exit status is 0.

Every run is logged to /var/log/go/robots.log with a timestamp and the arguments used.

The file it writes

Without block:

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

With block, these rules are added after the lines above, one block for each crawler:

User-agent: Amazonbot
Disallow: /

The crawlers are Amazonbot, PetalBot, DataForSeoBot, serpstatbot, dotbot, and SemrushBot.


Examples

# Standard robots.txt for all domains
/go/robots
# Standard file plus the crawler blocks
/go/robots block
# Check the result
curl https://example.com/robots.txt

Troubleshooting

Symptom Likely cause Fix
A customer's own robots.txt was replaced The command overwrites every file. Restore the file from a backup (see restore or snapshot). Back up before you run the command.
The Sitemap: line points to a page that doesn't exist The line is always https://www.<domain>/sitemap.xml, even if the site has no sitemap or doesn't use www. Edit the file by hand.
/admin/ is blocked, but it shouldn't be The Disallow: /admin/ rule is added to every site. Edit the file by hand.
Crawlers still visit after block robots.txt is only a request. Bad crawlers ignore it. Block them at the server with bot or bottop.
The site owner can't edit the file The file is created by root. Run fix to correct the owners and permissions.
robots.txt isn't served The domain's document root isn't the www folder, the CMS answers the request itself, or Cloudflare has an old copy. Check the document root, and clear the cache (see purge).
The bot rules appear without block The server's host name is ns1.mls.eg, ns2.mls.eg, or ns3.mls.eg. See Details.

Details

  1. Reads the server's host name.
  2. Goes through every folder in /home. For each one that has a www folder, it replaces www/robots.txt with the standard rules, using the folder name as the domain.
  3. If you passed block, or the host name is ns1.mls.eg, ns2.mls.eg, or ns3.mls.eg, it adds the crawler rules.

The folder name is used as the domain for the sitemap line, so an account whose folder name differs from its domain gets a wrong address.

Blocking these crawlers also stops the services behind them (SEO tools, Amazon's and Huawei's search crawlers) from indexing the site.

Item Used for
/home/<domain>/www/robots.txt The file that is written.
/etc/hostname The server's host name.
/var/log/go/robots.log The command log.

See also

Command Description
bot Block malicious bots from accessing Apache or Nginx
bots Track all bot visits to your server (good and bad bots)
bottop Block bad bots from visiting your server
fightmode Enable or disable Bot Fight Mode on Cloudflare
fix Fix GoAdmio file and folder permissions
purge Delete the Cloudflare edge cache (purge everything)