robots
Create (and overwrite) a robots.txt file in the web root of every hosted domain, optionally blocking several SEO and search-tool crawlers
- Command:
/go/robots - Category: Websites
- Required option: None
- Output: One line per domain
- Last updated: 2026-11-20
On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also
Overview
A robots.txt file tells search engines and other crawlers which parts of a site they may visit. /go/robots writes one for every hosting account on the server in a single run. For each domain with a /home/<domain>/www folder, it creates /home/<domain>/www/robots.txt.
The command doesn't need sudo. If you aren't root, it runs itself again with sudo.
[!WARNING] The command overwrites any
robots.txtthat already exists, including ones that your customers or their CMS created. There is no way to skip a domain.
Quick start
# Create the standard robots.txt for all domains
/go/robots
# Also block several SEO crawlers
/go/robots block
Syntax
/go/robots [block]
Options
| Argument | Required | Notes |
|---|---|---|
block |
No | Adds rules that block six crawlers from the whole site. |
Any other argument is ignored.
Output
Creating a robots.txt File for example.com...
Creating a robots.txt File for example.org...
There is one line for each domain. The command prints nothing else, and its exit status is 0.
Every run is logged to /var/log/go/robots.log with a timestamp and the arguments used.
The file it writes
Without block:
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
With block, these rules are added after the lines above, one block for each crawler:
User-agent: Amazonbot
Disallow: /
The crawlers are Amazonbot, PetalBot, DataForSeoBot, serpstatbot, dotbot, and SemrushBot.
Examples
# Standard robots.txt for all domains
/go/robots
# Standard file plus the crawler blocks
/go/robots block
# Check the result
curl https://example.com/robots.txt
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
A customer's own robots.txt was replaced |
The command overwrites every file. | Restore the file from a backup (see restore or snapshot). Back up before you run the command. |
The Sitemap: line points to a page that doesn't exist |
The line is always https://www.<domain>/sitemap.xml, even if the site has no sitemap or doesn't use www. |
Edit the file by hand. |
/admin/ is blocked, but it shouldn't be |
The Disallow: /admin/ rule is added to every site. |
Edit the file by hand. |
Crawlers still visit after block |
robots.txt is only a request. Bad crawlers ignore it. |
Block them at the server with bot or bottop. |
| The site owner can't edit the file | The file is created by root. | Run fix to correct the owners and permissions. |
robots.txt isn't served |
The domain's document root isn't the www folder, the CMS answers the request itself, or Cloudflare has an old copy. |
Check the document root, and clear the cache (see purge). |
The bot rules appear without block |
The server's host name is ns1.mls.eg, ns2.mls.eg, or ns3.mls.eg. |
See Details. |
Details
- Reads the server's host name.
- Goes through every folder in
/home. For each one that has awwwfolder, it replaceswww/robots.txtwith the standard rules, using the folder name as the domain. - If you passed
block, or the host name isns1.mls.eg,ns2.mls.eg, orns3.mls.eg, it adds the crawler rules.
The folder name is used as the domain for the sitemap line, so an account whose folder name differs from its domain gets a wrong address.
Blocking these crawlers also stops the services behind them (SEO tools, Amazon's and Huawei's search crawlers) from indexing the site.
| Item | Used for |
|---|---|
/home/<domain>/www/robots.txt |
The file that is written. |
/etc/hostname |
The server's host name. |
/var/log/go/robots.log |
The command log. |
See also
| Command | Description |
|---|---|
bot |
Block malicious bots from accessing Apache or Nginx |
bots |
Track all bot visits to your server (good and bad bots) |
bottop |
Block bad bots from visiting your server |
fightmode |
Enable or disable Bot Fight Mode on Cloudflare |
fix |
Fix GoAdmio file and folder permissions |
purge |
Delete the Cloudflare edge cache (purge everything) |