GoAdmio

heal

Monitor key services, memory, load, and disk, and automatically fix problems (Auto-Heal)

On this page: Overview · Quick start · Syntax · Options · Output · Examples · Troubleshooting · Details · See also


Overview

/go/heal is GoAdmio's Auto-Heal system. Each run checks the services and resources a hosting server depends on. When it finds a problem, it tries to fix it and sends you an email and a WhatsApp message.

It checks:

The command is meant to run on a schedule, for example from cron (see cron). You can switch Auto-Heal on and off with --enable and --disable.

[!WARNING] If the web server stays down after a restart, Auto-Heal reboots the server. See Web server check.


Quick start

# Run all checks now
/go/heal

# Turn Auto-Heal off or on
/go/heal --disable
/go/heal --enable

Syntax

/go/heal [--enable | --disable]

Options

Option Required Default Notes
--enable No — Turns Auto-Heal on and exits.
--disable No — Turns Auto-Heal off and exits. Later runs do nothing until you enable it again.

The setting is stored in config.db. With no options, the command runs every check, unless Auto-Heal is disabled.

The disk check uses a threshold of 90% by default. To change it, put a number in /etc/disk:

echo 80 | sudo tee /etc/disk

If the file doesn't hold a plain number, 90 is used.


Output

The command prints one line for each check as it runs.

Checking MySQL Status...
Checking Nginx Status...
Checking FPM Status...
Checking SSHD Status...
Checking DROPBEAR Status...
Checking POSTFIX Status...
Checking DOVECOT Status...
Checking Memory Status...
Checking Disk Usage Status...
Checking GoAdmio Usernames...
Checking GoAdmio Databases Ownership...

When it takes action, it also prints a line such as Restarting MariaDB..., Restarting php83-fpm..., Installing SSH..., or Rebooting server....

Outcome What you get
--enable Auto-Heal has been enabled. Exit status 1.
--disable Auto-Heal has been disabled. Exit status 1.
Auto-Heal is off Auto-Heal is disabled. Exit status 1.
MinIO is configured on port 443 No output, no checks. Exit status 1.
Full run The check lines above, then the result of /go/welcome.

Every run is logged to /var/log/go/heal.log with a timestamp and the arguments used.


Examples

# Run all checks now
/go/heal
# Pause Auto-Heal while you do maintenance
/go/heal --disable

# ...and turn it back on afterwards
/go/heal --enable
# Alert and clean up when the disk is 80% full instead of 90%
echo 80 | sudo tee /etc/disk

Troubleshooting

Symptom Likely cause Fix
Auto-Heal is disabled. It was turned off with --disable. Run /go/heal --enable.
The command exits at once with no output MinIO is configured on port 443 (:443 in minio.conf). This is by design. Auto-Heal doesn't run on such servers.
The server rebooted by itself The web server check failed twice, so Auto-Heal restarted the web server and then rebooted. Make sure http://<server IP>:2020/conf.php works from the server itself. Check the firewall with checkfirewall. Run --disable while you investigate.
The web server check never runs The server's public IPv4 address couldn't be detected, for example because checkip.amazonaws.com is unreachable. Check outbound internet access.
Repeated "Restarting MariaDB" messages The MySQL root password in config.php is missing or wrong, so the check always fails. Fix the root password. See mysqlset.
Dropbear was installed on a server that didn't have it Auto-Heal treats a missing Dropbear service as a failure and installs it. See SSH and Dropbear.
Disk alert on every run Usage is still at or above the threshold. Free up space, or raise the threshold in /etc/disk.
Memory alert with an empty message The message text is only filled in when the server has 1500 MB of RAM or less. The usage percentage is in the subject line.
No email or WhatsApp messages The mail or WhatsApp settings are incomplete. Check the admin email, mobile, and WhatsApp settings, and the mail server itself.
"User go or admin does not exist" every run Only the go user is created by this check. Create the missing user by hand.

Details

How it works

flowchart TD
    A["/go/heal"] --> B{"--enable or --disable?"}
    B -- "Yes" --> C["Save the setting and exit"]
    B -- "No" --> D{"Auto-Heal enabled?"}
    D -- "No" --> E["Exit"]
    D -- "Yes" --> F{"MinIO on port 443?"}
    F -- "Yes" --> E
    F -- "No" --> G["MariaDB"]
    G --> H["Web server"]
    H --> I["PHP-FPM"]
    I --> J["SSH, Dropbear, Postfix, Dovecot"]
    J --> K["Memory, load, disk"]
    K --> L["Users and database ownership"]
    L --> M["/go/welcome"]

Checks and actions

Check Trigger Action Repeats
MariaDB The root user can't read mysql.user. Restarts MariaDB. If the error mentions mysql.sock, deletes the socket file and restarts again. Every run
Web server The panel page conf.php doesn't return 1, twice. Runs conf --installwebserver. If it is still down, reboots the server. Once per 6 hours
PHP-FPM A PHP-FPM version reports (dead). Restarts that version. Once per 6 hours per version
SSH sshd is dead, missing, or changed on disk. Restarts and enables it. If it still fails, runs ssh, then installssh. Every run
Dropbear dropbear is dead, missing, or changed on disk. Restarts and enables it. If it still fails, runs dropbear. Every run
Postfix postfix is dead. Runs /go/postconf, then restarts and enables Postfix. Every run
Dovecot dovecot is dead or reports a fatal error. Runs doveconf, then restarts and enables Dovecot. Every run
Memory More than 90% of RAM is in use. Runs fpm reload. Once per 6 hours
Server load The 1-minute load average is more than 200% of the CPU threads (for example 4.0 on 2 threads). Emails the top memory consumers and runs ddos block. Once per 6 hours
Disk Usage of / is at or above the threshold (default 90%). Runs clean and /go/delcache all. Every run
System users The go or admin user is missing. Runs /go/adduser go. Every run
Database ownership A file under /var/lib/sqlite isn't owned by go:go. Runs chown -R go:go /var/lib/sqlite. Every run

Every action except the user and ownership fixes is reported by email and WhatsApp with a subject starting with GoAdmio Auto-Heal System.

The 6-hour limit

To avoid repeating the same fix and the same alert, some actions leave a marker file in /usr/local/goadmio/tmp named after the check and the current period. The day is split into four periods by server time: 00:00–05:59, 06:00–11:59, 12:00–17:59, and 18:00–23:59. A marked action runs at most once per period. The script doesn't delete the marker files.

Web server check

The check requests http://<server IP>:2020/conf.php, which is the GoAdmio panel, and expects the answer 1. If it doesn't get it, it waits 15 seconds and tries again. If it still fails, it:

  1. sends an alert and runs conf --installwebserver
  2. waits 30 seconds, then checks again (with one more try after 15 seconds)
  3. if the panel still doesn't answer, sends an alert and runs reboot

The check connects to the server's own public IP, so it also fails if that address can't be reached from the server itself.

SSH and Dropbear

If SSH keeps failing after a restart, Auto-Heal rewrites its configuration with ssh and, as a last step, reinstalls it with installssh. In that case it sends a longer "Resolution Initiated" message to the administrator.

The Dropbear check works the same way. A server without Dropbear matches the "could not be found" condition, so Auto-Heal installs it. If you don't want Dropbear, you'll need to change the script or keep Auto-Heal disabled.

Files and requirements

Item Used for
/var/lib/sqlite/config.db The Auto-Heal on/off setting and the admin contact details.
/usr/local/goadmio/include/config/config.php The MariaDB root password.
/etc/disk The optional disk threshold.
/usr/local/goadmio/tmp The marker files.
/var/log/go/heal.log The command log.
/var/log/go/cron/ Output of installssh and dropbear when they run.
/go/sendmail, /go/sendwhatsapp Email and WhatsApp alerts.

Security notes


See also

Command Description
status Collect the status of every systemd service
help AI solution for auto-detecting and solving GoAdmio server problems
support AI support solution for GoAdmio server problems
troubleshoot Troubleshoot email problems in GoAdmio
conf Rebuild the Apache/Nginx configuration in GoAdmio
fpm Restart all installed PHP-FPM versions on your server
clean Clean and remove unnecessary files to free up space
ddos Track and block DDoS attack IPs in GoAdmio
mem Check the top ten memory-consuming commands or services
cron Set up main cron jobs in GoAdmio