Deployed a backup script on a new server, tested it manually — worked fine. Set up crontab, came back next morning, no backup. Turns out /usr/local/bin wasn’t in cron’s PATH so pg_dump just silently didn’t exist.
Switched to absolute path and it worked immediately. Fifteen minutes of debugging for a one-character fix. I keep making this mistake every time I set up a new box. At this point I should just have a checklist taped to my monitor.
hey, at least you caught it within a day.
EDIT : Next time make the cron job run when you’re finished deploying it.
This bit me so many times I have
# USE ABSOLUTE PATHS, ASSUME ALL COMMANDS ARE UNDEFINED, SPECIFY ALL VERSIONS, REDIRECT OUTPUTat the top of my local crontab so I remember from when I take something from it to prod, lol.
Edit: formatting whoopsie
When will this be part of the packaged default? :)
That’s one reason why I like using systemd timers instead of cronjobs. You can easily test it by running the unit instead of setting the job to “in a minute”.
Downside is that you don’t easily get notification mails for any output.
I use healthchecks + runitor for notifications. Would recommend. I posted another comment about them.
I’d recommend self-hosting healthchecks and using runitor for your cronjobs. Runitor pings healthchecks when the cronjob starts, then pings it again on completion with the status (success or fail) along with the stdout and stderr.
Healthchecks can be configured to expect a ping periodically (once per day, once per hour, whatever) and alert you if it doesn’t receive one.
For backups, Borgmatic has a healthchecks integration.
Looks like you need to collect your crontab logs somewhere you can read.
You can append
2>&1 | logger -t my-cronjobto any command, and it will write logs to system journal which you can view with
journalctl
