monit alerts not emailed: Resource temporarily unavailable

Monday, April 9th, 2012

While setting up Monit, a tool for easy monitoring of hosts and services, I ran into a problem. I had configured Monit to email alerts to my email address, using my personal mail server (IP/email addresses obfuscated to protect the innocence of my email inbox):

set mailserver 211.211.211.0
set alert ferry.boender@example.com
set httpd port 2812
  allow 0.0.0.0/0.0.0.0

check file test path /tmp/monittest
  if failed permission 644 then alert

After starting it with monit -c ./monitrc, I could reach the webserver at port 2812. I also saw that the check was failing:

# monit -c ./monitrc status
File 'test'
  status                            Permission failed
  monitoring status                 monitored
  permission                        600

However, it was not sending me emails with startup, and status error reports. My server’s mail log showed no incoming mail, making it seem like Monit wasn’t even trying to send email. Turning on Monit’s logging feature, I noticed:

# monit -c ./monitrc quit
# monit -c ./monitrc -l ./monit.log
# tail monit.log
error : 'test' permission test failed for /tmp/monittest -- current permission is 0600
error : Sendmail: error receiving data from the mailserver '211.211.211.0' -- Resource temporarily unavailable

I tried a manual connection to the mail server from the host where Monit was running, and it worked just fine.

The problem turned out to be a connection timeout to the mail server. Most mail servers nowadays wait a certain number of seconds before accepting connections. This reduces the rate at which spam can be delivered. Monit wasn’t waiting long enough before determining that the mail server wasn’t working, and bailed out of reporting errors with a ‘Resource temporarily unavailable‘.

The solution is easy. The set mailserver configuration allows you to specify a timeout:

set mailserver 213.19.146.54 with timeout 30 seconds

I’m happy to report that Monit is now sending email alerts just fine.