en.vmik.net

I encountered a situation where ESXi 7 Update 3g (build 20328353) stopped sending logs to a remote Syslog server. Upon further investigation, it turned out that it also stopped writing logs locally, and the logs in /scratch/log are not updated. Free disk space is not a problem.

During diagnostics, errors were detected in the /var/log/.vmsyslogd.err log:

vmsyslog.main            : CRITICAL] Dropping messages due to log stress (qsize = 25000)

I did not find adequate KB for version 7 on this topic, there was only KB for version 6.5/6.7 with a mention of this error, where it was written that “the problem has been fixed”.

The esxcli system syslog config get command correctly displays the status of the syslog settings, but esxcli system syslog reload does not lead to positive results, logs do not start to be written locally, and are not sent to the remote server.

Restarting the service from the host management interface with the Restart button also does not lead to any results. In the log, you can only see:

vmsyslog.main            : ERROR   ] reloading (3200395)

Which is similar to the result of esxcli system syslog reload.

Stopping and restarting the service from the ESXi interface fails because:

This service with 'vmsyslogd' is marked as 'required' and cannot be stopped.

All that remains is to stop it forcibly directly from the host:

ps -cC | grep vmsyslog
3418096  3418096  vmsyslogd             /bin/python /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyc 1

We determine the PID of vmsyslog, in this case 3418096, and kill it:

kill -9 3418096

The vmsyslog will show that the process was killed and then automatically restarted:

vmsyslog.main            : ERROR   ] Watchdog 3418095 fired (child 3418096 died with status 9)!
vmsyslog.main            : ERROR   ] Watchdog 3418095 exiting
vmsyslog                 : CRITICAL] vmsyslogd daemon starting (3418940)

After restarting, logs start to be written locally and sent to the remote server.