QuickFix: ESXi 7 and broken vmsyslog

I encountered a situation where ESXi 7 Update 3g (build 20328353) stopped sending logs to a remote Syslog server. Upon further investigation, it turned out that it also stopped writing logs locally, and the logs in /scratch/log are not updated. Free disk space is not a problem.

During diagnostics, errors were detected in the /var/log/.vmsyslogd.err log:

vmsyslog.main            : CRITICAL] Dropping messages due to log stress (qsize = 25000)

I did not find adequate KB for version 7 on this topic, there was only KB for version 6.5/6.7 with a mention of this error, where it was written that “the problem has been fixed”.

The esxcli system syslog config get command correctly displays the status of the syslog settings, but esxcli system syslog reload does not lead to positive results, logs do not start to be written locally, and are not sent to the remote server.

Restarting the service from the host management interface with the Restart button also does not lead to any results. In the log, you can only see:

vmsyslog.main            : ERROR   ] reloading (3200395)

Which is similar to the result of esxcli system syslog reload.

Stopping and restarting the service from the ESXi interface fails because:

This service with 'vmsyslogd' is marked as 'required' and cannot be stopped.

All that remains is to stop it forcibly directly from the host:

ps -cC | grep vmsyslog
3418096  3418096  vmsyslogd             /bin/python /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyc 1

We determine the PID of vmsyslog, in this case 3418096, and kill it:

kill -9 3418096

The vmsyslog will show that the process was killed and then automatically restarted:

vmsyslog.main            : ERROR   ] Watchdog 3418095 fired (child 3418096 died with status 9)!
vmsyslog.main            : ERROR   ] Watchdog 3418095 exiting
vmsyslog                 : CRITICAL] vmsyslogd daemon starting (3418940)

After restarting, logs start to be written locally and sent to the remote server.

Loading

What’s new at Nutanix University? Free NCA 6.10 and NCP-MCI 6.10 beta exams opportunities

Great news from Nutanix University!

Two new beta exams are available for scheduling:
Nutanix Certified Associate 6.10 (NCA 6.10) Beta exam. Blueprint guide;
Nutanix Certified Professional – Multicloud Infrastructure (NCP-MCI 6.10) Beta exam. Blueprint guide.

Both exams are based on the recently released 6.10 LTS version of AOS.

If you don’t know, Nutanix offers beta exams for free, and the first 250 participants can take the exam at no charge! Just use those discounts during checkout:
NCA 6.10 Beta Exam Discount Code: NCA610BETA
NCP-MCI 6.10 Beta Exam Discount Code: NCPMCI610BETA

The main difference between a beta exam and a regular one is that the beta exam is larger. It includes more questions and you need more time to complete it. The time limit for NCA 6.10 beta is 120 minutes and for NCP-MCI 6.10 beta is 180 minutes.

The deadline is November 3, and this is the last day to test.

Also, one thing to mention, is that you will not get your results immediately! Usually, it can take two or even three months after the deadline, so be patient.

Recommended free online courses for preparing:
Nutanix Hybrid Cloud Fundamentals (NHCF) and Enterprise Cloud Administration (ECA).

Let me remind you that on this blog I have two prep guides for both NCA and NCP exams:
Preparing for the Nutanix Certified Associate 6.5 Exam
Preparing for the Nutanix Certified Professional – Multicloud Infrastructure 6.5 Exam

Both guides are based on version 6.5, but if you use the same documentation and courses based on the new version, you will be fine.


Loading

Using Nutanix Prism Central Categories with Veeam Backup 12.2

Categories (in other systems they can be called labels or tags) are used to group entities based on a key-value pairing. For example, we can group a set of VMs and mark them as production environments and apply some policies, while another part of VMs will be marked as tests with a different type of policy applied.

With the release of Veeam Backup 12.2 and the added Prism Central integration, VBR is now aware of Prism Central Categories and can back up all VMs under the specified category in a backup job.

In this article, we will look at how to create a category in Prism Central, how to attach it to the VM, and, sure, how to use VBR with categories.

Continue reading “Using Nutanix Prism Central Categories with Veeam Backup 12.2”

Loading

Veeam Backup 12.2 and Nutanix Prism Central integration

With the release of 12.2, Veeam introduced a new long-awaited feature: Nutanix Prism Central integration, which simplifies the management of large Nutanix clusters, managed by Prism Central instance.

In this article, we will look at how to connect the VBR Server to the Prism Central Instance and perform backup and recovery tasks.

Continue reading “Veeam Backup 12.2 and Nutanix Prism Central integration”

Loading

Deploying Prism Central on Nutanix Community Edition 2.1 using offline bundle

Sometimes deploying Prism Central using an Internet connection is not an option, or a specific version may not be available. In this case, we can download PC deployment files from the Support Portal and deploy it using the Prism Central 1-click deploy from Prism Element method.

In this article, we will look at how to deploy a PC using an offline bundle.

Continue reading “Deploying Prism Central on Nutanix Community Edition 2.1 using offline bundle”

Loading

Deploying the Proxmox Virtual Environment 8 on VMware vSphere for lab purposes

When I look at the virtualization platform, I look not only at the functionality but also at the ecosystem, including backups. When Veeam released version 12.2, where Proxmox support was added, I decided that it was time to learn this platform for a bit.

Previously I didn’t have experience with PVE, and this post is my journey on deploying this virtualization platform for lab purposes.

I considered writing a series of posts but later decided to put all the content into one large post.

In this post, I will cover how to deploy PVE nodes, create a 3-node cluster, connect shared storage, create a VM, and enable high availability.

Continue reading “Deploying the Proxmox Virtual Environment 8 on VMware vSphere for lab purposes”

Loading

Veeam Backup & Replication 12.2 Release

A new Veeam release brings us a lot of new features including:

  1. Long awaited Proxmox VE Support!;
  2. Integration with Nutanix Prism Central;
  3. Support for backup and recovery for MongoDB;
  4. Full vSphere 8.0 Update 3 support;
  5. VeeamONE Proxmox VE, RedHat KVM, and Oracle KVM support;
  6. And more more!

You can check the Release Notes for VBR here and for VeeamONE here.

Loading

Nutanix Kubernetes Engine (NKE): Day-two operations. How to get a Kubeconfig file?

The Kubeconfig file contains information needed to connect to the Kubernetes cluster. In this article, we will look at how to get a Kubeconfig file to connect to the NKE-deployed Kubernetes clusters.

Continue reading “Nutanix Kubernetes Engine (NKE): Day-two operations. How to get a Kubeconfig file?”

Loading

Nutanix Kubernetes Engine (NKE): Day-two operations. Enabling Advanced Kubernetes Management

Advanced Kubernetes Management is a feature of the NKE that provides extra visibility for Kubernetes clusters in the Prism Central interface. It allows you to get information about deployed resources in clusters in a pretty simple manner.

In this article, we will look at how to enable and use the Advanced Kubernetes Management feature.

Continue reading “Nutanix Kubernetes Engine (NKE): Day-two operations. Enabling Advanced Kubernetes Management”

Loading

Nutanix Kubernetes Engine (NKE): Day-two operations. Updating Kubernetes and OS image versions.

This is the third article in the NKE series. Before, I wrote about how to deploy Kubernetes clusters using NKE and how to add or remove workers.

In this article, we will cover update procedures and look at how easily we can update the host OS version and Kubernetes using NKE.

Continue reading “Nutanix Kubernetes Engine (NKE): Day-two operations. Updating Kubernetes and OS image versions.”

Loading