↓ Skip to main content
Clouds and Unicorns
VCF Automation ran out of disk space? Fleet Manager's got you.
  1. Posts/

VCF Automation ran out of disk space? Fleet Manager's got you.

·3 mins·
Blog Posts Vcf Vcf-Automation Vra Vcf-Automation Postgres Storage Kubernetes

My life is full of equally true statements:

  • I have VCF Operations monitoring everything.
  • I have VCF Operations sending me Slack notifications.
  • Things go wrong in my lab all the time.
  • I ignore the alarms and notifications.
  • I wonder why things suddenly break just before a demo.

This post isn’t here to help you learn from my lessons to pay attention to your alarms, but it’s here on the assumption that you are like me, and you need to know how to get out of the pickle your life choices have gotten you in! It all started the day before I needed to do an automation demo:

Oh no

I resorted to rebooting, and then rebooting again, to no avail. I ended up using a different approach for my demo and a week or two later, I decided to try and fix it.

What was actually broken
#

The alert in question

It turns out that there was a space alarm that I ignored. I SSH’d into the VCF host and sure enough there was a full volume.

Oh no, a volume is full.

Not knowing what volume this was for, I did what any enterprising admin would do – I poked my nose where it doesn’t belong.

A database mount point

Upon discovering that this was the mount point for the PostgreSQL instance that serves VCFA, it was clear why things were unhealthy.

Doing some poking around, I didn’t see any obvious temporary files or other easy space reclamation targets. Rather than poking blindly, I managed to consult the oracle (ChatGPT), who recommended that I try to expand the PV. Its first recommendation was to check and see if the PV would autogrow if we expanded the disk:

Auto Expansion is on!

This was good news! It meant that as soon as I expanded the VMDK for the volume, it would expand and the database could recover.

Fleet Manager to the rescue
#

Fleet Manager has a resize option
I knew from my experience with Aria Suite Lifecycle Manager (the predecessor to Fleet Manager) that there were often day 2 capabilities to resize appliances. As it turns out, Fleet Manager carries on the tradition – inside the Lifecycle section of VCF Operations there is a resize option: Fleet Management -> Lifecycle -> Components -> (select VCF Automation) -> Storage Resize.
Storage resize UI

It shows you the details of all the volume groups and allows you to update their capacity, and once you tell it to go – under the hood it drives the PVC expansion through the supported channel, including whatever reconciliation it needs to do in the operator stack on the way.

Nature heals
#

Resize in progress
Once I kicked the resize off in Fleet Management, giving it 60 GB, I got to watch it work: the VMDK grew, the appliance detected the additional space, and finally updated the PV to the total size!

Back in business

At this point, I thought everything was great. Until I discovered a second error that had been both hidden and triggered by the database failure. I’ll save that for my next post, which is equal parts interesting and terrifying. Stay tuned.

Related

Reset the VCF NSX svc Account

·2 mins
Blog Posts Vcf Nsx
Last week, I ran into an issue in my lab when trying to commission additional hosts. After some digging, I realized that SDDC Manager was having trouble communicating with the NSX Manager for my management domain. In VCF Operations, my Fleet Management tasks were filled with messages indicating that the connection to NSX had been lost.

Day Two Automation Self Service Framework – Part 1

·3 mins
Blog Posts Examples Vra Vrops Vro Vra Vrops
In my years on the administration and engineering side of the VMware world a task I was frequently responsible for was creating policies around the standards of my environment and then defining a process to ensure those policies were adhered to. Some examples of the policies I authored was things around snapshot lifetime, security hardening or datastore freespace, etc. Often times these standards conflicted with the pressures I faced to complete requests in a timely fashion on the requests faced by my team and I. When faced with conflicting pressures like this you can chose to not enforce your established polices, which can have significant negative impacts when your team is audited or you suffer a negative impact to your environment as the result of not enforcing your established policies; or you can choose to not deliver the requested service in a timely fashion, which will likely have equally bad outcomes. In this multi-part blog series we will explore one path mitigating the conflicting pressures by utilizing vROPS to monitor for violations of a given policy, and when detected performing automatic remediation of policy violations, while allowing a for exceptions. We will close with exploring how we can build upon this monitoring framework to deliver self-service capabilities of day two actions.

Claude Commissions a VCF Host

·8 mins
Blog Posts Homelab Ai
Like many others, I jumped on the MinisForum MS-A2 train a few months ago. I swapped out my Dell r740xd which was a single host management domain, with 3x MS-A2s. In general, these have been performing really well. When I put a host into maintenance mode, things can get a little slow, but I’ve been really happy with the change, especially with the reduction in power consumption.