1

I’m having some trouble trying to sort out an issue with an ESXi server for a client. Everything I’ve found so far seems to indicate the system disk is completely full but I’m not 100% sure, and I don’t know what else to try at this stage. We can no longer login to either the web interface or the vSphere client using the root account, any other credentials also don’t work. If we try the web interface it accepts the credentials then immediately logs you out saying the session expired. If you try the vSphere client it says the account doesn’t have permission to log in, but I’m using the root account.

It has SSH available, and I can log in as root there but I can’t enable the shell using “shell.set --enabled True”, I get an error saying " Unknown command: `shell.set’ ". I’m not terribly experienced with ESXi so I’ve just been combing through documentation from vmware and googling things up to this point. I’ve yet to try and reboot the server because the client is using the VM’s on it still and I honestly don’t know if they’re configured to auto-start on a reboot, or if trying to reboot it through SSH will actually do a graceful shut down of the guest vm’s. I don’t know the last time this server was restarted or even a total list of all the VM’s that reside on the server as we just sort of inherited this as the previous support person wasn’t doing anything as far as maintenance/documentation goes.

Has anyone run into this sort of situation before, can you recommend a next step or am I way off base with how I’ve approached this? Thanks.

3 Spice ups
2

Can you get at the console? Can you enable shell from their?

Once in you can then check the disk space

df -h

3

I’m working remotely so not at the same location, but SSH just goes into the regular console and I can’t seem to enable the shell from there. I think the only option is gonna be to go on site the more I think about this.

4

What VMware version and edition ??

If you are using viClient, that means pre-6.7 ?? What is the vCenter running on or is it a VCSA ?

Any “thin provisioning” ?

5

What do you mean by ssh going into regular console?

If you can log in via ssh, there is no need to go to the server site. Just log in using ssh and check storage usage with “df -h” command.

AFAIK “shell.set” command only applies to vCenter Server Appliance, not ESXi hosts.

6

When I log in via SSH it gives me just basic commands, and is missing a lot of the standard linux ones like df, du, etc. I’m pretty unfamiliar with the VMware server stuff and terminology for ESXi / vCenter and whatnot so I may have mis-communicated what I’m actually connecting to. I just checked again and when trying to log in through the web portal it does indeed say vCenter Server Appliance so I guess this is different than ESXi which may explain why I’ve had so much trouble getting to the bottom of this…

7

I’m not sure exactly what version they’re using yet because this client has no prior documentation and I’m basically trying to discover everything and try to document as I go. They have another ESXi server running 5.5.0 so I would guess this one is probably similar or at least close in version number, definitely pre-6.7 anyway. I found out I was using the wrong terminology so this is actually a VCSA I’m trying to connect to, and I have no clue on how provisioning was configured, other than they have a QNAP NAS set up with iSCSI to store all the VM’s on.

8

ESXi - Host OS for the physical tin

vCentre - Windows central management tool for ESXi hosts

VCSA - Virtual appliance, replaces the above for ESXi hosts.

To get in to SSH for VCSA you first need to enable it.

https://yourVCSA:5480 - from here you can enable SSH

9

I can already log in via SSH to the VCSA but shell isn’t available (so just the limited console commands work, and shell.set isn’t working), and logging into the web interface via port 5480 immediately logs me out with a session expired error. Assuming I’ll need to do an onsite to sort this out at this stage.

10

Have you actually tried rebooting the VCSA, any temp logs and such will purge which may give you enough space to work on it.

It will not stop the guests or hosts from working while you do this either

1 Spice up
11

I haven’t tried rebooting it because I assumed that would stop the hosts. If it’s safe to reboot it without affecting the others then I’ll do that and hope it rotates / fixes the logs if that is causing the disk usage issue.

12

VCSA is purely a central management tool, rebooting it will postpone any changes and will stop DRS and HA working for the duration, but all hosts store a local copy of the configuration to work while the central management tool is not there.

13

Through some investigation and luck I was able to get access to one of the ESXi servers where the VCSA happened to be running (not sure why the client had set up in this way but w/e), so I was able to reboot it into bash, clear the log, found the root user password had expired causing the whole thing. Now I just have to fix the issues with 10+ other services not starting now, but at least it’s functional.