Issues with live migrating VM in Hyper-V failover cluster

[spiceuser-hze0m]

Hi

I’m having trouble live migrating a file server that uses a multiple CSV disks, It attempts to migrate, appears to finish, but stays on the same host. It logs this: 1205: The Cluster service failed to bring clustered role ’ FILESERVER ’ completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered role.

1069: Cluster resource 'Virtual Machine Configuration FILESERVER ’ of type ‘Virtual Machine Configuration’ in clustered role ‘FILESERVER’ failed. Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

I’ve done a bit of research but struggling to find a resolution. Can someone point me in the right direction?

[Max-Wozniak]

Where are the CSV disks located?

[spiceuser-hze0m]

Hi - thanks for the reply.

In the cluster storage folder on the C drive of the hosts.

However, I just noticed there is ClusterStorage.000 and ClusterStorage.001 too.

[supaplex-starwind]

Tell us a bit more about the configuration - what kind of storage are you using and how is it connected?

In the meanwhile try migrating the CSVs to the other node first and see if it works. This will ensure your clustered storage is working correctly. Afterward, try migrating the VM once again.

[spiceuser-hze0m]

I have 3 hosts, HV01, HV02 and HV03.

A SAN provides the storage using iSCSI. Each disk is attached to a host and shared in the cluster then allocated to a VM which is also made highly available.

Upon your suggestion, I migrated between other hosts. It appears that FS01 and FS02 (which have 2 CSV disks attached each) will freely migrate between HV02 and HV03, but not HV01.

However, other VMs that only have 1 CSV disk will migrate to HV01.

[supaplex-starwind]

I would recommend you checking the iSCSI connections from HV01 to the SAN, it looks like there is some problem with storage connectivity.

The other question I have is how is the storage connected to a VM. Since you call it “attached” I want to ask if it’s a traditional approach of a VHDX file sitting on top of CSV or some other method of allocating storage to a VM.

[Max-Wozniak]

I would say that one or more of the Volumes in Cluster Storage are not visible to HV01.

If everything is set up correctly then a reboot of the host should fix it

[spiceuser-hze0m]

Hi - sorry, yes is a VHDX file on the CSV disk.

I have tried rebooting, but it does the same thing. I noticed this other event in the logs:

5124: Cluster Shared Volumes root directory ‘C:\ClusterStorage’ already exists. The directory ‘C:\ClusterStorage’ was renamed to ‘C:\ClusterStorage.001’. Please verify that applications accessing data in this location have been updated as necessary.

What would cause it to try and recreate the ClusterStorage and have to rename it? Is this normal behaviour?

I have a SAN team on each host, it appears fine on HV01. Everything pings and the VMs already on the host are working as expected.

[Max-Wozniak]

Please run “Dir C:\ClusterStorage” without the quotes in cmd on all the hosts, post results

[spiceuser-hze0m]

HV01:

C:\Users\administrator>Dir C:\ClusterStorage
Volume in drive C has no label.
Volume Serial Number is 643C-F7CE

Directory of C:\ClusterStorage

03/02/2020 13:23 .
03/02/2020 13:23 …
03/02/2020 07:50 FS01 - E Shares [??\Volume{25c5c968-9647-4fa6-88e2-72cdac7ed3e4}]
03/02/2020 07:50 FS01 - H Users [??\Volume{9050e524-362a-4197-9120-50be55455275}]
03/02/2020 13:23 FS01 - admin Users
03/02/2020 07:52 FS02 - E Admin Shares [??\Volume{d1f70dd8-a4a3-413f-b6ad-a5a6f85cf11a}]
03/02/2020 07:50 FS02 - F Other Users [??\Volume{1e09b89d-a536-4c50-9053-c9e7f1bad7ce}]
03/02/2020 07:50 OS Disks [??\Volume{25cdb193-0a29-481e-97cf-a6b3e0233770}]
03/02/2020 07:50 Spare [??\Volume{d4f0a79e-bda5-4e23-858c-bff9dd030d29}]
0 File(s) 0 bytes
9 Dir(s) 99,653,840,896 bytes free

HV02:

C:\Users\administrator>Dir C:\ClusterStorage
Volume in drive C has no label.
Volume Serial Number is 1CF0-17EB

Directory of C:\ClusterStorage

02/02/2020 15:12 .
02/02/2020 15:12 …
02/02/2020 15:12 FS01 - E Shares [??\Volume{25c5c968-9647-4fa6-88e2-72cdac7ed3e4}]
02/02/2020 15:12 FS01 - H Users [??\Volume{9050e524-362a-4197-9120-50be55455275}]
02/02/2020 15:12 FS01 - admin Users [??\Volume{ca5408de-5b91-4f79-b190-1a83a5bf7253}]
02/02/2020 15:12 FS02 - E Admin Shares [??\Volume{d1f70dd8-a4a3-413f-b6ad-a5a6f85cf11a}]
02/02/2020 15:12 FS02 - F Other Users [??\Volume{1e09b89d-a536-4c50-9053-c9e7f1bad7ce}]
02/02/2020 15:12 OS Disks [??\Volume{25cdb193-0a29-481e-97cf-a6b3e0233770}]
02/02/2020 15:12 Spare [??\Volume{d4f0a79e-bda5-4e23-858c-bff9dd030d29}]
0 File(s) 0 bytes
9 Dir(s) 98,627,358,720 bytes free

HV03:

C:\Users\administrator>Dir C:\ClusterStorage
Volume in drive C has no label.
Volume Serial Number is 5615-9DF2

Directory of C:\ClusterStorage

02/02/2020 18:45 .
02/02/2020 18:45 …
02/02/2020 18:45 FS01 - E Shares [??\Volume{25c5c968-9647-4fa6-88e2-72cdac7ed3e4}]
02/02/2020 18:45 FS01 - H Users [??\Volume{9050e524-362a-4197-9120-50be55455275}]
02/02/2020 18:45 FS01 - adminUsers [??\Volume{ca5408de-5b91-4f79-b190-1a83a5bf7253}]
02/02/2020 18:45 FS02 - E Admin Shares [??\Volume{d1f70dd8-a4a3-413f-b6ad-a5a6f85cf11a}]
02/02/2020 18:45 FS02 - F Other Users [??\Volume{1e09b89d-a536-4c50-9053-c9e7f1bad7ce}]
02/02/2020 18:45 OS Disks [??\Volume{25cdb193-0a29-481e-97cf-a6b3e0233770}]
02/02/2020 18:45 Spare [??\Volume{d4f0a79e-bda5-4e23-858c-bff9dd030d29}]
0 File(s) 0 bytes
9 Dir(s) 43,627,778,048 bytes fre

[Max-Wozniak]

Here’s your issue in HV01

03/02/2020 13:23 FS01 - admin Users

Should be

Migrate everything from this folder and when it’s empty delete and recreate it.

[spiceuser-hze0m]

That’s done the trick! Thanks for all the help!