HyperV 2019 VMs 2x slower than HV 2012 host VMs

da-schmoo · 2021-08-20T17:17:33.000Z

Jason1121:

So the Passmark score increased but as far as you could tell the VM itself wasn’t necessarily running any different? Clicking around and doing things inside the VM felt the same to you?

It was the only host I had available that I could reboot. Only workload is a RDS server which was sitting idle at the moment and some Hyper-V replicas. Workload really shouldn’t be different before and after.

artrix · 2021-08-20T20:06:33.000Z

Not sure if you meant me? Yes the VMs did feel a tiny bit laggy on 2019 vs 2012. Its subtle as computers are so fast that even if you slow them down by 1/2 the windows experience from a user perspective is still usable. It when you get into a situation where a application that use to take 1 hour to complete now takes 2 hours is where you notice it.

artrix · 2021-08-20T20:31:21.000Z

Chris-Kelly:

Hyper-V on server 2019 uses the core scheduler by default, which may have an impact on VM CPU performance. Perhaps try switching to the classic scheduler to see if that alleviates your issue.

Manage Hyper-V hypervisor scheduler types | Microsoft Learn

Nice catch there. That sounds like it might be the issue. Oh glad I imaged with Macrium Reflect can revert back pretty fast to verify.

What you just linked is pretty big I think. I’ve read quite a few posts on 2019 and many users are saying they have to toss 2x the virtual cpus and ram to get them to be like 2016 but they never really said why. I’m wondering if this is not the reason for that. People just threw more Vcpus at the issue. Not sure about the ram issue though.

From what I have read 2012 and 2016 are both fast no issue or tweaks needed. People complain that 2016 had slow updates. Not sure if they mean just the initial installation or ALL updates even the ones ones in 2021.

2019 issues seem to be many. And people have actually given up and downgraded back to 2016. All of this sort of scares me off from 2019 as it looks like it might create too much WORK tracking down all the issue. People that start off with 2019 might never know because how many are going to downgrade to 2012 to see if that performs better? not too many. They just plug and play.

Slow VMs (you might have found the fix)
Slow SMB read spreads
other networking slowness

“From another thread”

We’ve seen ~ -50% performance degradation when moving workloads from W2012DC to W2019DC. The W2019 latencies and 3x9’s latencies are off the charts poor.
We’ve seen ~ -(8-9%) performance degradation W2012 and W2016 with W2012 being the best performing of the 3.

artrix · 2021-08-20T20:48:19.000Z

Da_Schmoo:

I ran a test before and after switching the core scheduler. After the switch, the Passmark CPU score increased by 70% in the guest. (Passmark CPU only test - not the full suite) Negligible change testing the host.

I made no changes to the VM itself.

Thanks for doing that!

I’m thinking there has to be many people using 2019 that are running 2x slower VMs and not realizing it.

da-schmoo · 2021-08-20T20:53:22.000Z

RAJOD:

Da_Schmoo:

I ran a test before and after switching the core scheduler. After the switch, the Passmark CPU score increased by 70% in the guest. (Passmark CPU only test - not the full suite) Negligible change testing the host.

I made no changes to the VM itself.

Thanks for doing that!

I’m thinking there has to be many people using 2019 that are running 2x slower VMs and not realizing it.

This needs to be tested in a real life scenario, not just relying on a benchmarking app though.

artrix · 2021-08-20T22:16:20.000Z

Da_Schmoo:

RAJOD:

Da_Schmoo:

I ran a test before and after switching the core scheduler. After the switch, the Passmark CPU score increased by 70% in the guest. (Passmark CPU only test - not the full suite) Negligible change testing the host.

I made no changes to the VM itself.

Thanks for doing that!

I’m thinking there has to be many people using 2019 that are running 2x slower VMs and not realizing it.

This needs to be tested in a real life scenario, not just relying on a benchmarking app though.

Oh I will be doing that too, this is the road to that. Thanks for helping.

tim7139 · 2021-08-20T23:22:24.000Z

From a workload point of view I have not noticed a decrease in performance moving from 2016 to 2019. I did find there seemed to be more adjustments needed to drivers and settings, and I observed benchmarks that didn’t match real world testing as closely (specifically disk and network).

artrix · 2021-08-21T02:12:00.000Z

Da_Schmoo:

I ran a test before and after switching the core scheduler. After the switch, the Passmark CPU score increased by 70% in the guest. (Passmark CPU only test - not the full suite) Negligible change testing the host.

I made no changes to the VM itself.

Ok I wiped the system and put 2019 back on.

bcdedit /set hypervisorschedulertype Classic (reboot)

Yes much better now. Funny neither 2016 or 2019 have near the single thread performance of 2012

2012 - Single thread - 1340

2016 - 932

2019 - (HV type 3 core) 655

2019 - (HV type 2 classic) 995

Anyway it puts 2019 more in line with 2016 in terms of VM cpu performance.

The over all cpu mark is actually a little better than 2016 after the HV change.

2012 - 5444

2016 - 5579

2019 - 2329 (default)

2019 - 5627 (after change to HV type 2 scheduler or classic)

chris-kelly · 2021-08-21T12:48:23.000Z

Is the VM you’re using to test a freshly created VM on 2019, or was it restored/migrated from an earlier version Hyper-V configuration? If you’re seeing the performance corrected by using the classic scheduler, and if the latter is true, check to see if the VM configuration is using the updated settings for the core scheduler.

The value of the following powershell command should be “0” for the VM:

Get-VM | Get-VMProcessor | FT VMName,HwThreadCountPerCore

If it’s “1”, that might also reduce the available processing grunt the VM has when using the core scheduler, since it would ignore hyperthreaded cores assigned to the VM.

Source : Windows Server 2019 Hyper-V might only use half the available CPU’s? – Kevin Holman's Blog

*Edit: Original Microsoft Docs article linked in source article: About Hyper-V hypervisor scheduler type selection | Microsoft Learn *

Strangely the core scheduler change wasn’t included in either of the 2016 or 2019 “what’s new in hyper-v” article on Microsoft docs, I just remember seeing it in a compatibility matrix on some other Microsoft article somewhere, which I seem to have lost the link to

gregghill · 2021-08-21T17:14:39.000Z

I have yet to install Server 2019 as the Hyper-V host, but I have read multiple threads here and elsewhere about how slow it is, so I am bookmarking this page!

Gregg

da-schmoo · 2021-08-21T17:23:34.000Z

I wonder if this might be tied in with the network slowness people have been noticing.

@mysticfoxde

artrix · 2021-08-21T20:35:02.000Z

Chris-Kelly:

Is the VM you’re using to test a freshly created VM on 2019, or was it restored/migrated from an earlier version Hyper-V configuration? If you’re seeing the performance corrected by using the classic scheduler, and if the latter is true, check to see if the VM configuration is using the updated settings for the core scheduler.

I tried it both ways. New VM and imported VMs no change in benchmarks.
I have till 10 2023 plenty of time to iron this out. Right now everything on the domain been smooth sailing for years. If its not broken i’m not in a hurry to fix it and open a new can O worms plus 2022 might be worth a look.

I’ll try your suggestion thanks.

BTW Eric Siron has a pretty good write up on this.

Altaro DOJO | Hyper-V – 25 Jul 19

What is the Hyper-V Core Scheduler?

The core scheduler provides a strong inter-virtual machine barrier against cache side-channel attacks eg Spectre variants. This guide explains how to use it

mysticfoxde · 2021-08-22T19:46:15.000Z

Hi Da_Schmoo,

Da_Schmoo:

I wonder if this might be tied in with the network slowness people have been noticing.

Thanks for the hint.
This sounded very interesting to me at the beginning, but unfortunately after a few tests I have to say that the change to the core scheduler is also performance-relevant for one or the other, but it is not decisive for the network problems discussed in the other post. Regarding the network problem (and also other problems), however, I have made very decisive advances in the last few weeks and am in close contact with Microsoft regarding these “discoveries”.

Please keep your fingers crossed for me, then the windows performance things might get a little better soon.

Best Regards from Germany
Alex

@da-schmoo

supaplex-starwind · 2021-08-24T04:57:51.000Z

I have never used CPU benchmarking for Hyper-V virtual machines. Pretty curious about the outcomes of Windows Server 2022.

There is a 20%-30% impact on storage performance in Windows Server 2019 with the latest updates. A pure installation without updates performs much better.

Networking is another complicated story that differs depending on the network adapter vendor and driver version.

In addition to the scheduler, what I have noticed, though, is that incorrect vCPU configuration may dramatically impact the performance of the virtual machine https://www.hyper-v.io/several-tips-hints-full-throttle-hyper-v-performance/ . Playing around the total number of vCPUs, NUMA configuration, and resource control may significantly boost the performance, especially if multiple virtual machines are benchmarked simultaneously.

artrix · 2021-08-31T22:55:34.000Z

Supaplex:

I have never used CPU benchmarking for Hyper-V virtual machines. Pretty curious about the outcomes of Windows Server 2022.

There is a 20%-30% impact on storage performance in Windows Server 2019 with the latest updates. A pure installation without updates performs much better.

Networking is another complicated story that differs depending on the network adapter vendor and driver version.

In addition to the scheduler, what I have noticed, though, is that incorrect vCPU configuration may dramatically impact the performance of the virtual machine https://www.hyper-v.io/several-tips-hints-full-throttle-hyper-v-performance/ . Playing around the total number of vCPUs, NUMA configuration, and resource control may significantly boost the performance, especially if multiple virtual machines are benchmarked simultaneously.

30 percent disk performance hit on just a patch? That is on top of of the meltdown/spectre slowdowns. I think PC performance it moving in the wrong direction.

mysticfoxde · 2021-09-01T04:24:03.000Z

RAJOD:

I think PC performance it moving in the wrong direction.

I have to confirm that, unfortunately. &

The real system performance (“end user feeling”) of windows systems, changes compared to the information in the data sheets of the hardware, felt by 180 degrees in the opposite direction.

@artrix

supaplex-starwind · 2021-09-05T16:05:37.000Z

I assume it is because hardware improves faster than software, and most of us are running systems that are vastly overprovisioned in terms of performance utilization. So why care about writing an optimized code if there is plenty of resources anyway?

mysticfoxde · 2021-09-05T16:39:00.000Z

Hi Supaplex,

Supaplex:

I assume it is because hardware improves faster than software, and most of us are running systems that are vastly overprovisioned in terms of performance utilization. So why care about writing an optimized code if there is plenty of resources anyway?

I’m a little further on with regard to the problem and yes, the hardware is definitely controlled / addressed by the software in places that are suboptimal.

If everything goes well, I will explain the reason for the CPU load during the RDMA transfer in more detail today or tomorrow.

Best Regards from Germany
Alex

@supaplex

mysticfoxde · 2021-09-05T19:02:49.000Z

Hi Friends,

I will probably not get a detailed description today.

Nevertheless, I would like to give a little tip this evening.

If you want to know where a lot of the performance of your server 2019 will go over the Jordan, you should take a closer look at the following counters on the corresponding systems with performance monitoring.

\Memory\Cache Faults/sec

\Memory\Page Faults/sec

\Memory\Demand Zero Faults/sec

\Memory\Nullvorderungsfehler/s (I couldn’t find it in English)

\Memory\Free System Page Table Entries

\System\Context Switches/sec

\Processor(_Total)\Interrupts/sec

b156b6d9-f057-4133-bdf5-c527cfc1891e-PERFCOUNTER.jpg800×450 94.6 KB

The picture above is for information only and doesn’t show anything conspicuous.

Think of it right now as a kind of “calm before the storm” and take a look at the state of your own systems.

Incidentally, the problem is not only seen with Server 2019, you can also call up the same counters on your Windows 10 or 11 clients for the snake of fun.

Well, I have to take back the “for the sake of fun”, if someone react too (justified) aggressive about sennless performanceloss , I would urgend recommend,
not to look to the performance counter, without to have a professional moral or psychological support on your side.

Best Regards from Germany

Alex

@tim7139 @gregghill @da-schmoo @artrix @supaplex

artrix · 2021-09-07T09:25:02.000Z

Oh there is no shortage of inefficient code. Hardware has hit a wall in past 6 years computers are not getting much faster at single threaded apps in a long time.

We have moved sideways with more cores is all.

This was more about loosing performance in software by 2x not 2 percent.

You had a game that played at 60 fps now 30 fps. You cant buy a faster video card because you already have the fastest. Hardware cant fix sloppy software.

People just get use to the new normal of slowness. 30 fps is the new 60 fps.