I worked one particular contract where I was forced to take my lunch at 11:35 every day, and it was all virtualisations fault!

To set the scene it was a company who wasn’t really used to having developers, they had a load of SQL analysts and some mainframe developers but SQL developers writing T-SQL, C# and SSIS code was new to them. The IT management had decided that buying actual computers wasn’t necessary for development. We could use the standard PC’s and RDP onto a Windows 2008 R2 server with Visual Studio installed and work on that.

I know it doesn’t sound ideal but it was ok, the project was exciting, it even involved putting in a continuous deployment pipeline by accident, but we will leave that story for another day.

The setup worked pretty well but for some reason, every day at 11:35, the CPU of my machine went crazy for about twenty minutes. This happened every single day, so after a while, I asked around, and other people were seeing it but being honest I think they were probably used to taking an early lunchtime or maybe it was a late elevensies :).

When something like this happens, I normally break out task manager or “task mangler” as it is affectionately known in certain circles. The thing using the CPU was always the same it was the anti-virus updater. I don’t mean an anti-virus scan but just the check for an updated definition.

I don’t like this sort of thing, so I spent some time trying to work out what the updater did and when I ran it by myself it was fast, it took me a little while to realise that it wasn’t the updater process that was the problem. Getting to this point may or may not have involved me disabling the anti-virus :)

With the updater disabled, I found other things that should not use lots of CPU like notepad without any text in suddenly using 100% CPU for a few minutes. So every day the virtual machine I was using and everyone else’s virtual machine was slow enough to cause us to start our lunchtimes early, what the hell was going on?

The next logical place to investigate was the physical hardware, are our virtual machines on the same, hardware and we had a noisy neighbour? Who was it?

Our virtual machines were on different hardware, no cigar :(

In the end, it was the updater but only because the schedule for the updater was the same on every server and the time syncing between every machine was perfect. Every day at the same time, normally fast virtual machines which were happy to share some decent CPU’s all wanted to use a little bit of CPU at exactly the same time which caused every physical CPU to run at over capacity which ended up taking ages.

The IT team staggered the schedule for checking for anti-virus updates, and the problem went away, they even inadvertently managed to re-enable my anti-virus, and I went back to taking my lunch time at a more reasonable 12:00 :)

So if you virtualize a CPU don’t forget that you are sharing it and if you want to use it all, one person might win but you’ll probably both lose.

Comments:

Ashish Kataria

October 24, 2017 - 06:42

Haha Interesting story..

Haha
Interesting story..

David Plaut

November 3, 2017 - 11:33

virtualized CPU: detective story

Nice detective story. Thanks for sharing.