Blog

My server scoring said every VM had a spinning disk

A fleet inventory that scores every server from 100 down, with a sentence for each point lost. On its first real data it marked every virtual machine's disk as a hard drive. What a guest kernel cannot know, and how to stop an assumption from turning into data.

· 8 min read

  • Linux
  • Python
  • Virtualisation
  • Inventory
  • Data quality

The goal was for the platform to say which servers in the fleet were strong, which were weak, and — more usefully — why. So I built a configuration inventory: a read-only collector, uploaded to each server over SSH, run, read and deleted on every pass, with its output normalised into one schema and scored.

Every server starts at 100 and loses points for specific, named reasons. Every deduction writes a sentence: “Low RAM”, “Non-SSD primary storage”, “Legacy NIC driver detected: e1000”. A grade tells you a server is weak; the notes tell you what to change.

Then the first real data came in, and every virtual machine in the fleet had a spinning hard disk.

How the collector decides what a disk is

The logic was reasonable, and it is the logic most tools use:

rotational = read_file(f"/sys/block/{dev}/queue/rotational")

if transport == "nvme" or dev.startswith("nvme"):
    disk_type = "NVMe"
elif safe_int(rotational) == 0:
    disk_type = "SSD"
elif safe_int(rotational) == 1:
    disk_type = "HDD"

NVMe is identified by its transport. Everything else is decided by the kernel’s rotational flag for the device: 0 means non-rotational, which in practice means solid-state; 1 means rotational.

On the bare-metal servers, that was right. Their SATA SSDs reported 0 and were classified as SSDs.

On every KVM guest, the primary disk was vda — transport virtual, model 0x1af4 — and it reported rotational = 1.

Database grid of storage devices: two bare-metal servers show SATA SSDs with discard supported and queue depth 64; the KVM guests show device vda, type HDD, transport virtual, model 0x1af4, queue depth 256.
Bare-metal SSDs classified correctly. Every virtual disk — transport virtual, model 0x1af4, the virtio vendor ID — comes through as HDD.

What the guest cannot know

0x1af4 is the PCI vendor ID used for virtio devices — paravirtualised hardware presented by the hypervisor. A virtio disk is an abstraction: behind it, on the host, there might be an NVMe drive, an SSD array, a spinning disk or a network volume. The guest is not told which.

The rotational flag reports what the block layer was told about the device. Unless the hypervisor or the driver says the device is non-rotational, the guest has nothing to go on, and the flag says rotational. Some hypervisors can advertise a virtual disk as non-rotational — but only when someone has configured them to.

So the collector was not wrong about what it read. The guest kernel was not wrong about what it reported. The inference was wrong: it treated “the guest was told this device is rotational” as “this server has a spinning disk”.

WHAT THE GUEST CAN SEE host storage — NVMe? SSD array? HDD? invisible from inside the VM hypervisor presents a virtio disk vendor 0x1af4 · transport virtual /sys/block/vda/queue/rotational = 1 nobody told the guest otherwise disk_type = "HDD" an inference, stored as a fact WHAT THE SCORE DID WITH IT start 100 virtualised a real property of the server −10 non-SSD primary storage an artefact of the guest's view −20 30 points on two lines, one of them an artefact — with a note that reads "Non-SSD primary storage".
The guest reported honestly what it had been told. The score turned that into a deduction, and the deduction into advice — both wrong.

Why it mattered more than a label

A mislabelled column would have been a cosmetic bug. This one flowed into the score.

Each VM had already lost 10 points for being virtualised — a real property, and a fair deduction for a platform where virtualisation costs something. Then it lost another 20 for “non-SSD primary storage”. The same underlying fact, this is a VM, was being charged twice: once honestly, and once through an artefact.

And the note attached to the deduction was advice. A report telling you to move a VM onto SSD storage — when it may already be on NVMe — is worse than no report, because it sends someone to fix a problem that does not exist.

The most frustrating part was that the collector already knew. It recorded the transport as virtual and the model as 0x1af4 in the same row. The disk-type decision just never looked at them.

Not every VM finding was an artefact

It would be easy to overcorrect and exempt virtual machines from hardware scoring altogether. The data argued against that.

Some VMs had network interfaces using the e1000 driver — an emulation of an old physical Intel NIC — and each lost 15 points for a legacy driver. That one is real: an emulated legacy NIC genuinely performs worse than a paravirtualised virtio one, and switching is an actual improvement someone can make.

So the question is never “is this a VM?” It is “is this measurement a property of the machine, or an artefact of looking at it from inside?” The e1000 finding is the first kind. The HDD finding is the second.

Assumptions need to stay labelled all the way through

Network interfaces had the mirror-image problem. Virtual NICs — virtio_net, vmxnet3, xen_netfront — are often up but report no speed at all. Rather than score a working interface as broken, the collector falls back to assuming 1 Gbps for a known virtual driver that is up, and tags the value virtualized_fallback, so the assumption is visible in the data.

That was the right instinct, and only half done. The tag stayed in the row, but the number flowed into the NIC score exactly like a measurement. A value that was assumed should either be left out of the score or carry its flag through to the grade. Labelling an assumption at collection and then scoring it as a fact is just a slower way of hiding it.

The same principle, applied to writing data back

One part of the design did get this right from the start. The inventory can write its findings back to the fleet’s master servers table — the record other systems read. That sync refuses to run from a partial collection:

Terminal: the orchestrator is run for one server with --mode hardware and --sync-server-profile. It connects, uploads and executes the collector, then fails with 'database ingestion failed: refusing to sync the servers profile from a partial collection; re-run with --mode all'. The result shows status failed with null output paths. Red hand-drawn circles highlight the error and the status.
A hardware-only run carries no operating-system data, so syncing it would overwrite the recorded OS with nothing. The guard makes that impossible rather than unlikely. (Address redacted; the circles are my own annotation from the time.)

Missing data is not data. A field the collector did not collect must not overwrite a field it collected yesterday. It is the same rule as the disk problem, pointed the other way: don’t let the absence of information become a value.

New metrics get the same caution. Network drivers expose different offload features and channel parameters, so the list of metrics cannot be fixed in advance. Anything never seen before is registered automatically — and arrives unapproved until someone reviews it, so a self-extending schema does not quietly fill with near-duplicate names.

What I would change

  • Classify virtual storage as its own type. Transport virtual or a virtio vendor ID should produce virtual, and it should not be scored against physical media at all.
  • Keep evidence next to every inference. Store the raw rotational value, the transport and the model, and derive the class from all three — so a wrong rule can be fixed and re-applied without re-collecting.
  • Carry assumption flags into the score. Either exclude assumed values, or make the grade say it rests on them.
  • Calibrate the thresholds. The deductions are sensible judgement, but nobody has shown that a grade predicts how a server performs under load. The honest next step is to correlate grades with measured throughput and adjust the weights to match.

A score is only as honest as its inputs. The notes column was the most useful thing the inventory produced — and the thing that made the mistake impossible to miss, because it said, in plain words, something that was not true.

This came out of the node configuration inventory, which also covers the read-only collection model and the full scoring tables.

References

← All posts