Lately I got a message from our network engineer that the F5 Enterprise Manager was unresponsive and after a reboot it usually did it again for some time. But this behavior was starting to irritate him, so he asked me if I could take a look at it. We use the F5 Enterprise Manger Virtual Edition VM to manage the F5 appliances. When looking at the console I saw: Err kernel: hda: DMA Timeout error.

So this clearly had something to do with the disk, and when you do a search for the error, you’ll get all kinds of hardware failures on hard drives. But this is a VM and its disks are virtual, they can’t have hardware-failure, only corruption if you want it broken. So what was this error doing in the console and why was the system halted?

Searching the internet I came across a knowledgebase article from NetApp (kb41511) and VMware (kb1009465) which described the cause and resolution.
The problem occurs with high latency or a path failover of the storage. Normally when you install VMware tools, it automatically sets the right values for storage I/O timeout (i.e. 180 seconds). But the virtual appliance uses third party tools which, when you’ll look at the OS configuration, sets the I/O timeout to 60 seconds.

Now for the solution. We have to login the virtual appliance and change the I/O timeout.
Step 1 : Login to the console of the virtual appliance
Step 2: Goto /etc/udev/rules.d/
Step 3: Create a new file called “99-vmware-scsi-udev.rules”
Step 4: Open 99-vmware-scsi-udev.rules in your favorite editor and paste the text below:

ACTION=="add", SUBSYSTEM=="ide" , SYSFS{type}=="1", RUN+="/bin/sh -c 'echo 180 > /sys$DEVPATH/timeout'"

ACTION=="add", SUBSYSTEM=="ide" , SYSFS{vendor}=="VMware, " , SYSFS{model}=="VMware Virtual S", RUN+="/bin/sh -c 'echo 180 > /sys/$DEVPATH/timeout'"

ACTION=="add", BUS=="ide", SYSFS{vendor}=="VMware, " , SYSFS{model}=="VMware Virtual S", RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"

Step 5: reboot the virtual appliance.