Performance charts and Troubleshooting

There are really good tools and document available on troubleshooting vmware. I’m learning and as I go, I want to share with you all. Though production scenario are quite different but below approach might help you in finding bottlenecks in advance. I intend to show how to keep tab on the below values over a period of time. Say Weekly or monthly. Any changes or variation in these values means there is problem in the system which needs to be address ASAP.


Three important counters to find disk bottleneck

  1. Kernel Disk Command Latency: it is VMKernel processing time for each SCSI Command. So fast VMKernel can process SCSI command is clear from this counter. Ideally it should be between 0 – 1 millisecond. If it crosses 4 millisecond then you need look at either increasing CPU or queue depth. Below is real time observation of one of my ESXi box. You should actually observer this counter for week’s time to check future expansion



  1. Physical Device command latency: It is the amount of time taken by physical device to complete the SCSI command. Here physical device is HBA,Switch fabric or ports. If this value is more than 15 millisecond it can indicate a problem with storage array.




Remember to go by average, in this naa.6000eb316dd2aeb80000000000000020 is at one time 315 millisecond but on average it goes to only 24 sec. Remember these are real time values, we should actually be watching these values our week or monthly period to keep tab on them

  1. Queue command latency: How long SCSI command spends time in VMKernel queue. It should be ideally zero and so it is seen below.If it not zero then there is again problem in the storage array.




5 comments on “Performance charts and Troubleshooting

  1. Is it possible when the Physical Device Command Latency spikes the pipe from ESX host to storage device is not big enuff?

  2. Preetam –

    Trying to find some good sites that talk about storage problems for vm environments and came across your blog. First — check out for our capacity analyzer to do some of what you are doing above. Second – I am going to include a link to your blog in my coming post. Thanks.

  3. Pingback: 2010 in review (Thank you for this) « Designing vSphere

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s