Friday, October 31, 2014

Does it really make sense to defrag Domino data on a SAN or in a VM ?

We hear this a lot, "My ESX admins say we don't need to defrag Domino because it is running on a virtual platform", or even this chestnut, "Our storage is SAN based and our SAN admins are saying we don't need to defrag Domino". So does it really make sense to defrag Domino in these circumstances?

Takes deep breath!....... Yes it does make sense to defrag your virtualized Domino servers running in a VMWare environment (or SAN connected) and we have many Defrag.NSF customers running their Domino servers on VMware (or SAN connected). Performance-wise there is already a slight penalty for running as a VM compared to a physical box, so we have found that in a virtualised environment it is even more beneficial to use Defrag.NSF to improve overall performance of not only the Domino server but also the O/S itself.

The issues caused by fragmentation in the virtualized environment do impact Domino performance and also the O/S because as far as the Windows O/S is concerned it is still working with fragmented files and because of this Windows has to issue more I/O requests to access these files. This creates Split I/O's where unnecessary multiple I/O requests are generated to access fragments of each file and that contributes to I/O bottlenecks with some requests being blocked while other are fulfilled. Basically the end result is it still perceives the fragmentation and has to deal with it the same as if you were running the O/S and Domino server on a physical box.

Thankfully VMWare have updated their latest documentation for VSphere 5.5 to reflect the recommendation of defragging the Guest OS.

When using a SAN, it is a similar situation because again the files are still seen by Windows as fragmented, it is totally unaware of how the files are stored or even that it exists as a VM, it will still issue the extra I/O requests which is bad.

Here is a very good blog post by Wes Morgan on this subject . Wes gets it!

Here is another post on Preemptive's blog with a brief description, this time discussing a SAN, similar in concept in the way the O/S "sees" the fragmentation and behaves accordingly. Again, the emphasis here is, the O/S "sees" fragmentation and behaves accordingly.

For a long time silent on these issues, IBM finally, in a move that supported our points, included the recommendation to defrag Domino even when on a SAN in their official documentation concerning Domino maintenance and performance , EVEN WHEN IT IS ON A SAN!

IBM agrees, as noted in their "Optimising Lotus Domino Administration" Redbook.

Regardless of what type of storage is implemented, NTFS fragmentation left unaddressed will eventually lead to a fragmented MFT, Domino is very good at generating the large amounts of fragmentation that can lead to this situation. The MFT has to be constantly updated with the on-disk location of all these individual fragments, as a result it can grow to fill the MFT pre-allocated space and start to fragment itself by writing new entries out amongst your data files. If left unaddressed while running a Domino server this can eventually and effectively totally cripple a system, and yes, we have seen this, we have seen Domino servers with fragmentation you would not believe, we have had customers come to us with this very problem and wanting to prevent it happening again.

Here is the Domino Data Directory analysis of one such Domino server, a production server with almost 1 Million fragments and the MFT is also fragmented. The screen-shot below was sent to us by a company who had been advised "it makes no sense to defrag NetApp storage". Here's what the OS was seeing.

 From the OS point of view the fragmentation analysis shows a totally fragmented disk.  At that point it does not matter to the OS that you are on VMWare and VMWare sources and SAN admins have told you not to defrag, as far as it's concerned and from it's perception it's working with a level of fragmentation that it was not designed to tolerate or deal with, as a result it cannot function effectively and the system suffers accordingly. At the bottom half of this blog post we discuss this, even though blog discusses primarily about how FTI's can help cause the issue. Like we said previously, Domino is very good at this!

Yes folks, fragmentation is very real to the OS and it needs to deal with what it perceives, it is real to the Domino server running on the OS and so is the performance hit, it is also real to your Backup software that needs to read/copy the fragmented files, a process that becomes slower and slower as fragmentation increases. Running Domino virtualised or SAN storage is not a Get Out Of Jail Free card with regard to fragmentation and it's effects, getting the best from your system involves optimising the "system" and that includes the Guest OS and the Domino server.

Don't  be like this guy.

No comments: