This is why you should move your Full Text Indexes

This blog post is a reminder about Full Text Indexes and why you should at the first opportunity just move them out of your Domino server's data directory, if you are running 8.5.3 you can very easily move them to a different drive with the FTBasePath notes.ini setting and then running an Updall -f, to automatically create them on the new location and delete the old ones. The Full Text Indexes have an insidious negative effect on the fragmentation of the freespace and also the other files (your nsf's) they share a drive with, so it just makes sense to separate them from your data. A lot of people seem to want to defrag the Full Text Index files but if you think defragging them is the answer then keep reading and I will show you why doing so is quite pointless and really a massive waste of server resources and time.

Here's an example of what happens time and time again, THIS HAPPENS EVERY TIME THE UPDATER REINDEXES A DATABASE

The Full Text Index associated with a user's mail file is cleanly defragged as per the pic below, at this point it is in one contiguous fragment on the disk.

We also see that the associated database has 12 unindexed documents since the last update less than an hour ago at 12:27PM (The hourly Chronos run).

At 1:18 PM Chronos starts it's hourly run and the database is re-indexed.

A check about a minute later shows all documents are now indexed again, but this has an insidious side-effect.

We can see below that the reindexing has now created a lot of fragmentation and scattered what was previously a contiguous index file into fragments all over the disk, this of course also creates new opportunities for fragmentation of your nsf files that happen share that drive.

An hour later Chronos runs and the process begins again...

And Again....

And Again...

Looking the same Full Text Index again and confirm  things are getting worse.

It doesn't all end there though, this repeated fragmenting activity also creates an enormous number of attribute entries in the MFT (Master File Table), some of these indexes can be very large (have you checked out the sizes of your user's FTI's lately?) and as seen above become extremely fragmented over and over, this is all happening under the covers 24/7 and this is just decribing what's happening with the FTI's, add to that the regular fragmenting of the nsf files through normal usage and things can look even worse. 

The MFT has to be constantly updated with the on-disk location of all these individual fragments, as a result it can grow to fill the MFT pre-allocated space and start to fragment itself by writing new entries out amongst your data files. As the situation worsens, eventually "non-resident" $ATTRIBUTE LISTS are created, this is where the lists of file attributes and every individual fragment location on the disk has become so extensive that the lists (meta data - basically information about the information about the information about the location of all fragments and files on the disk) have to be stored outside the MFT as "non-resident" and amongst the also fragmented nsf files on the drive.

Do yourself, your server and your users a favour, upgrade to 8.5.3 and get those Full Text Indexes off your Data drive!

Oh Yeah!, and start using Defrag.NSF now to keep all this under control BEFORE it gets to the point where you actually notice a problem!