Learn about the technologies behind the Internet with The TCP/IP Guide!|
NOTE: Using robot software to mass-download the site degrades the server and is prohibited. See here for more.
Find The PC Guide helpful? Please consider a donation to The PC Guide Tip Jar. Visa/MC/Paypal accepted.
|View over 750 of my fine art photos any time for free at DesktopScenes.com!|
FAT32 Performance Tradeoff: FAT32 Cluster Sizes and FAT Sizes
It is generally believed to be a "rule" of cluster size selection that "smaller is better". As FAT16 partitions have gotten larger and slack waste has gone through the roof, the push toward using FAT32 to reduce cluster sizes has been tremendous. While FAT32 does allow the use of larger hard disks and greatly reduced cluster sizes, there is an important performance consideration in using FAT32 that is not often talked about. Now that huge hard disks with dozens of gigabytes have made FAT32 essential for newer systems, you often don't have a practical choice between FAT16 and FAT32 any more. However, the principles in this page are still relevant, depending on how you are setting up your hard disk. They can also help influence your decisions regarding how to partition very large drives under FAT32.
Let's consider a partition that is just under 2,048 MB, the largest that FAT16 can support. If this partition is set up under FAT16, it will result in a file allocation table with 65,526 clusters in it, with each cluster taking up 32 kiB of disk space. The large cluster size will indeed result in a great deal of wasted space on the disk in most cases. Therefore, often it will be recommended that FAT32 be used on this volume, which will result in the cluster size being knocked down from 32 kiB to 4 kiB. This will, in fact, reduce slack on the disk by an enormous amount, often close to 90%, and potentially free hundreds of megabytes of previously wasted disk space. It is usually the right thing to do in this situation.
However, there is another side to this; you don't get this reduced cluster size for free. Since each cluster is smaller, there have to be more of them to cover the same amount of disk. So instead of 65,526 clusters, we will now have 524,208. Further more, the FAT entries in FAT32 are 32 bits wide (4 bytes), as opposed to FAT16's 16-bit entries (2 bytes each). The end result is that the size of the FAT is 16 times larger for FAT32 than it is for FAT16! The following table summarizes:
Still worse, if we increase the size of the FAT32 volume from 2 GiB in size to 8 GiB, the size of the FAT increases from around 2 MiB to a rather hefty 8 MiB. The significance of this is not the fact that the FAT32 volume will have to waste several megabytes of space on the disk to hold the FAT (after all, it is saving far more space than that by reducing slack a great deal). The real problem is that the FAT is referred to a lot during normal use of the disk, since it holds all the cluster pointers for every file in the volume. Having the FAT greatly increase in size can negatively impact system speed.
Virtually every system employs disk caching to hold in memory disk structures that are frequently accessed, like the FAT. The disk cache employs part of main memory to hold disk information that is needed regularly, to save having to read it from the disk each time (which is very slow compared to the memory). When the FAT is small, like the 128 kiB FAT used for FAT16, the entire FAT can be held in memory easily, and any time we need to look up something in the FAT it is there at our "fingertips". When the table increases in size to 8 MiB for example, the system is forced to choose between two unsavory alternatives: either use up a large amount of memory to hold the FAT, or don't hold it in memory.
For this reason, it is important to limit the size of the file allocation table to a reasonably-sized number. In fact, in most cases it is a matter of finding a balance between cluster size and FAT size. A good illustration of this is the cluster size selections made by FAT32 itself. Since FAT32 can handle around 268 million maximum clusters, the 4 kiB cluster size is technically able to support a disk volume 1 TiB (1,024 GiB) in size. The little problem here is that the FAT size would be astronomical--over 1 GB! (268 million times 4 bytes per entry).
For this reason, FAT32 only uses 4 kiB clusters for volumes up to 8 GiB in size, and then quickly "upshifts" to larger clusters, as this table shows:
Note: I am not really sure
what the maximum partition size is for a FAT32 partition. :^) I have heard various
different numbers, but nobody seems to be able to produce an authoritative answer.
"Officially" it is 2,048 GiB (2 TiB), but there are likely to be other limiting
As you can see, despite the claims that FAT32 would solve large cluster problems for a long time, it really hasn't. As soon as you hit 32 GiB partitions, you are back to the dreaded 32 kiB clusters we all knew and hated in FAT16. Obviously 32 GiB is a lot better than having this happen at 1 GiB, of course, but still, FAT32 is more of a temporary work-around than a permanent solution. When FAT32 was first introduced, many people said "Yeah, but 32 GiB is huge. I'll probably never have a disk that large and if I do, I won't care about a little wasted disk space!" In fact, it took less than five years for hard disk makers to move from selling 1 to 2 GB hard disks, to selling ones 32 GB in size or more! And those same people are finding they do care about the wasted disk space, though perhaps less than they did when they only had 1 GB. :^)
The table below is a little exercise I did for fun, to show the size of the FAT (in MiB) as the partition size increases, for various cluster sizes. You can see why FAT32 doesn't stick with 4 kiB clusters for very long--if it did, you'd need enormous amounts of memory just to hold the table. (A 60 GB partition, if formatted with 4 kiB clusters, would result in a FAT that is 64 MiB in size, which is about as much as the entire system memory in many newer PCs.) The entries in bold show what FAT32 will choose for a partition of the given size; by going up in cluster size Microsoft keeps the size of the FAT to no more than about 8 MiB through partitions of size 64 GiB:
The last entry, 2 terabytes, is for fun, to show how laughable I find it when people go on about 2 TiB hard disks "being supported by FAT32". Technically they are, I guess, if you want to deal with a FAT that is 256 MiB in size and go back to having 40% of your disk wasted by slack. We had better hope our system memory goes up by a factor of 1,000 before our hard disks do again. We also better hope that no other little "surprises" show up as disks get larger: several did pop up when the 64 GiB barrier was reached, such as difficulties with the FDISK program.
Really, what this page shows is that the FAT file system is stretched beyond its limits even with FAT32. To get both good performance and disk space efficiency for very large volumes requires a clean break with the past and the use of a high performance file system like NTFS. I should make clear that I am not recommending against the use of FAT32 on Windows 9x/ME systems. With modern drives of 50 to 100 GB, FAT16 is just too impractical, with its 2 GiB partition limit. At the same time, I want to make sure that people realize that FAT32 has its own limitations. It is really more of a kludge than a permanent solution to the problems of large partitions and cluster-based file allocation.