I searched posts in this forum and didn't see this question posed before, but I can't believe I'm the first one to wonder if I could or should do routine maintenance on a flash drive like I do on a HD, floppy or Zip disk. Can it be reformatted? How about scandisk and defrag? I'll wait for a reply before performing any of these actions.

No you should not need to defragment, a USB flash drive is a digital memory device, not a spinning disk.

Careful how you play. USB drives can written to by various utilities. It is possible, that you could get unexpected results. Depending on what drive you want to play with, be sure you check the vendor's website and use their utilities........

It really depends on the file system rather than the fact that such drives don't contain a spinning disk inside them. Mostly FAT16 seems to be used but bear in mind that some of these flash drives are getting into big capacities and other file systems are used particularly for use on MACs, for example.

The main "problem area" is to do with non standard mbrs (usually only one non-bootable primary partition) on those that emulate HDDs. I have accessed many of these drives with hex editors without problems and have run chkdsk and defrag and formatted them from windows without problems - which doesnt mean no problems will ever occur.

First of all, you don't really want to do defrag a flash drive because flash memory doesn't last forever. The onboard controller in a flash drive is designed to stagger the use of the sections of flash memory on the chip(s) so that they age evenly. By running defrag, you're aging certain portions of your flash memory faster than others.

Scandisk doesn't hurt a flash drive, but it doesn't necessarily help, either. Again, the onboard controller keeps track of bad sectors by verifying all data after a write cycle. If the verification fails, the onboard controller quarantines that section and never uses it again.

Formatting is also done by the onboard controller, which uses some of the total space for the file system. For instance, my 128MB flash drive uses 5MB for the file system and core routines, leaving 123MB of usable space.

The whole reason for having the USB & drive controller built into a flash drive is to take care of all this maintainance and to allow flash drives to be bootable. Don't try to usurp its authority!

you're aging certain portions of your flash memory faster than others
Now that's interesting and particularly interesting if one is going to be booting to an OS on such drives. What I am getting at is the VM in such a situation. Since (with or without a fixed size VM) there is going to be an awful lot more read/writes made to the same or similar areas of the drive. I guess it would thus be advisable to try to avoid a pagefile altogether and to only boot to such drives when there is loads of RAM on the system.


Formatting is also done by the onboard controllerAlso interesting and something I had not really ever thought about. How does this compare with the role of the onbard controller when formatting magnetic media?

How does this compare with the role of the onbard controller when formatting magnetic media?Normal hard drives have a shared controller architecture, where both the hard drive and the system have a piece of the puzzle, so to speak. And hard drives are based on many many years of universal drivers for both Windows and Mac OS, which can natively handle hard drives of all sorts (including USB external ones).

However, there's no such set of universal drivers for the various types & capacities of flash memory. Flash drives have a USB controller for handling the USB bus interface (which is universal for the most part) and of course most USB implementations have a bootable spec. But on the inside, the flash memory controller itself is in full control. It shares it duty with nothing else. This is a good thing, because there's no set rule for how flash memory works. There are many types of flash memory, many voltage schemes, many sizes, many chip configurations, etc. A full internal controller for the flash memory itself is the only solution. This also makes them highly flexible, because the file structure is internal and is thus fully compatible with any system capable of interfacing with a USB removable drive. The files themselves may not make any sense from Windows to Mac OS, but the drive itself is still usable.

As to the aging effect, this is really only a concern if you exceed the write cycles of the flash memory (which is in the range of tens of thousands of rewrites). Small embedded systems have been using flash memory for hard drive duties for years, and of course tiny Linux-based boxes using CompactFlash cards for hard drive duties are also popular. Running an OS off a flash drive is of little concern as long as you realize that you have to replace it every now and then. But to intentionally make things worse by defragging a flash drive is unnecessary.

The files themselves may not make any sense from Windows to Mac OS, but the drive itself is still usable.
That I don't follow unless you are talking about sectors and not files as such. As I had understood things, any drive controller (following a true factory low level format) presents the usable disk space to any operating system as an "organisation of usable bytes". The geometry may vary but under modern systems this space is usually addressable as LBA sectors containing 512 bytes (of data and which doesnt include any metadata) per sector.

For most (but not all) operating systems such raw space needs to be initialised with one or more partitions which then have file systems written to them by normal formating using an operating system file such as format.com for example.

When you say "Formatting is also done by the onboard controller" and "The files themselves may not make any sense from Windows" this states or implies that such controller-formatting is organising the file system - as distinct from the addressable space - and that is what I am having trouble grasping. I have variously formatted USB flash drives with FAT, FAT32 and NTFS file systems (and I believe that Mac users can format with OSX). I have used WinXP, BootItNG and the HP download from http://h18007.www1.hp.com/support/files/hpcpqdt/us/download/20306.html at different times and with different results.

We may well be talking at cross purposes but this is an area of great interest to me so any help in understanding is much appreciated - particularly when it relates to data recovery.

I am treating my thumb drive with a bit more respect after reading this thread.
:)

Have read up a few articles on Flash File Systems and there's a pretty good Wikipedia about Flash Memory at http://en.wikipedia.org/wiki/Flash_memory.

I now see that there are two important problem areas. One is that erasure is in blocks and is slow and the other is to distribute as much wear levelling as possible.

In reading about the flash file systems such as FFS2, FTL JFFS etc I guess that is what you, Saphalline, was referring to. I also noted the comment: "However, in practice most flash media is used with the old FAT filesystem for compatibility purposes."

I suppose that since read/write is best minimised on such media that an operating system using FAT would be better than NTFS (when installed on such memory) since the latter is a journaling system with many more i/o operations. On the other hand some figures on the expected life of such memory under different real life conditions would be nice to get one's hands on. I mean - hard drives dont last forever either so would one expect to get a comparable life out of such media.

As ever it amazes me how much more there is to learn - but that is also the beauty of the way technology is developing.

Paul if I remember right when I bought my flash drives they said "data retention up to 10 years".

I would take it that means 10 years is the max. So I would guess maybe 7-8 years for normal operations. Maybe 5 for heavy use. That's just my guess though.

It would be interesting to see some real world stats.

The files themselves may not make any sense from Windows to Mac OS, but the drive itself is still usable.Haha, sorry, I guess this one was confusing. What I meant by this is that you can put files on a flash drive using a Windows system, and then put that same flash drive into a Mac system and transfer the files over. However, Windows and Mac don't always use the same file types! A VXD driver file from a Windows system would mean nothing to a Mac system. That's all I was saying.

Yes, flash memory is quite interesting, isn't it? ;) NOR gates vs NAND gates, different topologies, block erasures, limited insulator lifespan, quantum-tunneling, all sorts of cool stuff. One of the reasons that you have to be gentle with flash memory is specifically because of the block erasures inherent in the NAND gate usage. Since you can only erase entire blocks, making a small change to a file essentially means erasing a block and then rewriting the portion of the block where the file resides. In that sense, I hadn't thought about the overhead of NTFS vs FAT, but that's a good point. Less wear & tear is always good.

Number of write cycles for each type of flash memory? That would be a HUGE list! :eek: The problem here is that they're all different. Even among the same manufacturer. For instance, a normal SD card from Kingston may have x number of write cycles for its lifespan, but a high-speed SD card from Kingston might have 0.8x for its lifespan. Faster flash memory generally ages faster because the extra speed and/or voltage wears out the insulator faster. And of course the oldies like slow CompactFlash have really long lifespans, because they use larger gate elements and thus thicker layers of insulator and they're slower. Newer types of flash memory like xD use multi-level gate technologies where each gate element can store multiple bits. This level of precision, coupled with thinner layers of insulator would seem to make it age faster, but the much lower voltages and smaller eV capacitancies increase its lifespan.

Even the manufacturers have no set formula for figuring out the lifespan of a flash memory product. They essentially just take a sample, set it up for constant rewriting, and let them go! Then they measure the death toll. That's about it! But of course this doesn't take into account actual aging over time, as in how does it store data over years and years if untouched? All forms of solid-state memory suffer some form of leakage over time, but how do you truly measure this? My advice is not to leave any sensitive data on a flash drive in storage for long periods of time :p but theoretically, you could. At least for some types of flash memory.

For the formatting, the onboard flash memory controller is responsible for translating the file system and for keeping track of evening out the wear & tear, but I'm not entirely certain about the implications of organizing the memory itself into sectors while at the same time the memory is orgainized into blocks and layers. I suppose in that case the file system installed would lay on top of the native file system of the onboard controller. It would have to be something like that because flash memory is solid-state memory, based on chips capable of random access. It's not a platter of expansive and contiguous bits. The only "order" to any type of RAM/ROM chip is implied. It's like the difference between a race track with definitive circular elements and a direction that you're supposed to follow, and an office building. Finding a car on a race track is easy! Finding the "Associate VP" in an office building is not quite as intuitive!

Very interesting. The first flash drive I got(Kingmax 128MB) I formatted it with NTFS just to play around. It was unusable. It wouldn't transfer anything at all. Tried FAT, same thing. I had to use the recovery utility to get it back to normal. I haven't tried to format with my 1.5GB(Rosewill) one yet. Actually I don't see any reason to format it. They work on any computer I have put them in. :)

I nominate saphalline to be the first human to have his brain cloned.

hi,

i beginer here, hope everyone can help......thx 1st........
i'm currently implementing the FAT structure on to the nand flash and i found some difficulties, hope anyone able to help me.

i had this nand flash -> 256MB: 1page = (2046 + 64)bytes, 64pages = 1 block, total block number = 2048.
izzit 1 sector = 1 page, and 1 cluster = 1 block? if this is the case, my cluster size is larger than 65535bytes which the microsoft claim this may cause problems. :confused:

besides, 1 logical address represent 1 block? so my total logical address is 2000? :confused:

all the information i get is 1 sector = 512+16 byte where the structure can be known but 64byte redundancy area,what is the stucture???

if i need to transfer data from pc to nand flash through the usb, izzit my nand flash mush had the fat file structure? or i can implement another type of file management system to the nand flash?

anyone able to help can direct email me oso ->ycherjier[AT]yahoo.com

I nominate saphalline to be the first human to have his brain cloned.

I second that nomination

I 3rd it!!!
;)

Problem:

I asked disk keeper to run boot time defragmentation on my external drive, however when I rebooted my computer it completely ignored the scheduled event, leaving it scheduled for next boot time. I now have the problem of needing to run check disk on my drive before I can do any defragmentation on it (the computer refuses to defragment a drive that has disk checking scheduled). Is there any way to make windows recognise the disk and perform the disk check?


Yes I found the solution:

Here is a solution to override this issue of not being able to run the ckdsk when you boot up for any external USB drive. Since the USB drivers arent loaded before the Checkdisk can run .. you need to do it manually.

Here is what you do..

1. Have the drive plugged into your computer.
2. Under "My Computer" find your external drive, usually E: or whatever...
3. RIGHT CLICK on that drive
4. Select PROPERTIES
5. Select TOOLS
6. Click CHECK NOW button under the ERROR CHECKING option
7. The you will be asked for check 2 choices..
8. Check BOTH options..( this will take awhile depending on how bad your drive is...)
9. Then you can go back to the orginal TOOLS tab and choose the DEFRAGMENT option
10. Choose that option.. it should use the defrag software you have installed on your computer to do the defragmentation on that drive.
11. Let it run..it will take awhile and then close it.

It should have your external drive check disked and defragment when your finished.

I did this and it worked for me.. I hope it works for anyone else who uses this tip!!

Let me know if it does or doesnt.. I would be happy to know if it works for anyone else!!

"Format, scandisk, defrag a USB flash drive?"
At THIS THREAD (http://www.pcguide.com/vb/showthread.php?t=51735) sburtchin showed me how to add USB4DOS drivers to an "Emergency Boot CD" [EBCD].
I'm now able to run most of the MANY utilities on the EBCD so as to check not only the internal HDD, but also the external USB devices including my external 80 GB USB HDD and 1 GB USB memory stick.

I noticed when I ran "Microsoft Scandisk" that it checked ALL the drives [file system and cluster checking] including the USB HDD and memory stick.
[Cluster checking froze the software part way through the 1st partition on the external USB HDD :( ]

Things it might do [haven't checked them all]...
1. Browse contents with "File Manager".
2. Scan drives for faults in file system and clusters.
3. Un-Format.
4. Un-Erase.
5. FDISK partition.
6. Save image backups.
7. Zero-fill.

Still trying to figure out just how much customisation is possible, and what is necessary to make various programs work on external USB devices.

e.g. Attempting at this time to make the "Drive Fitness Test Utility" work with external USB devices, as seen HERE (http://www.pcguide.com/vb/showthread.php?t=51781).

8 gig micro disk, usb drive you can defrag them.........