It might possible be that you have a security porgram running in the background.
Even though your drive might not be badly fragged Defragging sorts\moves your progs so that they start quicker and this is what takes up the time. (hope I have explained that bit corretly)
Best way to defrag (IMHO) is:
Disconnect from net.
Close ALL running programs.
Run Defrag and then reboot.
If your drive is badly fragged it will take a long time. It is also if possible running it in safe mode.