Boot into Safe Mode using the [F8] key or hold down [Ctrl] during boot. Remove all modems from Device Manager. If one is a hardware based modem, use it and leave it in the board, restart and windows should find and install it.
To determine whether you have a software modem, look for system requirements on the box it came in, or on the manufacturer's website. If it requires a certain CPU speed to run, it's a software modem. If the name and model says LT winmodem, HSF or HCF it's a software job. They have to be installed using the setup.exe or install.exe file in the drivers, not the Add New Hardware Wizard. Often you have to remove the modem from the motherboard before installing the software, then physically install it after the software is installed.
Also with software modems sometimes you have to enable COM2 in BIOS, depending on what COM port the modem wants to use by default. Start with COM2 disabled in BIOS. (Serial Port 3F8) You say yours wanted COM5. That would mean it also would use the same IRQ as COM1, which is the mouse if you have a serial mouse. Let us know if that is the case. If you have a PS/2 mouse it uses IRQ 12, no problem there.
So many idiots, and only six bullets...
Note: Please post your questions on the forums, not in my email.
Computer Information Links