Unfortunately there isn't a simple software solution. Much depends on what modem you are using, what OS, how good your dialup connection is, what protocols your ISP is using and whether or not you can utilise call waiting on your telephone line.
Older modems used to just disconnect but nowadays the software mostly deals with incoming calls without disconnecting. Call waiting needs to available and enabled for this to have any chance of success, since only then can the "beeps" be heard informing of an incoming call. In all other instances your line just seems to be engaged.
If both you and your ISP have V92 Modem protocols ENABLED and not just READY then there are a number of software options available. V92 has however been very slow to be taken-up by a lot of ISPs.
Earlier this year I installed a V92 Conexant ModemBlaster PCI modem, which does inform me with a popup box of an incoming call - but it still can.t be configured to automatically disconnect the internet connection until my ISP upgrades to the V92 protocols.
I think your best option is to go with a hardware solution and buy one of the little boxes that sits between the telephone jack and your computer and which should be able to disconnect you when there is an incoming call; (you will still need call waiting on the telephone line).
Other options involve renting a second telephone line or having broadband internet access.