I come to this from a background in Car PCs so I have faced a few of these problems.
You are absolutely correct on the size of the motherboard however the PSU I would use is tiny and is designed to cope with an input range of 6-26v.
http://www.mini-itx.com/store/?c=10#p1835 Also as you will see there is no problem with it cutting out when the battery runs out. If you have more money to spend you could get a Nano-ITX which is 12x12cm.
Heat should not be a problem. Our puposes are not at all power hungry. We can get the slowest, cheapest EPIA motherboard and without maxing out the CPU it will not give off much heat. A bit of airflow is an advantage but not totally required. If you were to put this in very cramped area with no air intake I would advise to use the versions with a fan as this will stir around what little air there is.
For a harddrive you can use either a laptop one or a compact flash card. There are no special mounting options required, laptop drives are a lot studier than their desktop counterparts. Compact flash has the advantages of size, speed, even more robustness and using a lot less power. The price has fallen dramatically due to the popularity of digital cameras and you can get a 1GB card for £14 or a 2GB for £18. With compact flash you must use the EWF, it is optional for a laptop hardrive.
Running windows XP and using some optimisations it is easy to get the boot time down to 20s or less if you are more skilled. There are a number of things to do including disabling hardware we are not using, running from compact flash, use tinyxp or nlite to strip down xp to what we basically need, borrow minlogon and the EWF from windows embedded, store a hibernation file so that each time we start we are returning from hibernation instead of loading everything up again.
EWF is the Enhanced Write Filter. What this does is store any changes we make to the disk to ram instead and then we can decide to merge them back in or forget them entirely. It is required for compact flash because CF only has a set number of writes and with windows we would use this up in a matter of days. In this application we can store a hibernation file on drive/CF to start from. The applications never write to disk, they simply communicate over WiFi so when the power is cut off all changes are lost and we start again with our hibernation file at next boot. This way we don't have to worry about corrupt files and need no shutdown control.
WiFi range is from 100m-400m with standard antennae depending on whether you have a/b or the latest g. With a directional antenna you could get a range of several km but you won't be able to see the boat by then. :)
Software you would have to code, but I will probably make mine in such a way that it is quite configurable and share it with whoever is interested.
For control, there are a number of options, you could use a PDA or PSP or a laptop with a standard joystick or steering wheel. You could even control it with Bluetooth from your mobile phone. I was planning to use a Playstation controller with a usb adapter (connected to a laptop) as this is most like a normal RC controller.
A lot of this kit is quite expensive, my estimate puts it at around £125-£175 if you buy it all brand new. But the good thing is that because they are all standard components, some you may have lying around already or otherwise there are a lot for sale on ebay.