Morning all
You are caught between the' devil and the deep blue sea ' with this one. The Turnigy motor you are using is designed for an aircraft. Hence the name Aerodrive and a Kv rating of 2300 is fairly common but it would be open to forced air coming from the propellor in a plane which cools it down...a lot.
An 11.1v Lipo times 2300 means a rotational speed of about 27,600 rpm if the battery is fully charged (12v. plus) which is the same speed as a high speed woodworkers router. It is probably pushing out at about 1/2 a horsepower as well!!!! Of course it is going to get very, very hot in a boat.
At the same time you want a racing hydroplane to go really fast so to do this people put bigger and bigger propellors on Like the 40mm you have, which makes it get even hotter still. That is until it blows up.
The motor is designed to run at that speed believe it or not but it needs to be cooled, a powerful fan will help but its not enough. You could change the mounting to a water-cooled one as well, there are several out there for a 35mm diameter motor, it does not have to be a matched turnigy one.
I would suggest a two bladed Hydro prop of about 35mm diameter, which that motor can drive at 26000 rpm . K-series Graupner props are good if you can find them where you live. Or look for an aluminium one of the same diameter, they are quite curved and are very efficient at high speeds. You may find that sizes even smaller work better in fact, like 30mm or 32mm.
Putting a 750 kv motor in will solve the heat problem a bit, but it will not go like a hydroplane.
I'm afraid the best other option would be a 50mm diameter by 70 long 2000 plus KV. inrunner with a watercooled jacket, a 120 amp ESC as minimum, and even then it will run a 35 K racing or Hydro prop. But it is designed for a boat and will do what you want....a lot of money though.
My thoughts only (Based on past experience).......have fun.
Ron.