In an enclosed volume like the inside of a boat, if cooling is needed, the important thing is to move the heat out of the boat. A heatsink and/or a fan helps the motor transfer heat from itself to its surroundings, but if its an enclosed box, the surroundings get hot as well.
If the motor is big enough for the boat and if not being forced to labour by having a stiff prop shaft or too much prop it shouldn't get too hot anyway. Generally speaking, with motors, "too hot" means "pain on touching".