Hi, I can confirm that the Viper is designed to work on a fully charged 12 volt battery, it's in some advertising and I do this myself. As Dave says, opinions vary about ESC brands, I've never had a problem with any Mtroniks unit, in 20 years.
I think your problem is simple, the 600 motor is way over the top for a small boat. Try either a 385, these are OK up to 15 volts, I use them in 2 models, a 45 inch cruiser and 2 in a Bardic ferry, 42 inches long, both run on 9.6 volts so you should be fine.
If the 385 doesn't appeal then a 540 LN ( NOT a standard 540)is probably a straight swap but is probably overkill.
Chas