Hi Graham,
If I might chip in (as an engineer, not and experienced boat builder). Why use a flexible coupling at all?
The vibration comes from several sources: the 'kick' of the motor, any out of balance mass in the rotating parts and - if you fit a Hooke's joint - any minute misalignment in the coupling. The first two you can't do much about (although I doubt there is any significant imbalance in the shaft at your rpm).
Your prop shaft is (presumably) supported at the prop end by a bearing. The other end is not supported so it's free to wobble about. If you couple it rigidly to the motor shaft (in a straight line) then the whole shaft is supported at both ends and it won't vibrate. I use a simple rigid collet style coupling and I don't get vibration even though I'm running close to 20 000 rpm (until I crash
)
What I did was to support the prop shaft at the prop end, fix it to the motor and then fasten the motor down. That way the shaft and motor are in a straight line automatically. Finally I glued the prop tube in place nicely centred on the shaft. It's too late for you to do that but you could fix the shaft to the motor and then bed the mount down on epoxy to centre the shaft in the tube. The alignment of the tube is not critical - it's just the shaft and motor need to be in a dead straight line - hence the rigid coupling.
You know that the style of coupling you have currently isn't going to work for geometric reasons. You don't need a flex coupling at all and if you fit one (any type) you are actually inviting the shaft to flail around and vibrate. A lot of people don't realise that the way to kill vibration is to make everything as stiff and rigid as possible. (If it can't move, it can't vibrate
)
I'm not sure how much success you will have with putting foam under the motor mount. Soft mounting will actually amplify the vibration below a certain speed. Just what that speed is depends on how stiff the mounting is. I'd bolt it down solid (but see my PM). I do agree that a lot of the noise you are hearing is the hull responding to the vibration. It'll probably be a lot better when the deck is glued on.
So that's my five eggs. Make the motor and the shaft one nice rigid unit and fit everything else around it. The prop tube can sit wherever it likes!