Hi there,
I've sailed dinghies for years - I can't speak about RC sailing, but answering straightforward "angle" questions probably isn't what you need.
First off, you want the centre of lift for the sail layout just behind the centre of sideways resistance for the hull. How to test this? Your model should, when on a beam reach, want to sail straight or, preferably, turn into the wind just a little bit with the rudder straight. Angling the mast backwards or forwards is the easiest way to adjust the position of the centre of lift. If the boat wants to turn downwind on a beam reach, angle the mast back. If the boat wants to turn upwind badly on a beam reach, angle the mast forward. An alternative to angling the mast is moving the ballast - easy in a dinghy, harder on an RC sailboat - but angling the mast aft is the same as moving ballast forwards.
Secondly, you ask about the angle between mast and mainboom. It all depends on the cut of the sail. This angle is not as important as the tension on the sail: you want a flatter sail for upwind work, and a fuller sail for reaching and downwind sailing. In dinghy sailing you can slightly loosen the kicking strap holding down the boom when sailing off the wind. In dinghy and RC sailing I suppose the mainsheet helps haul down the boom when sailing close-hauled. It's the shape of the sail that you need to consider - not so much what angle it is, or, indeed, what planes the boom can move through. Forget the numbers, think of the forces and shapes.
Sailing is an art - hard enough when you're in the boat and able to make adjustments. It must be far harder for the RC crowd. But I would recommend three things, first off:
1/ Get the centre of lift/centre of sideways resistance right. If it's wrong, and particularly if you're trying to reach or sail upwind, you'll burn off a surprising amount of energy using your rudder to counter the weather or lee helm. That's bad: imagine spending ages making and polishing a hull for minimal resistance - and then sticking a big plough at the back of it.
2/ Don't sail with the sails too tightly drawn in nor too loose. Loose sails are obvious (they flap!), and so for novices to sailing it's often easy for them to fall into the habit of sailing with sails too tight, since there's often little visual evidence to give that away. But tight sails are inefficent, and if you have a jib, often doubly so - the slot between the main and jib is a force multiplier. You WANT high speed air over the back of the main to maximise your power. You do NOT WANT to stifle that slot by overtightening the sheets. Play with the sheet settings all the time. Not wildly in and out, of course, but subtly around the working side of the "luff" point. Learn to find and feel what your boat wants most.
3/ Think about the vectors. Modern, high performance sailing vessels can move at an appreciable percentage of the wind's speed. Some can sail faster than the wind, on certain headings. Think not only about the direction of the wind over the lake, but consider that direction when looked at from the point of view of the moving boat. ANY speed on a boat will move the apparent wind direction forwards. As the boat accelerates, the sheets will need to be gently tightened - but bear in mind warning 2/! Don't overtighten!
I said three things. There's an obvious fourth:
4/ Practice. As much as you can, in as many conditions as you're able.
Regards,
Andy