M M - I follow this procedure: 1. Calculate the geometric centers of hull, finkeel, rudder, and sails. Then compute the summation values, weighting each individual geometric center by it's area and distance from an arbitrary datum (I use the bow). This is exactly like computing center of gravity for an airplane, if you are a pilot. These values provide the static COE (center of effort, sails) and static CLR (center of lateral resistance, hull+finkeel+rudder).
2. The static centers are a start, but will be insufficient since what we really need are the dynamic centers, and they will not be the same as the static centers. You can see this effect (static different from dynamic) when your ship first gathers way on a beat - she starts off skidding to leeward, then, as she gathers speed, the keel seems to "bite" and she heads up. The bite comes from the keel starting to develop lift (as opposed to just drag when the ship was motionless), and the heading up comes from the fact that the center of lift is forward of the center of drag, giving a weathervane effect.
3. Treat the ship as if she were composed of 2 wings, one above and one below the waterline. For wings, the center of lift (dynamic) is located at the 1/4 chord point. Thus, the dynamic CLR will be located half way between the static CLR and the bow. This displacement of lift (our COE and CLR) from the geometric center of the wing is why metal airplanes have their single spar forward of the center of area of the wing, btw.
Similarly, the dynamic COE will be located forward of the static COE; this is trickier to find since it must be weighted by sail effectiveness. Sails affect those around them, so all sails are not equal, even if they have equal area. As an approximation, I shift the static COE forward to the 1/4 chord point just like for the CLR, thus finding the approximate dynamic COE. If the dynamic CLR and the dynamic COE line up, then the ship will (probably) tack and wear. If the COE is aft of the CLR, the ship will weathervane, making tacks easy and wears difficult. The opposite situation makes wears easy and tacks hard.
4. I place the finkeel (all my ships have one) so that it's impact on the dynamic CLR makes the dynamic CLR line up with the approximate dynamic COE. I always plan for adjustment of the finkeel location, though. That is, I make provision so that I can shift it forward or aft as necessary to get the balance I want ie. so the ship will both tack and wear.
5. Even if you get the CLR and COE lined up, your ship may still have maneuvering difficulties, though. For example, my topsail schooner Aldebaran won't both wear and tack, no matter where I place the fin keel, unless she is flying some square sails; tacking is easy under f&a only, but wears are very difficult. This is because I can't let out the mainsheet far enough to get the main boom perpendicular to the hull (due to the placement of the mainmast backstays). I need to be flying, at least, the foretopsail squaresail to get her to wear - I can use the drive of this sail to force the bow off wind for the wear. It works even better if I am flying square sails on both fore and mainmast (incidentally, showing me why squaresails persisted so long at sea, namely they really increase maneuverability). So, all the calculation in the world may not be sufficient to get a ship to perform the way you want *smiles*. Moving the finkeel, (or changing the rake of the masts or size of the sails, as suggested by Andrew), will give you some more shots in the locker on the way to achieving good maneuvering performance. Leave some wiggle room in your design
Hope this helps.