I cut down my last post - to avoid trying to rewrite War and Peace. Also, there are a lot of interdependencies between the rudder and turn fin (length, blade area and placement of rudder and turn fin), which are both acting in more than one dimension - as well as factors in the hull design, speed, torque reaction, prop lift, trim tabs if present etc etc. I'm definitely not an expert on any of this either, so it's difficult to give hard and fast answers. I meant to be a little bit ambiguous, as I didn't want to give something as statement of fact without being certain myself. What I meant to allude to was that I believe (in most cases at least), that adding a turn fin should reduce the tendency to flip - rather than increase it. To some extent though, it may depend on the balance of the hull itself (in all dimensions), as well as the interactions between rudder, turn fin, torque, and others. The way I see it, some of the highlights are:
1) When you turn right, the rudder will start to turn the boat. The rudder will generate a sideways force, and also a rotational force (caused by the rudder acting as a lever) - rotating the hull, and the rudder clockwise (as viewed from behind the transom - this effect having a large dependency on the length and area of the rudder, as well as the rudder offset, and torque). The rotational effect on a right turn is usually in the same direction as torque reaction - which increases the effect. The rudder - just by it's presence in the water, will generate drag and lift. The drag and lift will increase when you turn the rudder. As the boat leans into the turn, and the rudder becomes non-vertical, it also starts to act as a trim tab. The more the boat leans, the more the rudder acts as a trim tab, and the less it acts as a rudder. The rudder as a turn fin effect has been written about by John Finch. He has used angled rudders to enhance this effect, and remove the need for separate trim tabs.
2) As the boat leans into the turn, the turn fin enters the water. The lift from the turn fin should help to couteract the rotational effect of the rudder. The force from the rudder is now trying to drive the transom sideways, and the turn fin is acting as a brake in the sideways direction - creating the effect where the boat pivots on the turn fin.
3) As the rudder straightens out, the transom is still trying to slip sideways. The resultant rotational forces from the turn fin and rudder will help to right the boat.
From these visualisations, I believe that the turn fin will reduce a tendency to flip side over side as the boat leans into a turn, but there are effects in other dimensions. Especially lift (from the rudder and turn fin), and diagonal lift (from the rudder), at the transom, possibly causing the keel to "dig in" at the deepest point (maybe causing a rising spin, or an end over end flip - transom going up and over diagonally), or the possibility of a side over side flip in the opposite direction as you straighten up - especially if you overcook the turn, and need to correct. This should be lessened because you will come out of the turn faster than you go into it, and this is also (usually at least) in opposition to torque effect. This counterclockwise rotational force as you straighten up though, is presumably increased by the addition of a turn fin.
Please take these as thoughts and observations - designed to invoke thoughts, comments, additions - if anyone's interested enough (I might learn something that way).
Ian