That's an easy one to answer- for every halving of volume you must double the pressure.
If you have say a 1 litre ballast tank/bag and you have a 1 litre reservoir, then your pump will need to compress the air to one atmosphere (14.7psi) above ambient pressure. If you wanted to squeeze the same volume into a 500ml pot, then you would need two atmospheres (29.6 psi), and so on.
In the case of the diaphragm pumps, they make about 1 atmosphere, beyond that you're pushing your luck, and they're inefficient at pumping higher pressures anyway unless you series pumps up.
One of the problems with older submarines, and indeed some newer ones is high freeboard, and the need for large volume tanks. If you're using a recirculating system then you need fairly high volume reservoirs or you have to run at higher pressures, which comes with its own set of issues.
If it's difficult to find the space for a reservoir, then I wouldn't rule out a partially vented tank system, perhaps to get the boat to decks awash, then take the tower under with a recirculating system based on any system of your choice. The diaphragm pumps work well with this system unlike piston based pumps, which can't afford to ingest any water.