I remember Manksters Akula had some problems with stability on its first outing. Needed some more foam up top to make it stable. An idea passed on to me by Ron Perrott was to add free flooding tanks above the waterline. The trick with these is to make the holes on the bottom of the tank small, so they take a little time to fill. That way if the boat rocks and rolls on the surface the tanks won't fill, therefore acting like buoyancy tanks, and help keep the boat stable, but when you fill the tank to dive, they will flood, thus minimising displacement required.
You would need to experiment with the hole size depending on how long your tank takes to fill, but this sounds like a very neat way to dial in a bit more stability if it's required, whilst keeping the ballast tank volumes small(er).
If you want to try a practical experiment, get a small drinks bottle, drill a small hole in the top, say 1mm, and a larger hole in the bottom, maybe 3-4mm, try pushing the bottle underwater, and you'll notice that it has a strong upthrust, and very little water is taken in.