Have you got Norberts book? Tells you in the back appendix the right dimensions for the sizes you are likely to use in a sub.
In case you haven't, assuming the bore is fairly decent you should aim for about 12-15% compression of the o-ring thickness e.g. a 4mm thick o-ring will be squished down about 0.5mm, which would give you 1mm of compression over the diameter, as the ring is compressed on both sides. Maximum compression for an o-ring is generally about 25% of the thickness, but you'll find it very difficult to push the seal home with that level, and you certainly shouldn't require more than 20% unless your bore is very poorly toleranced.
You also need to make the o-ring gap wider so the o-ring can squish into it (the rubber doesn't technically compress it extrudes). So for our 4mm o-ring we would want at least a 4.5mm gap assuming 0.5mm compression, as the ring will expand 0.5mm width wise as it forms an ellipse when squeezed.
You can use Nitrile or silicone o-rings. The latter are softer, but they can be more fragile than nitrile, and they're typically more expensive and a little harder to get hold of.
Does that make sense, or sound like double dutch?