just a thought, could you place the hammer in threaded inserts into the bench top, then when needed you can bolt the drill into the inserts, and then they could also be used for your scroll saw
or maybe the furniture ones that go into a hole at right angles, drilled in from the edge of the bench