Auctions are tricky.
However, auction houses can plug in to specialist markets. And if something is of special value they can be very good at exploiting it so that they get the best return. Ultimately it can be pot luck, because of two or three buyers who specialize turn up and they start a bidding war, you can do well. However, this is a perfect storm. Sometimes an lot can be withdrawn and resubmitted to several auctions if there is just not enough interest. Especiallly if you put a reserve on it.
I once got a fantastic 1930s bed for 50p. A the same auction (about 1975) two guys started a bidding war for a length of garden hosepipe that went for 7.50, which was more than a new hose would have cost at the time.
Another option for attracting specialist buyers, often trade, it to put it on Ebay with a reserve.
The other thing to consider, if the photos and items of memorabilia are personal and related to specific ships, is to gift them to an RN museum. At least you will know that they have gone to a good home.