Richard, I sympathise entirely, makes you want to bang your head against the wall!
However, if you are transferring your ISA then the provider to whom you are moving to will do all the work for you, just give them your full details and they will contact the existing holder and make all the arrangements. No need to contact your existing provider at all. You mustn't withdraw the money yourself otherwise you will lose ISA status as you probably know.
Other things to watch out for are exit penalties by your existing provider and of course you need to be sure that your new provider will accept transfers in. Some don't, others want you to take out a new ISA with them first before they will accept transfer of an existing ISA.
As they say, 'Revenge is a dish best served cold!'
Colin