Any ESC from back then will pulse the motor at transmitter frame rate, 50 times per second, and the motor will produce sound at that frequency. More modern ones run at a higher frequency, and whistle, or howl, depending on how you regard such sounds. Both are handy means of letting you know that something is trying to happen. Some of the latest run at yet higher frequency, and appear silent to humans, but probably annoy the dogs in the park being walked by their people.
Relay chatter - are the batteries in a good state? Relays impose a considerable load, and might be causing the supply voltage to vary. The control part of the ESC relies on stable voltage to sort out what is wanted of it, varying voltage produces varying answers, such as whether the relay should be operated or not. The system should be set up so that the relay is only required for reverse.