Frankie is near enough. I, too, left the gentlemans club before the broadband provision, but from the knowledge I gained about wideband distribution, I would guess that there is a broadband "box" in each exchange building. Maybe boxes, depending on how many major ISPs are interested in that area. Smaller players might just rent space from the big guys to connect through to their own boxes in more central locations, and what you get is traffic choking as the attempt is made to squeeze several gallons a minute through a system only intended for a few pints per minute.
Swapping telephone service provider, especially if you keep the same number at the same address, basically just involves changing accounts departments, changing ISP might involve physically re-routing your line through a different box, and this fresh box might well have an entire different set of operating parameters. You only find out if this is an improvement or not later on.
If a line is routed to a specific building, it is very unlikely to be routed to one further away, because that is an expensive process, and usually pointless. Having said that, it is not unknown for a small exchange to be incorporated into a nearby larger one. The local cable network would remain centred on the manhole outside the old one, but be physically re-routed to its new connection point - if it was in an area linked numbering scheme, there would be no change of number. If your local exchange has suddenly sprouted "for sale" signs, this could be a clue. Usually there is something about it in the local press.