Hi All,
Well I've finally found the cause of the whistling. See below.
I went for a drive today with a roll of tape in hand and used it on various bits of trim while I was out and about. The whistling stopped when I put some tape over the area marked in the image.
The thing is the passenger side mirror has this gap as it's where the mirror is hinged, but there is no whistling on that side. The only thing I have noticed is that the gap on the drivers side mirror is slightly wider than it is on the passenger side. Makes me wonder if something happened to the mirror prior to me owning the car.
The only issue I have now is how to remedy the problem. If it was a fixed piece of trim I could easily fill in the gap, but as this part of the hinge that isn't going to work.