Yeah, this is like saying that an ophthalmologist, who knows the physiology of eyes, would be better at judging the visual appeal of a painting or a fashion design.
What does knowing all about the olfactory organ have to do with having a "correct" judgment about something subjective like, "Do you like the smell of this perfume"?
An appropriate expert for this matter would be a perfume critic or someone else with deep knowledge and experience with that art form.
However, it would still be a weird argument to say, "Because YOUR subjective opinion differs with the subjective opinion of an expert in the field, you must have a defect in your sense of smell."
If I think a movie is good, even though an expert movie reviewer thinks it's bad, is their a defect in MY sense of movies? Maybe there's a defect in HIS jaded sensibilities.