I have an IgE reaction to milk so that is a full blown allergy. Maybe I need to seek other advise.
Definitely get other advice, because what he/she told you is full-out nuts.
It is possible to have an allergy to one of the components of milk that makes some milk products more tolerated (for example, butter has very little casein). However, if you have an IgE-mediated reaction, even a very little of the protein can set off a very bad reaction, so even people who are allergic to only one type of protein still generally avoid ALL milk products. This isn't usually something people with this type of reaction need to be told by an allergist - they're already avoiding these types of products because they make them
sick.
An IgE-type allergy usually results in hives/flushing, wheezing, vomiting, throat closing and possibly a drop in blood pressure. Is this the type of thing that happens to you?
It's also completely possible - actually common - to have a positive result on an IgE allergy test and not have an allergy at all.