Boo and TwinTurbo have given you excellent advice.
I'd seriously press your allergist on the subject of food challenges to some tree nuts. Find out (for certain) what you can-- and cannot-- have in your home. The advice you've been given sounds VERY much like "let them eat cake" coming from your allergist; some of them truly can't quite grasp what it is to live with food allergy, and others can't quite grasp what it means to live with a very low threshold dose (we've had both experiences with allergists, even very good ones).
So ultimately, day-to-day management and avoidance? Your allergist isn't the expert. YOU are.
So gather information, but then make your own decisions about risk/benefit, IMO.
Some allergens we keep in the house, and some we don't.
Not in the house: crustaceans, cashew, pistachio, peanut-- the reasoning is that the eliciting doses involved are either too unpredictable or too scary-low and/or the reaction history is very, very scary-severe. A major-- or in a couple of cases, minor-- error with one of those could cause death no matter what transpired afterwards. So.
Not in the house because they simply don't NEED to be, and they are likely anaphylaxis triggers-- mango and walnut.
Not in the house in some FORMS: soy, almond, egg.
In the house, just avoidance for those who are allergic-- several things, mostly fruits and vegetables; milk.
I mention this to let you know that most of us have a highly idiosyncratic household protocol surrounding this stuff. Mostly, it's stuff that we've figured out taking into account our own life experience, the needs and desires of OTHER members of the family, cultural preferences, etc. etc.
I grew up in a house that did NOT restrict my allergens-- ever. Now, did that force tolerance? Well, maybe. But I developed a shellfish allergy as an adult, and I'd been eating shrimp and other crustaceans regularly my entire life.
Allergic is allergic, and if you aren't destined to be, then so be it, IMO.
While topical reactions to things like poison ivy are not in the same category as IgE-mediated food allergy, it's an instructive example-- additional exposure just racks up ticks on your life-o-meter there. About 80% of people are born with a magic number. Reach that and you're allergic. For some unlucky souls, it's 0 or 1. For some others, it's 987, or 46. No real way to know until you learn the hard way.
I do wonder about that given the number of ADULTS with nut or shellfish allergy that seemed to develop after a period of exposure. If exposure is protective then that shouldn't happen-- but it clearly DOES.