There's a few ways you could handle this. I'm assuming you have to protect the long term relationship with the allergist?
A home isn't a lab and kids aren't cooperative test subjects. You could point out a couple of items posted by Dr. Wood, a Johns Hopkins allergist who himself has a peanut allergy. On his site is a note about how hard it is in reality for younger kids to manage allergens because they will spread allergens. He himself had his worst reaction from cross-contamination from a colleague's wife's cookies, who assured them they were peanut free. Turned out to be a spatula used to lift pb cookies off a tray were used for the non-pb cookies.
If it were me I'd tell him I can do a lot but I can't control the world, my house is not a lab, my kids are not test subjects and I don't get to punch out on food allergies at 5:00 pm.
Having said that we do live with most of our allergens in the house. I say 'most' because we have so many, particularly my youngest child. There are two we are most careful about: peanut and milk. I don't have either of those in the house because my children's eliciting doses to anaphylaxis are so minute. I also have to go according to a management style that is possible for my kids given their age and developmental stage.
Roasted peanut proteins tend to be the most durable. It takes more wiping, chewing, washing than baked egg or baked milk. It's also a much more narrow category of food item than dairy, egg, wheat, soy (because it's used in so many things). That's something to consider. Now if your child has a wide threshold for peanut then that's different, and no matter how high the numbers they tell you absolutely nothing of threshold. If all you have are high numbers but no history of reaction, especially anaphylactic you may have more room for contamination.
The allergist is going on conventional wisdom based on correlation, I don't believe there's been any publications that provide a direct causation from delayed introduction. In fact, I believe that never became truly set amongst the entire medical community if for no other reason that no one knows how sensitization in allergic disease happens exactly and why some proteins and not others. He's mistaken outright if his idea is that specific allergens run in families rather than allergic disease itself. It would behoove him to keep an open mind about that and not assume because you're not introducing lobster at age 2 then your child will become allergic to lobster. If it were that simple this whole thing would be solved already.
Dr. Sicherer was our previous allergist. I asked him about the idea of highly allergenic foods, introduction and 'keeping them in the diet'. His response (which I am paraphrasing) is he believed that was a slight misinterpretation of OIT data where tolerance was cultivated. He said although sensitization can occur at any time as a reasonable butt-covering possibility, that if you're not allergic to something to begin with there's no value in 'keeping it in the diet'. One simply isn't allergic to it. For reference I was asking about crustaceans and the idea that our son will eat them but not consistently. He did not see that as a concern. This was probably mid 2011.
One good compromise could be if older child is peanut allergic only to start using tolerated tree nuts that all the kids can eat. That should ease the allergist's emotional state a bit, open up new foods for all. The kids w/o allergies could avoid overt peanut until a little older when all the kids can take more responsibility. It's not healthy to create a situation where kids get disciplined due to allergen management.
I would for sure want to find out about peanut for the other kids by school age.