Did this happen in a meeting with the principal?
Keep this phrase handy: "the same as his nondisabled peers." For instance, he needs to have the same experience on field trips as his nondisabled peers. His nondisabled peers are not required to be singled out; neither should your son. He should not be singled out based on his disability. He should have the same access to field trips as his nondisabled peers.
Moving to our current state, our principal wanted to ditch out 504, too. I simply said we were not interested in removing the 504 designation, as nothing changed for his qualification. Also, I was explicit that we wanted the designation going into college.
That's all I have time for right now.
hth