I'll be back later, but it sounds as though you have a particular protein allergy to fish. Have you been evaluated by an allergist at any point in time?
Your history seems really, really solid-- and as though an allergist would have very little difficulty helping you figure out PRECISE advice for avoidance.
It's great that you've kept such careful track of what makes you ill and what doesn't.
It's been my experience with a crustacean allergy that shellfish and fish is often
handled as one item in restaurants and manufacturing/sales. What that means, unfortunately, is that I have to be quite careful where I get my fish from-- cross-contamination pretty much always makes me pretty ill.
It sounds like you may have a reasonably high threshold for your allergen, though-- which would explain the 'sometimes it's okay and sometimes not' phenomenon.
I'd also talk to an allergist about whether you need to carry epinephrine. If you plan to eat fish from sources with possible
cross-contamination, given your symptoms, you might need to carry emergency meds just in case.