Welcome Alexandra!
Wow, this sounds very confusing and unpleasant.
First of all, there are different categories of fish, so you can be allergic to one kind of fish but not another. This can make figuring it out on your own even more difficult.
Did you have any other symptoms besides the vertigo? Vertigo on its own would be a pretty unusual presentation of an IgE mediated allergy, but one thing I've learned here is nothing is impossible.
I think you should see an allergist who specializes in food allergies to help figure this one out - and avoid fish for now.
FWIW, my MIL got bad food poisoning from oysters, and the symptoms did indeed last for about 3-4 weeks. It was a nightmare.
Good luck, and I hope you get some answers!