Yes, it's a progressive thing-- but it's also WAY individual in terms of whether or not you're one of the "lucky" individuals who is reactive to the possibly cross-reactive fruits.
Not everyone will eventually react to them, by any means.
It's a high percentage of latex allergic people (relatively speaking, that is-- for example, only a small percentage of people with peanut allergies react to other legumes via this kind of mistaken identity kind of thing-- about 5%, actually... and with latex allergy, that percentage is higher-- more like 30-50%) who are reactive.
It's not clear from looking at the literature in this area, though, whether ongoing exposure to things that you DO tolerate 'encourages' your immune system to begin thinking of those related things as threats eventually.
It seems (anecdotally) to be either you do or you don't, or maybe (like me) only when your allergy cup is otherwise REALLY full. So I avoid bananas now if I know that I'm having other allergy stuff going on that would make it more likely-- but I don't actively avoid bananas otherwise. Does that make sense? It's more of an individual determination based on my quality of life, though-- my family already works around a LOT of other food allergies, and banana/kiwi are some of the things that don't cause other family members problems with oral-allergy-syndrome. It is what it is, you know?
Has your allergist talked to you about using epinephrine? Odds are good that if you're having asthma as a result of latex exposure-- s/he SHOULD. That's probably not asthma-- and if it gets bad, it won't respond like it, either. Please take a look at our main board and the discussions about when anaphylaxis LOOKS like asthma. It's really a challenge to manage that.