They could simply

A) dislike X

B) hate/despise X

C) came to the logical conclusion, that X is bad/wrong/shouldn’t be/whatever

D) genereally mistrusting against X due to a careful nature

E) have had traumatic experience with X (e.g. Being raped/attacked by a member of a specific ethnicity) and hence totally overreacting to an otherwise harmless stimulus, even forgetting the rules of civil behaviour

Those all don’t mean there’s the medical condition of a phobia for X.

A real xenophobic has an irrational fear of anything unknown/alien. Doesn’t mean the person just hates e.g. Mexicans for no real reason. It might even like them once they get to know the better, which often just won’t happen as phobics tend to avoid the cause their phobia instead of treating it.

I just dislike the lax use of medical terms until they’re bereft of any real meaning.

So, a person who yanks “speak English!” to someone, could have many reasons to do. None are neither polite nor politically correct. While the asshole is probably just the uneducated asswipe, the phobic could be helped and probably even feels bad afterwards for being so compulsive and insulting.