I think it’s a matter of degree. If the tag is sharp/stiff/crumpled enough to actually redden or mark your skin or is especially itchy, most people will notice and be bothered. If it’s one of those flat, soft, shiny tags lying correctly, most people don’t notice them.

I don’t always remove mine because that sometimes makes it worse, but I wear my undershirts inside out, which sidesteps the problem. For shirts I wear without undershirts, I’m okay with most of the soft tags at the neck most of the time, but care instructions are sometimes sewn into the side and that drives me fucking nuts.