I’ve never considered it a swear word. It’s more of a euphomism for “shit”. My criteria for “swear” word is whether it is something a kid can say in school with getting yelled at. I realize this may have changed since I was a kid.