No arguments against your thoughts - my thoughts would be the same. Hi-vis is always safer except when someone is shooting at you.
But my feelings differ. On a hike, I can often look for miles and not notice any other person, tent or structure. That would be true less often if more people wore hi-vis outer wear.
I'm not saying anyone is wrong wearing day-glo, not at all. But I personally try to have a lower visual impact.
Admittedly, at least one piece of high-vis clothing is a safer option because you can then be found more easily.
I do wear blaze orange during hunting season. More so, I put some on the dog (a black dog can look an awful lot like a black bear through the trees).
And when road-bicycling, I'm ALWAYS in high-vis clothing.
Edited to add to HWC's thoughts: +1 on use around town. I just don't wear blaze orange, fluorescent yellow, or vomit green to the grocery store.