I have a hard time with this question myself.
As a whole I prefer muted colors to blend into the environment with (grey, black, dark greens and blues). However as I'm moving more into mountaineering type adventures I'm also starting to see the benefits of all those outlandish neon reds and yellows. The visibility they offer provides a small safety factor.
I think the compromise I'm finding is muted colors for most of my base and mid layers with bright colors for outerwear. That way if weather turns foul, (only time I'll be wearing my outerwear) it's better to be seen. This helps with everything from keeping a group together, potential rescue, to even unexpected hunting season safety.
This has come in handy once when I wore a safety orange colored windshirt in a misty dusk storm and stumbled upon some elk hunters. I hadn't realized my weekend was the first day of the season and was quite glad I stood out from the environment so much.
For shelters I think bright ones for winter/alpine conditions are best (highly visibility in snow storms and makes for better landscape photography than white or stealth) and any other conditions stealthy colors are preferred.