Weird, I was thinking the exact same thing running today- wondering how far I could backpack in my Vans classics.
I spent my entire youth (and now my adult life) in Vans, not far off from Chucks. When I started rock climbing, they were actually my approach shoes of choice, especially out in Joshua Tree; Chucks, and Vans have great grip and feel on that rock.
I still do the majority of my living, including work, in Vans. My only gripe for taking them backpacking would be that they don't dry quickly. But if that's not an issue, go for it, wear the chucks. I think they'd actually cause less trouble for most people than a big blister-making pair of boots.