I did backpacking, mountaineering, and hiking for 38 years before being involved in scouts. With three daughters, I took them backpacking as long as I had influence over them, but they really were not into it. I started scouting with my son 4 years ago, and he is now aging out of scouts, almost got his eagle done.
We barged into a car camping troop, and I started leading backpacking trips. Half the troop loved the change, half the troop and parents hated it. They aged out and are gone, so now the troop does a lot of backpacking and its the norm.
My son is now more interested in friends, video games, girlfriend, and sports. We include his high school friends on all the trips we can, and go on non-scout backpacks. Most of the scout backpacks I lead, he does not go on.
If you get a lot of enjoyment out of outdoor stuff, keep doing it. It doesn't have to be with scouts, but scouts give you a bit of cover because you can claim some kind of humanitarian credit. Eventually you will have to come to an agreement with the wife, such as including her on trips, or her going to visit her mother while you are on a trip.
You can't make a kid go hiking for long. Including his friends is one enticement to add, and if he doesn't connect with scouts in his troop, going on non-scout trips is fine. If he has an interest that can be stoked while on a backpack, that helps, like herptology, photography, fishing, peak climbing, or cooking.
Kids hate to change troops, but maybe you and he could go to a lake that another troop is going to, and get to know some of the other kids on the trip. Maybe some of his classmates are in another troop you could visit.
It would be a tough one if your outdoor trips depend on your son's involvement, or on your association with scouts.