For anal-retentive completeness: A passport isn't the only option. Merchant Marine IDs, US Military photo ID, and airline transport crew IDs are some of the alternatives. But, really, for you, a passport is needed. The passport cards are smaller, cheaper, and more waterproof than the full passport book. But won't do anything for you outside of North America. Does anyone know if you can have both card-style and booklet-style at the same time?
Here's another thing to consider if traveling with kids: I got surprised when on a father-son road trip with my 9 year old on the Alaskan Highway from Kenai, Alaska to Portland, Oregon. Canadian Customs/Immigration asked if I had custody papers with me. "Er, no. We're married. Happily so. She knows we're on vacation together. You could call her and ask - here are her phone numbers." That worked.
It didn't occur to me because numerous times the boys have gone one way and the girls another in the Caribbean, England, Germany, and Eastern Europe. But in an airport, you're instantly more respectable than on the road (even through we had passports, new car, full insurance, are white, clean shaven, employed). So in the future, I'd have a notarized letter from my wife that she knows and is cool with international travels.
I doubt that would be an issue for BPing across the border. But once you hop onto the train or bus or a car, you're scum again and might want more paperwork along.