Those fake chamois-type towels (like shamwow) are the bomb.
You can get them for about 2.50 ea shipped on ebay, and they are made in Germany just like the shamwow ones. I use them cut in half, at about 1.25oz and it's plenty big to dry my whole body, tarp walls, whatever, quickly and completely. Even wet, after wringing out, they absorb almost the same amount of water as when completely dry. Try it at home in a measuring cup. Feels real nice on top of my FlexAir pillow at night, too. Soaks up the sleep slobber!
Pack towels are so hard to handle (they roll up in your hand when wiping) and mostly too small, and won't absorb anywhere near as much water. I hate them.
I always wear a bandana on my head or neck and use it for a pre-filter and pot lifter. They are cheap and the bright ones are great for hunting season. The buff thing might be better. Always wanted to try one, but I have all these bandanas around.