First, the weight is being carried by your feet and legs, regardless of whether it's in the pack or worn on the body. That's why the total "skin-out" weight is the most important.
Second, weighing what you wear hopefully reduces "cheating" by carrying items in your pockets but not weighing them because they are not in your pack. These little items can add up! I personally include them in pack weight, but not everyone does.
Third, everything you take, no matter how small, should be on that list, whether worn or carried. That way you won't forget something important. If you have it on the list, you might as well weigh it.
Fourth, there are definite weight savings to be had in clothing and footwear on your body as well as in your pack, and you won't know what they are until you weigh the alternatives. Two examples in which I was able to make significant weight reductions: Switch from boots to trail runners, 1 lb. Switch from aluminum trekking poles to carbon fiber, 0.5 lb. Yes, I definitely noticed the 1.5 pounds afterwards!
Fifth, your total clothing, both worn and carried, should be considered as a single system, so you're not taking more (or less) clothing than you need.