Brett, if the baffles started out poofy and aren't now, it's pretty unlikely that you've lost any down out of them.
I once had a bag that had obviously lost a bunch of loft, so I thought I'd wash it to revive it. I washed it, dried it thoroughly, and found that it still didn't loft as well as it should. Short version? I had to wash it twice more (no drying in between) to get all my oily hiker funk off the bag. Generally speaking you don't want to overwash the bag, but I'm bettin' you'll be fine washing it another time or two. And do make sure to dry it thoroughly w/the balls, etc.
If it's still a problem, give WM a call...