If you have polyester inside, it doesn't stretch like fleece, or does it?
If inner doesn't stretch then you'de have to make it bigger to fit, but you want the fleece to be a little tight so it stretches to fit so it stays on your head in the wind. And if there's an inner fabric, there might be less friction between that and fleece so it slides off more.
I don't care if the unfinished seam shows, but that's just me.
But, I don't mean to discourage you. Hat requires so little fabric you can afford to make prototype, try it, modify, iterate until happy. Feed back your experience.