Thanks for the replies.
I suppose I could go without a seam there, but I'd really rather have something there. I was thinking maybe instead of one long seam around the neck I could do little short groups of seams, that way it holds together and when I pull it over my head it would expand in between the groups of seams. Anybody else have tips or tricks, or experience with doing something like that?
Ken: That's totally what I'm going for. I had never seen one of those until you just said that and I google searched it. I had no clue they made something like this (I mean I was pretty sure, I just didn't know what it was called or anything like that). Thanks for that! I may just use the fabric for something else and shell out the money for a buff haha.