I have made a lot of sun hats, and the best brim stiffener I have used is just the stiffest interfacing that they have at the fabric store. I fuse it to both the top and bottom of the brim. In other words, there are two layers of it sandwiched between the two layers of cloth of the brim. Then I stitch through all layers in concentric rings, about 1/4" apart. This seems to make a reasonably stiff brim.
The only "bad" brims I've ever made were when I tried to use linen instead of cotton, or heavy cotton. Those brims sagged, I think because the fabric is heavy. They sagged more when the air was damp. I can't figure out how the tilley hat people make their hemp hat brims not do that. Maybe they have some really stiff brim stuff.
You are probably using a synthetic fabric though for your hat.