You won't prevent them all, but the least you can do is wash your face at least once per day and keep the balaclava as clean as possible. On multi day trips everything may/will get oily from facial oils.
Guess you can get a different balaclava if that one bothers you. I prefer sorta loose balaclavas since tighter ones (beanies, etc too) seem to hurt my ears if worn a long time.
I have oily skin and get them too. Join the club? lol