Many years ago, I bought some Thinsulate Liteloft 110 to make some insulated salopette liners. When it arrived, I opened the bag up, and revelled in the lovely soft warmth... So much so that I stuck my face in it.
As it turned out, this wasn't a great idea, and my forehead came up in a rather angry, red allergic reaction. It subsided fairly quickly. My conclusion was that I'd been 'spiked' by a stray fibre, and this had triggered an immune response.
Some PL comes with a scrim fabric. I'd be tempted to suggest looking for this, and putting the scrim on the PowerDry side.
Of course, I might just have been unlucky or especially sensitive to fibres of that nature for some reason, and you might not have any trouble at all.
* I used Pertex 4 for the shell, and have never had any repeat of the problem. They've been in use, on and off, since 1997, and survived almost being cut from my body by the staff in A&E in Briancon, France. Fortunately, I pointed out the nice series of velcro closure tabs down the outside, just in time...