What about taking a pair of braces, removing the four clips at the ends, cut the elastic so that you have two small lengths say each of a few inches, attach the buckles at each end of both lengths of elastic and clip one end to your top and the other end to your belt, pocket etc so that you have one either side. A mini brace that you can clip on and off at will.
Alternatively, attach a button / loop to your tops & pants so you can button your shirt in so to speak.
Maybe you could use velcro in a similar way or safety pins.
What about wearing two light weight shirts at the same time. Would they both ride up or would one stay on your back.
How tight is your pack againts your back? Is it moving with each step so that it drags the shirt up with each step? Could adjusting the tightness improve things.
I haven't had the problem you describe so these were just the thoughts that came to mind, so no idea if they would work in practice.