I used to work in the PET industry and now work for a large international food company. PET (it's actually labelled #1) will leech acetadehyde, which has a fruity taste, is an approved food additive, and is found naturally in many food products (e.g. coffee, bread, fruit, yogurt). The acetaldehyde is generated during manufacture, so the majority of it comes out during the first use, and is rarely detected by people like me unless the manufacturere did poor job. If you repeatedly use a PET bottle, the amount would likely get to very low levels. There is very little of other components (which could be added for processing or color) which will come out.
However, PET is considered a "clean plastic." Other polymers like PC, HDPE, and PP will leach many more compounds. This includes unreacted monomer, antioxidants carriers, & blowing aids. I recently did a study of off-flavors that are created by PET, PP, and HDPE and can tell you that PET contributed the least amount of odor/flavor. I talked with many of the large resin companies, bottle manufacturers, and consultants during this study. I have seen studies talking about the compounds in PET bottles, only to realize that most of the compounds (other than acetadehyde) came from the closure or the liner inside the closure.
Yes, PET will yellow and degrade over time when exposed to UV (I also did a study on that several years ago). The rate of that degradation is much less than PC or HDPE. PC usually has those antioxidate additives to prevent that, and still shows more degradation than PET.
If you are truly sensitive or worried, use glass, but I think the downsides there are greater.