SIMPLE EXPLANATION - If the fabric surface is smooth, it will feel cool and if the fabric surface is rough, it will feel warm.
DETAILED EXPLANATION - Fiber deformation is proportional to the 3rd power of the fiber length and indirectly proportional to the 4th power of its diameter. That is why the micro-fiber fabrics, even if the protruding fibers are short, deform easily under pressure to a certain extent and partially copy the acting body. Thus, the contact area is always large, the amount of heat taken away form the contacting body is high and the contact feeling is cool, in spite of the smooth and pleasantly soft surface. Similar phenomenon appears after the enzymatic or chemical treatment of fabrics: this relatively drastic action results in the disintegration of the fiber ending into several fine micro-fibrils, which behave as micro-fibers.
If the surface fabric is coarse or is an axial curled fine fiber it will feel warm to you. Animal hairs, even if long, provide a warm and smooth feeling because the animal hairs are frequently curly, and their endings are tapered. Coarse surface fibers, do to their relatively low flexibility, may scratch the skin of sensitive folks. Vlad’s wool forum posts address this phenomenon (smile).
Any mechanical treating of fabrics, like brushing or carding, brings the warmer feeling. This is because the original compact and smooth plane surface of dense woven fabrics with high mass and hence high thermal capacity is being replaced by the irregular surface featuring lower mass, irregular thickness of a structure composed of some soft and easily deformed fibers. Some surface fibers are not split, and due to their relatively large diameter and short length they do not bend easily under pressure, thus conserving the surface less compressible, but full of thermal insulating air pores of low thermal absorptivity.