Barcelona and Madrid are both great cities. I feel very lucky to have spent a lot of time in Spain. Some other cities to visit: Salamanca, near Portugual, has beautiful architecture. The Basque country, too. San Sebastian is possibly the most beautiful city in the region, but Bilbao is worth visiting for the Guggenheim museum and old town, and the little seaside village of Hondarabbia. Outside Madrid are towns like Segovia, Toledo, and Avila, Toledo is probably the most famous. And Andalucia in the south, Grenada for the Alhambra, and Cordoba is very charming. Seville is very touristic but the old quarter is fantastic and you have to visit the bull ring for a tour!
The Lonely Planet guide books are always an excellent source of info and they tend to be honest, too—some guidebooks manage to find charm everywhere, but LP tells it like it is, so you don't waste time or money. As for taking the train, I found buses to actually be better. They are quite cheap and reasonably comfortable. A friend and I took the bus from Madrid to Grenada, and then travelled by bus all over Andalucia, and it was fine.
Missed the Picasso museum in Barcelona! I spent too much time in the old town...