We're going to Madrid, Spain, in April. What clothes should we take for sightseeing & casual dining?

We're middle-aged & very casual, usually go around in T-shirts & jeans or shorts. We live in the desert (Phoenix, Arizona) & feel chilly when the temperature drops below 75 F (24 C). We want to pack as few clothes as possible but don't want to offend local people with an overly casual look when visiting museums, cathedrals, & such. What do you recommend we pack?