The trouble is, while weather is not climate, climate is weather averaged over time and place. It's like the 4 runs from a bases loaded homer or the 0 runs from a struck out time at bat are not RBI, but RBI is an average of such events.
If the climate changes in such a way that there is more flooding, or bigger temperature swings, or more sandstorms (as examples) in some particular place place buildings will deteriorate faster, and since it is an aggregate effect of weather changes in that place, one can blame it on the change in the local climate.