Seeking a picture book, illustrations probably pen and ink, blueish tones. A large, poor man lived in a small house. A rick small man lives in a mansion. They are each becoming tired of their own houses, so agree to swap, expecting to love each other’s house. They initially do, but ultimately discover that the preferred their own house. Date: I was born in 1976, so earliest read would have been 1979-80. Location: I live in the UK (England).
Maybe Benjamin Budge and Barnaby Ball by Florence P Heide 1979
It is as you describe, but they don’t go back to their old houses.