This is a very vague memory and unfortunately I only remember a few minor details. It was a colorful children’s book with a little cottage surrounded by a moat. A little man rowed a boat shaped like a swan (or a duck) to and from the cottage. It is NOT The Old Man at the Moat, although I realize it sounds similar. The man in that book has reddish hair and is shown as rather large, whereas I think the man in this book was small and had brown hair and a brown beard. It was also a different style of illustration – everything was drawn smaller and was show as being farther away (vs. large and up close as in The Old Man at the Moat). I remember lots of pinks and purples and one picture of the cottage that showed it viewed from the left side (if you were looking at it from the outside). The cottage was also on a very small plot of land – almost like a circle had been cut out just to fit the house. I thought at first it might be Magic Afternoon (similar vibe and illustration style), but that cottage doesn’t have a moat!