This book was from my elementary school library, read around 1967. The book was older, 1950s or 1940s. A family that was either down on their luck, or through an inheritance, move to a house in the country that came with African-American servants. This might have been in the South and it seems as though there was an adjustment. I recall food scenes, such as how much the family enjoyed the fresh baked biscuits. This might have been their first encounter with them. In another scene, the mother twists her ankle while walking through the garden and sits alone until she was found because she was in too much pain to walk. What struck me was that this was the first book I read that was told from both an adult and a child’s viewpoint. There was a mystery involved too but I don’t recall whether the house was haunted.
It sounds kind of like The House of Dies Drear by Virginia Hamilton, which was published in 1968.