I borrowed this book from the library, probably during the 1970s, but it may have been published earlier. My vague memories include: two children, probably a boy and a girl, had magical adventures. There was a friendly witch or female magician and a good but scary male magician who was always invisible. This may have been by choice, and there was a backstory of a romantic relationship with the witch. One of the children was transported by the invisible magician and found it rather frightening but eventually trusted the magician. This was not the main plot!
The book was almost certainly British and would have been a chapter book (not a term used at that time).