The maternal grandmother of James and Robert Haldane, a former great society beauty, looked after the children. The boys two uncles were experienced military men; one a retired Colonel, who later became their guardian, the other a Captain (soon to be Admiral) in the navy. As experienced men of the world they understood the importance of a good education more than most of the Scottish gentry, so they ensured that their nephews received the best they could. They were both sent to the grammar school in Dundee. However, their grandmother died in 1777, so their uncle, Admiral, Viscount Duncan, decided to send them to board at the High School in Edinburgh, which had a good reputation. Two of their classmates were John Campbell, the African missionary and Greville Ewing, the minister of the Independent congregation in Glasgow — men with whom the Haldanes were to be later intimately connected. The boys missed a stable home life, but otherwise they lived an exceedingly privileged one, living like the wealthy gentlemen they were.
The building is now part of Edinburgh University.