The Weld family purchased the Lulworth Estate from the Howard family in 1641. For reasons that are unknown, Sutton Poyntz was not included in the sale. The Weld family came originally from Eaton near Tarporley, in Cheshire. Some time in the 1590's, Humphrey Weld, a younger son, took the road from Cheshire to London, where he did rather well, becoming a senior member of the Worshipful Company of Grocers; he was Sherriff of London in 1599, knighted in 1603, and Lord Mayor of London in 1608.

It was Humphrey's grandson, another Humphrey, who purchased the Lulworth Estate, and settled at Luworth Castle (which had been completed about 30 years earlier). It is not obvious what brought him to Dorset, but he later served as Governor of Portland Castle, and as MP for Christchurch. Humphrey had no sons, and on his death the Lulworth Estates were inherited by his nephew William, whose descendants still own and live at Lulworth.

In the 1790's, William's great grandson, Thomas, purchased the Manor of Sutton Poyntz from the Harvey family, thus reuniting Sutton Poyntz with the rest of the Lulworth Estate. Sutton Poyntz was then owned by the Weld family for over 120 years, but was sold in lots in 1925, as part of the planning for a triple dose of death duties, soon after the death of Reginald Weld aged 81, with the Estate being inherited first by Reginald's 69-year old brother and then by a 71-year old cousin.

Thomas Weld, who purchased Sutton Poyntz, had an older brother Edward who died quite young. Edward married twice, the second time to a girl called Maria Smythe. After Edward's death, Maria married again, to Thomas Fitzherbert who also died quite young. Maria Fitzherbert had one other marriage (albeit not officially recognised) to George, Prince of Wales, later King George IV.

The Welds of Lulworth were of the Catholic faith. Thomas Weld's oldest son, another Thomas, took the priesthood, and in 1830 became the first English Cardinal in well over 100 years (if one does not count Henry Stuart, brother to the Young Pretender Charles Stuart, who was Italian by birth and upbringing). The previous English Cardinal was Philip Howard, a great-great nephew of Thomas Howard, 1st Earl of Suffolk, who had owned the Lulworth Estate and Sutton Poyntz.