For a period of about 100 years, essentially corresponding with the 13th Century, Sutton Poyntz was owned by members of the Poyntz family, who lived initially in Gloucestershire and then at Curry Mallet in Somerset. Between 1315 and 1361, Sutton Poyntz was not owned by the Poyntz family, being part of the dowry given to the widow of one of the Barons Poyntz of Curry Mallet.

In 1311, Sir Nicholas Poyntz died, and Sutton Poyntz is known to have been among the lands that he held at his death. Hugh Poyntz, his son and heir, was a minor (aged about 18), and therefore became a ward of the King, as was the custom then. The King, also in accordance with the custom, 'sold' the right of wardship, probably to the highest bidder; in this instance to William le Latimer.

After Sir Nicholas's death in 1311, Maud (also known as Matilda), Sir Nicholas's second wife, re-married, to a Hertfordshire land-owner called Sir Roger de Chaundos, 1st Lord Chaundos (or Chandos as the name later appeared). It appears that Sutton Poyntz was among Dorset land gifted to Maud as a dowry on her second marriage. A document dated 1315 in the Close Rolls lists the properties granted to Maud in dowry; they include some parts of the manor of Sutton Poyntz (Putton and East Chickerell), but do not name Sutton Poyntz explicitly. However later documents (1330 and 1346) do explicitly state that Sutton Poyntz was part of Maud's dowry.

Little seems to be known definitely of the origins of the Chandos family, but Sir Roger lived in Herefordshire, and was Sheriff of that county, and also Sheriff of Glamorgan, on various occasions. The Sir John Chandos who fought at Crecy was probably a cousin to Sir Roger. Sir Roger was an ancestor, via a female line, of the Dukes of Buckingham and Chandos (to whom Jane Austen was distantly related), including the 1st Duke of Chandos who was a patron of Handel, and to whom the Chandos Anthems were dedicated.

Sir Roger and Maud had a son, Sir Thomas, and a grandson Sir John (who married a daughter of Guy, Lord Bryan); after this the male line died out. The title of Lord of Chaundos continued, passed down via a sister of Sir John who married into the Brydges family, a descendant of whom married the First Duke of Buckingham and Chandos.

The grant of property as dowry was only for the lifetime of Maud, so when she died, in 1361, ownership of the manor of Sutton Poyntz returned to the Poyntz family, and specifically to Sir Nicholas Poyntz, the step-grandson of Maud.