The Gilbert family have lived in the castle since 1329 and in 1400's and 1500's it was enlarged and fortified in response to increasing French attacks on Plymouth. Perhaps its most famous resident ( and most notorious) was Sir Humphrey Gilbert who was one of Queen Elizabeth I's "sea dogs" . He was half brother to sir Walter Raleigh and in his early life left Devon to attend Court in London where he hoped to be noticed by the Queen. His career in the court of Queen Elizabeth was successful at first but an unfortunate manner, a scandalous private life and a pencahnt for the occult meant that he soon fell from favour and left London.
His second career as an explorer led to him founding the colony of New Foundland. He ended his days going down with his ship Squirrel in a storm off the Azores refusing to seek safety on the larger Golden Hind that was accompanying it on a crossing of the Atlantic.