Around 1400, Ritske Jelmera, a developed man belonging to the Frisian nobility, built a castle in Ballum that was inherited by Hayo Jelmera, later known as Hayo van Cammingha. After those years followed a two hundred-year reign of the Camminghas. At that time there was some disagreement with the Sea Beggars, but in general they managed to remain neutral on the island, even in the war between Spain and the Netherlands. This neutrality was recognized by the States of Friesland, but not by the States General and the States of Holland, who themselves believed they had a claim on the island. Until 1600 it remained restless on Ameland, because the islanders believed that the rulers of Ameland had too much power. This was put to an end with a heavy hand and in 1629, when Spain again recognized Ameland’s neutrality, peace returned. The last Cammingha died in 1681 and the island passed into the hands of the Theo Schwartsenberg Hohenlandsberg family, who sold it in 1704 to Johan Willem Friso, hereditary stadtholder of Friesland and Prince of Orange. After having changed occupants several times, the castle passed into the hands of merchant Jan Scheltema, who had bought it for demolition, and in 1829 it was razed to the ground.