The original castle was a small fort built around 1320 by Clan MacDougall, who were then Lords of Lorn. Around 1388 the Stewarts took over the Lordship of Lorn, and it is believed that they built the castle in its present form around the 1440s. The Stewarts' relative King James IV of Scotland visited the castle, and a drunken bet around 1620 resulted in the castle passing to Clan Campbell. After the castle changed hands between these clans a couple of times, the Campbells finally abandoned it about 1840 when it lost its roof. In 1908 the castle was bought by Charles Stewart of Achara, who carried out basic conservation work. In 1965 Lt. Col. D. R. Stewart Allward acquired the castle and over about ten years fully restored it. Castle Stalker remains in private ownership and is open to the public at selected times during the summer