The site of Lowther Castle has been occupied by the Lowther family since 1150. The house was largely built between 1806 and 1814 for William Lowther, 1st Earl of Lonsdale and designed by Robert Smirke in his first major commission. It incorporates fragments of the previous house on the site, which was completed in 1685 for John Lowther, 1st Viscount Lonsdale. It is open to the public and is a grade II listed building.
The house takes the form of a sham castle, and was known as Lowther Hall before the 1806 rebuilding. It consists of a nine-bay central block with angle turrets, a porte-cochère on the entrance front, and a larger tower in the centre, which is linked by low wings to angle pavilions; the whole building is embattled. The windows are a mixture of Gothic pointed arches and flat-topped in the Tudor style. There is a service wing to the east. In front of the house is a large forecourt enclosed a battlemented wall containing several turrets and a gatehouse.