The boat house belonged to the the manor house of Frederick Philipse. Frederick was the only son of Philip Philipse who inherited the third share of the original patent for the large tract of land on the east coast of the Hudson known as the Philipse patent. Philip's sister Susannah married Beverly Robinson and they lost all their land when they supported England in the Revolutionary War. His sister Mary married Roger Morris of Yonkers. Frederick's daughter Mary married Samuel Gouverneur who held land adjacent to the Philipse manor. The Philipse family also had a mill noted on a 1854 map. It was situated where there is presently a golf course at the intersection of Philipse Brook Road and Snake Hill Road.