This house belonged to Joshua Hett Smith of Haverstraw. In the plot involving Benedict Arnold turning over the plans of West Point to the British, Arnold went to Smith’s house although their original assignation was in Dobbs Ferry. At that time he was prevented from the crossing the river by British gunboats. Smith knew of the plan and agreed to vacate the house with his wife and children so the two , Major John Andre and Arnold could meet in private. When a second meeting date was arranged, the plan was to row Arnold to the Vulture (a British ship) and he and John Andre and Beverley Robinson would all meet at some place of safety. However Joshua Smith who was in charge of getting Arnold to the Vulture couldn’t find a boat or anyone to row him there. On the night of September 21, Smith and Arnold had to threaten Smith’s tenants with arrest to coerce them to row Smith to the Vulture to pick up Andre. Andre and Arnold met on the West shore of the Hudson , Arnold turned over the plans to West Point at that time and Andre was actually ready to return to the ship at 4 a.m. that night but the tenants once again refused to row him to the ship. So Andre and Arnold returned to Smith’s house for the night. The next morning another incident forced them to change their plan to return Andre to the Vulture. Colonel James Livingston decided to fire at the Vulture from cannons which he had secretly moved to Teller’s Point. The Vulture moved farther south forcing Andre to make another way back to the British lines.
Arnold now returned to his headquarters, the house called Beverly in Garrison leaving Andre to find his own way back with only some passes from him stating that "Mr. Anderson" could pass the American lines. Smith insisted they travel by land and that Andre put on a disguise. They crossed the Hudson at King’s Ferry. They spent the night at the Miller house. The next day they passed several sentry points and Smith left Andre to continue to White Plains on his own. When Andre was stopped on the Tarrytown road by John Paulding, David Williams and Isaac Van Wort , he was captured and eventually hanged as a spy.
After the exploits of Arnold and Andre were known, Smith was arrested and put on trial. He was acquitted of treason but Washington ordered that he be held in jail in Fishkill. He escaped in women’s clothes and later moved to England. (Traitor, the Case of Benedict Arnold by Jean Fritz, New York, G.P. Putnam’s Sons, 1981