A race track was established at Paulus Hook (also known as Powles Hook) around 1770 by Cornelius Van Vorst. The land was almost an island at high tide. It is the closest land to Manhattan and so became a ferry and trade route between New York and Philadelphia.
This is most likely the Hudson Hotel built by a group of investors after the revolution. These investors included Joseph Blomfield, then Governor of New Jersey. Paulus Hook was also where Robert Fulton established his shipyard and so was the center of steamboat activities for many years.