In Gotham, A History of New York City by Edwin Burrows and Mike Wallace, the development of building on 20th street is discussed. In the 1820's Clement Clark Moore donated a parcel of land which had been an apple orchard west of ninth avenue between 20th and 21st streets to the Episcopal Church. The General Theological Seminary was built on this site in 1827. Nearby another lot donated by Moore to the Church was to become the site of St. Peter's Episcopal Church.
Another church, the Unitarian Church of All Souls was erected in 1855 at the corner of Fourth Avenue and 20th Street.