This mountain was undoubtedly named for its conical shape which resembles an old fashioned cone or loaf of sugar. The mountain was part of the Beverly Robinson estate at the time of the Revolution and served as a back drop to his mansion. In the late 1800's William Henry Osborn bought up all the land around Sugar Loaf Mountain. The Osborn castle was built on Castle Rock, the mountain just north of Sugar Loaf. In 1974, William Henry Osborn's descendants donated Sugar Loaf and the surrounding land to the Taconic State Park Commission.
In 1866 Benson Lossing describes a visit to Sugarloaf Mountain in his The Hudson From the Wilderness to the Sea.
"It was mid-autumn when we visited Beverly House, and the Sugar-loaf Mountain, at the foot of which it stands, exhibited those gorgeous hues which give such unequalled splendor to American forests at that season of the year. The beautiful hues of the foliage of the maple, hickory, chestnut, birch, sassafras and several other kinds of deciduous trees in the Northern and Middle States, seen just before the falling of the leaf in autumn, are almost unknown in Europe. ...From the summit, is a grand and extensive view of the surrounding scenery, which Dr. Dwight (afterwards President of Yale College) described in 1778, as 'majestic, solemn, wild , and melancholy.' Dwight was then chaplain of a Connecticut regiment stationed at West Point, and ascended the Sugar Loaf with the soldier-poet Colonel Humphreys."