iles Benham built this house about 1820, nearly as soon as he arrived in Penn Yan. There's some evidence that it was used as a "public house" or inn in 1823 when the first Board of Supervisors of the new County met in Miles Benham's tavern to conduct business. Benham lived here and with his partner evidently used the house as a shoe and leather shop, a commercial outlet for the tannery down by the creek in the rear of this property. Bordwell left the area and Benham some years later sold his house to John C. Babcock. The tax rolls show Benham as owner in 1853, Babcock in 1856 and James Dwight Morgan about 1860. The property maps show the innkeeper Mrs. Cobb at this location in 1854, Babcock in 1857 and Morgan in 1865 and 1876.
Cleveland says Morgan bought the house in 1857 from John Babcock. Actually, Babcock's wife Celestia sold it to Morgan's wife Catharine in that year, as the "lot and premises with the brick house formerly occupied by Miles Benham." This property was bounded on the north by that of Darius W. Adams, and on the south by a common right of way shared by this lot and that belonging to James Benham, and occupied in 1857 by Miles Benham's brother George.
Morgan had opened the first hardware store in Penn Yan on the "Morgan corner," where 101 Main Street stands today. This was in 1831, just when business was moving in earnest to the foot of the street. In later years J. D. Morgan and his wife moved to the Rice house, built in 1855 on empty land at the corner of Clinton and Hamilton Streets.
The Main Street house stayed in the Morgan family's hands until 1884, when it was sold by Mary Jane Morgan to William Carey Allen, and identified as the same house occupied by James D. Morgan and his wife Catharine at the time of their deaths. In the meantime Mary Jane had added property at the back. She used the additional land for a hennery; it abutted on the Benhams' duck pen and the penstock of the old tannery. Both pieces were included in her sale to Allen in 1884.
In 1909 Dr. Frank Sampson purchased the house, and later on it was occupied by both Drs. Sampson, Frank and Alan. Later still, in 1929, it became the home of E. Carlton Foster and John A. Hatch, also both doctors. In 1950 the building formally became the property of the Foster-Hatch Medical Group, one of the first group practices in the country. Today it houses a number of professional offices.