n 1818, when William Morrison Oliver bought these three acres from Abraham Wagener, the property was just raw land, a little knoll where the local people came to pick huckleberries. Oliver, who had arrived in Penn Yan very recently with his twin brother, Dr. Andrew F. Oliver, wanted to run for office, and at that time to be eligible for elective office, a man was required to be a freeholder.
As the settlement grew, more streets were laid off, and in 1824 Oliver rounded off his purchase by acquiring another acre (also from Wagener) behind the first two. The Oliver property thus ran through from Main to Liberty Street, with a 10 rod and 17 links (just over 176 feet) frontage on both streets.
Some time in the mid 1820s, Oliver built the stately temple-front mansion that still stands on the lot. He held on to the entire property until his death in 1865. His heirs were forced to sell the property to pay his enormous debts, to the speculator George R. Youngs. It was Youngs that broke the property up, selling off several lots on Liberty Street, and the whole Main Street frontage to Peter S. Oliver, the eldest son of Oliver's brother.
Now P. S. Oliver's wife was Mary Ann Clark, the young widow of James Brown Jr., the steward of the Universal Friend. Her four daughters by Brown were the family raised in the old Oliver mansion.
One of the daughters, Elizabeth Friend Brown, eventually married Edson Potter, the basket manufacturer. In 1880, after her stepfather's death, Elizabeth bought the northern part of the Oliver property, buying out the interest of her three sisters, all of whom had married and left the area.
The paper manufacturer William H. Fox (he built the first paper mill west of the Mississippi that used straw instead of rags as the raw material) and his family bought the mansion, with its slighly smaller lot; and Edson and Elizabeth Potter built the fantastic Eastlake dwelling next door, near where the original William M. Oliver had built his law office. It has been said that these small two-room professional offices were unique to Penn Yan; be that as it may, this one had served a number of functions for a great many years, and was regarded as extremely old-fashioned when it was removed to build the Potter House.