lizabeth Friend Brown Potter was the daughter of James Brown Jr. of the Universal Friends' Society, and of Anna Maria (Clark) who was raised in the Friend's household. Elizabeth was the stepdaughter of Peter S. Oliver, who married Maria after her first husband's death (she had married Brown while in her teens and he was in his 60s). And she was the wife of Edson Potter, the adopted son of Jeptha A. Potter, who was the childless last male offspring of the family of Judge William Potter, who financed the Friend in her early itinerant preaching days and in her community-building dream in the western wilderness. Jeptha's mother was a Wilkinson, niece of the Universal Friend. One might imagine that history and heritage ruled in Elizabeth's household, and it did.
The lot on which this house stands was part of the three acres or so that William M. Oliver acquired as raw land from Abraham Wagener beginning in 1818. On Oliver's death the whole property was sold by his heirs to pay his debts, to George R. Youngs, himself also a scion of one of the Society's families, but who told the census taker at about this time that his occupation was "Speculator." In 1865 he acquired this huge property for essentially nothing, and sold it to William M. Oliver's nephew Peter Sutfin Oliver. The following year the younger Oliver married Anna Maria Brown and took in her four orphaned daughters. They regarded him with great affection, apparently; but held fast to their association with the Universal Friend, a woman who had died long before they were born (and after whom Elizabeth was named).
The northern third of the huge Oliver lot was occupied only by the Judge's law office, built at the same time as the house and closely resembling it in style if not in size and scale; later on it was the first home of the Yates County Bank. These small two-room professional offices were apparently endemic to Penn Yan; they are mentioned in a slightly puzzled manner by some of the many visitors who came through and described the place. By Cleveland's day (he wrote in 1874) this one was evidently vacant. "At the N.W. corner of the [Oliver] lot stands the law office of Judge Oliver which has been co-existant with the mansion and long a land mark of the village, but now perhaps rather obtrusive from its uselessness and antiquity -"
P.S. Oliver died in 1879, and his heirs wanted to sell the property. The main house was sold to local paper manufacturer William H. Fox; but the northern part was bought by Oliver's stepdaughter Elizabeth Friend from her sisters Elizabeth Maria Kern of Philadelphia and Mary Caroline Hanford of Penn Yan. Elizabeth was already married to Edson Potter. The couple had this house built beginning in 1880.
It's an astonishing and exuberant departure from the era's prevailing Queen Anne, owing much perhaps to Potter's business background, which centered on timber and wood-working. It's unknown at this point who the architect was, obviously someone of very original creativity. One can't help hoping the design owes something to Elizabeth, who went on after her husband's early death to great financial success running his businesses.
Elizabeth Potter's children inherited the property from her. In 1950 her ailing son Arnold James Potter passed the property in his will to his nephew Rodney Pierce. Five years later it finally left the family when Pierce, then a resident of Wayne, Steuben County, sold it to the trustees of Lodge 1722, Benevolent Protective society of Elks. Though it has by now long since passed out of their hands, most people in the village will remember the Elks logo on the front bay, and refer to the house as the "Old Elks."