The Greek revival-style mansion was built for whaling Captain Albert Rogers in 1843. The belvedere (sometimes called a widow’s watch) on the top of the roof was a popular feature on many whaling Captain’s homes during the mid 19th century and can be found on many other homes on the South Fork. Captain Rogers died in 1854.

In 1899 the dwelling was purchased by Samuel Longstreth Parrish; an attorney from New York City, summer colonist, and founder of the Parrish Art Museum. He began creating additions in the Colonial revival-style in 1911, and had doubled the size of the house by the start of World War I in 1914. In 1926 he moved the entire house back 100 feet from Main Street and constructed the block of brick-fronted stores that stand in front.

Mr. Parrish died in 1932 after being hit by an automobile in NYC. The Village of Southampton bought this building in 1943 from the Parrish family and used it as a Red Cross headquarters and later as a YMCA. In 1952 the Southampton Colonial Society leased the house and grounds and began restoration.

The house is filled with furnishings donated by members of the Southampton community and date mostly from the Victorian (1837-1901) and Edwardian eras (1901-1910). There are temporary exhibits on the first floor including a 1890s Parlor and a 1910 Dining Room. The second floor has a heraldry exhibit, an 1840s bed chamber, and an 1880s maid’s room, exhibits rooms of toys, silver objects made by Elias Pelletreau (1726-1810), and historic photographs from the Hurricane of 1938.

On the grounds behind the mansion is Old Southampton Village with historic structures collected from different areas of Southampton. They include a 19th century paint store, a blacksmith’s shop, a cobbler’s shop, a one-room schoolhouse, and a colonial era barn that was seized by British soldiers to lodge their horses during the Revolutionary War.