In 1908, Sarah and Charles Allis decided to build a residence along Milwaukee’s tree-lined Prospect Avenue. When they began planning, Prospect Avenue was not as densely built as it is today. Many of the families that lived in Yankee Hill were moving closer to Lake Michigan and building mansions along Prospect and in the Water Tower area of Milwaukee. One factor that probably influenced their decision to build on this specific site was the proximity of one of Sarah’s sisters — Mary Ball Camp — whose home once stood where the museum's courtyard is today.

Sarah and Charles turned to Alexander Eschweiler — a prominent local architect — to design their new home. The house he built is strongly influenced by the English Tudor style, with symmetrical bay windows and a steeply pitched slate roof. Construction began in 1909 and was completed in 1911. The home’s walls are poured concrete, intended to fireproof the residence and protect its art collection. The exterior walls are surfaced with mauve-brown Ohio brick, and trimmed with Lake Superior sandstone. Every room in the mansion has a fireplace, mostly marble-faced, and all are gas burning. The mansion had all the modern conveniences of the time, including an intercom system, a central vacuum, and an electric freezer that made cubed ice for cocktails.

A coach house was located to the west, fronted with a semicircular drive and enclosed by a brick wall and a wrought-iron fence with gates designed by another Milwaukeean, Cyril Colnik. The coach house now serves as an entrance to the museum, and leads to the Great Hall, a 1998 addition to the home for community engagement and events.