Charles and Sarah decided in 1908 to commission a residence on Royall Place and Prospect Avenue — in the heart of Milwaukee’s “Gold Coast” — to house what had become a world-class art collection. They built the mansion with the intention of bequeathing it and their collection to the people of Milwaukee.

The house that architect Alexander Eschweiler built for Charles and Sarah is strongly influenced by the English Tudor style, with symmetrical bay windows and a steeply pitched English slate roof. Construction began in 1909 and was completed in 1911. The mansion was constructed with poured concrete, with the intent of fireproofing the residence and its art collection. The outside walls are surfaced with mauve-brown Ohio brick trimmed with Lake Superior sandstone. The former coach house was located to the west, fronted with a semicircular drive and enclosed by a brick wall and a wrought-iron fence with gates designed by another Milwaukeean, Cyril Colnik.

Charles resided in the mansion until his death in 1918. Sarah continued her residence in the home until her death in 1945.

At the time of the Allis gift, the Milwaukee War Memorial was being planned, and the city was unsure how the Allis mansion and art collection fit into its plans. It was finally decided that the Milwaukee Public Library would receive the gift for use as an art library and museum — the Charles Allis Art Library. In 1979, the house and its contents were transferred to Milwaukee County and renamed the Charles Allis Art Museum. The mansion still houses the Allis collection of Asian porcelains, European bronze sculptures, American and European paintings, and fine furniture, as well as Charles’ art history library.