Milledgeville’s oldest standing church, St. Stephen’s is located adjacent to the old capitol building on Statehouse Square. It was occupied in late November 1864 by members of the 107th New York Cavalry Regiment as part of Sherman’s detachment bound for Savannah on their infamous march through Georgia. They burned pews and poured syrup into the pipes of the church’s organ.  A Georgia historic marker placed in 1955 notes: This Church was organized in 1841 through the efforts of Bishop Stephen Elliott. the church building was completed in 1843 and consecrated Dec. 10. The vestibule, annex and Gothic roof were added later. The handmade chancel furniture was given by an early parishioner, John Wilcox. Rev. Rufus White was probably the first Rector and J.M. Cotting and C.J. Paine the first Wardens. In 1864 the building was damaged when Federal troops dynamited the nearby arsenal. In 1909 a new organ was presented by George W. Perkins of New York who had heard that Sherman’s troops stabled horses in the building and further damaged its contents.

Milledgeville Historic District, National Register of Historic Places