Prisoners began arriving at the prison in late February of 1864 and by early June the prison population had climbed to 20, 000. Consequently, it was decided that a larger prison was necessary, and by mid-June work was begun to enlarge the prison. The prison's walls were extended 610 feet to the north, encompassing an area of roughly 10 acres, bringing the total prison area to 26. 5 acres. The extension was built by a crew of Union prisoners consisting of 100 whites and 30 African Americans in about 14 days.

On July 1, the northern extension was opened to the prisoners who subsequently tore down the original north stockade wall, then used the timbers for fuel and building materials. By August, over 33, 000 Union prisoners were held in the 26. 5 acre prison. Due to the threat of Union raids (Sherman's troops were marching on Atlanta), General Winder ordered the building of defensive earthworks and a middle and outer stockade around the prison. Construction of the earthworks began July 20 th. These earthworks consisted of Star Fort located southwest of the prison, a redoubt located northwest of the north gate, and six redan's.

The middle and outer stockades were hastily constructed of unhewn pine logs set vertically in wall trenches that were about four feet deep. The middle stockade posts projected roughly 12 feet above the ground surface and encircled the inner prison stockade as well as the corner redan's. The outer stockade, which was never completed, was meant to encompass the entire complex of earthworks and stockades. The posts of the outer stockade extended about five feet above the ground surface. By early September, Sherman's troops had occupied Atlanta and the threat of Union raids on Andersonville prompted the transfer of most of the Union prisoners to other camps in Georgia and South Carolina. By mid-November, all but about 1500 prisoners had been shipped out of Andersonville, and only a few guards remained to police them.

Transfers to Andersonville in late December increased the numbers of prisoners once again, but even then the prison population totaled only about 5000 persons. The number of prisoners at the prison would remain this low until the war ended in April of 1865. During the 15 months during which Andersonville was operated, almost 13, 000 Union prisoners died there of malnutrition, exposure, and disease; Andersonville became synonymous with the atrocities which both North and South soldiers experienced as prisoners of war. After the war ended, the plot of ground near the prison where nearly 13, 000 Union soldiers had been buried was administered by the United States government as a National Cemetery. The prison reverted to private hands and was planted in cotton and other crops until the land was acquired by the Grand Army of the Republic of George in 1891. During their administration, stone monuments were constructed to mark various portions of the prison including the four corners of the inner stockade and the North and South Gates..