All four were members of the southern California-based "Rise Above Movement," which prosecutors described as a militant white supremacist and anti-Semitic group.
The four "traveled to Charlottesville to encourage, promote, incite, participate in and commit violent acts in furtherance of a riot," according to a criminal complaint filed in Charlottesville federal court.
The complaint said the four had already attended political rallies in Huntington Beach and Berkeley, California in 2017 where they attacked other protestors.
Their own "extensive" social media presence documented their violent actions in the California protests and at the August 11-12 "Unite The Right" protest in Charlottesville, said district attorney Thomas Cullen.
Each was charged with multiple counts of inciting to riot, which Cullen said could bring a maximum 10 years in prison.
Cullen said the investigation was aided by information in a joint investigation by the ProPublica investigative journalism group and PBS television's Frontline show.
The protests saw hundreds of neo-Nazi sympathisers, some carrying firearms, yelling white nationalist slogans while wielding flaming torches in scenes eerily reminiscent of racist rallies held in America's South before the Civil Rights movement.
They had gathered to protest efforts to remove statues of Confederate leaders, including the Confederacy's top general, Robert E Lee.
On the second day fighting broke out between neo-Nazi supporters and anti-fascists from a black-clad group called Antifa.
The violence culminated with a man driving a car into a crowd of counter-protesters, killing a woman and injuring 19 people.
The man who drove the car, James Fields, has been charged with murder and hate crimes.
The event put President Donald Trump in a harsh spotlight after he criticised both the right and the left, amid overwhelming evidence that the neo-Nazis were the principle source of violence.
"You had some very bad people in that group, but you also had people that were very fine people, on both sides," Trump said.
That drew tough criticism from his predecessor in the White House, Barack Obama.
"How hard can that be, to say that Nazis are bad?" Obama said.