Court documents also describe assaults that included choking, slapping and closed-fist punches - all while his attorneys claim he earned less than $3,000 per year and was "called the N-word repeatedly", according to the Post and Courier.
The lawsuit accused Edwards and his brother, Ernest J. Edwards, the owner of the restaurant, of slavery, discrimination and labor violations.
Bobby Paul Edwards, 52, is accused in the indictment of using "force, threats of force, physical restraint and coercion" to enslave Smith at J&J Cafeteria in Conway, S.C. Conway is just inland from Myrtle Beach.
Edwards was arrested on Tuesday in connection with the indictment and of course, pleaded not guilty during a Wednesday hearing, but was ordered to be held without bail.
Edwards joined as manager in 2010, and allegedly forced Smith to "work from dawn until late into the night, seven days a week, with little or no pay, no benefits and no vacation time". He faces up to 20 years in prison if found guilty of the federal felony of "forced labor", according to the USA attorney general's Civil Rights Division. He will also have to pay restitution to Smith.
Smith was taken into the care of adult protective services and Edwards was charged with assault and battery.
Saying some witnessed the alleged abuse, the lawsuit notes that Edwards went after Smith with a belt buckle for being too slow to replenish food items on the buffet line.
J&J Cafeteria in Conway South Carolina
"Plaintiff was heard crying like a child and yelling, 'No, Bobby, please!' After this beating, Defendant Bobby forced Plaintiff to get back to work", read the complaint filed in 2015.
Edwards allegedly told Smith that he had a bank account with more than $30,000 of his earnings, but Smith said he was never paid any of that money or given access to the account.
At the time of the lawsuit, Smith's lawyers said: 'The conduct in this case is as troubling as anything I h ave seen in nearly 20 years of practicing law, attorney W. Mullins McLeod Jr. said. The allegations came to light after a waitress told her mother-in-law of the abuse, and the woman went to state social workers.
According to Geneane Caines, who had a daughter-in-law that worked in the restaurant, abuse was part of the overall culture.
Though the indictment remains sealed, numerous details surrounding Smith's alleged enslavement were revealed in a 2015 lawsuit against Edwards; the man's brother and restaurant owner, Ernest J. Edwards; and the establishment, The Washington Post reported.
Smith's attorney, David Aylor, said Smith was "very appreciative" of the efforts by federal prosecutors and believes that "justice will be served" in the case, he told the Post and Courier.
Smith told WMBF in Myrtle Beach in 2015 that he began washing dishes after school at J&J when he was 12 years old.