Small was famed for keeping rampant Jonah Lomu at bay when South Africa won the World Cup final in Johannesburg, a historic moment for the country just a year after the end of white-minority apartheid rule.
The 50-year-old was rushed to hospital in Johannesburg on Tuesday night after suffering a suspected heart attack.
James Small was a patriot who fought like a soldier in the field of play and was prepared to die with his boots on.
Small made his debut against the All Blacks in 1992 and in his final Test in 1997, he scored his 20th try to break Dane Gerber's record.
He later coached domestic sides.
Ruben Kruger died from brain cancer in 2010, and Joost van der Westhuizen from motor neurone disease in 2017.
"The African National Congress (ANC) is shocked and saddened by the passing of former Springbok player, James Small".
Former England cricketer Kevin Pietersen branded Small his childhood "rugby hero", while World Rugby chief executive Brett Gosper also added his condolences. Small became the first ever Springbok to be sent off when showing dissent to referee Ed Morison in a 1993 Test with Australia. "Our honest condolences are extended to his children and the entire Small family on their sad loss", he said.
"He played with extraordinary passion and commitment, for he knew that every time he walked on to the field, he carried the hopes of the nation on his shoulders". Our deepest sympathies go out to the family, friends and the Springboks'.
Small had worked as a coach before his death, assisting Robert du Preez at NWU-Pukke and the Leopards.