If Rabada is found guilty of the level two offence, it carries a minimum sanction of three points.
With Rabada only one infraction away from a suspension, the possibility of him missing the following two Test matches is yet to be ruled out.
Match referee Jeff Crowe charged Rabada ahead of play on day two at Port Elizabeth.
Rabada will contest the charge on the basis that he claims the contact with Smith was unintentional, South Africa team spokeswoman Lerato Malekutu said. "I have spoken to him a few times about it, to try and get him to relax a bit", Holding told ESPNcricinfo.
And Philander recognised the void Rabada could leave, saying: "I don't even recall there being a send-off".
"The way "KG" was running in this afternoon was pretty unbelievable to watch", Philander said.
A Level 2 breach of the ICC code of conduct carries a maximum of a 100 per cent match fee fine and between 3-4 demerit points.
Smith had raised eyebrows at the toss with his decision to bat in what were clearly bowling conditions but stated that if the batsmen could get through the first hour a big first innings score is on the cards. "Let's hope that's not the case and we'll have a four-pronged attack come Cape Town", Philander said.
Rabada knows this as well as anybody.
Rabada finished with five for 96, having done all his damage on either side of tea, taking the key wicket of Australia captain Steve Smith, then following up by dismissing the Marsh brothers - Shaun and Mitchell - in his next over. Smith was given out by Dharmasena and, in celebration, Rabada continued in his follow-through, shouting "Yes, yes", in Smith's direction.
Smith's facial expression suggested he was far from pleased with the spray.
In a spell of searing pace just before the tea interval, South Africa's spearhead answered his captain Faf du Plessis' call for the senior players within the Proteas set-up to the lead the revival. Australia battled to create opportunities to take wickets, though spinner Nathan Lyon did find good turn off the wicket and was hard to score off.