That left large sections of the crowd chanting their displeasure but with each run scored by the home side, those jeers turned to cheers as South Africa look to level the four-match series having lost the first test in Durban by 118 runs.
Rabada denies the contact was intentional but faces being banned for the final two tests of the series if found guilty.
Rabada doesn't have a great disciplinary record, and if a further three demerit points are added to his five already accrued, he will face a two-Test suspension and will be fined 100% of his match fee.
A Cricket South Africa spokesperson said Rabada would defend the charge and argue that the contact with Smith was accidental.
Pat Cummins was the only successful Australian bowler, bowling Rabada to collect both South African wickets of the innings.
He is now a figure of hate in South Africa after his confrontation with South Africa wicketkeeper Quinton de Kock in a fractious first test in Durban last weekend. "It will be a very important morning and fortunately our bowlers have had a day to rest". Even on the first day of the Test match, CSA officials were seen clicking the photographs with the spectators wearing offensive masks which indirectly pointed to the Australian opener and the spat between De Kock and him.
On the stroke of lunch Bancroft fell, edging behind a sumptuous outswinger from Philander that any fast bowler in the world would have been proud of, and the teams walked off with score at 1-98.
Warner had motored to 60, his third half-century in as many innings on this ground, before Ngidi produced the ball of the day - a delivery that wobbled through the air, nipped off the seam, beat the batsman on the inside and clipped the bails.
After winning a leg before decision against the Australian captain, Rabada shouted in celebration and approached Smith aggressively.
The tourists continued to chip away at the other end though, with the reverse-swinging ball collecting four quick wickets after tea. The allrounder did not field late in the day when it was Australia's turn to bowl. "Ideally we would obviously like a sizeable lead, the ball is tailing in and we have to bat last in the second innings".
AB de Villiers counter-attacked to hit 74 not out off 81 balls to take South Africa to 263 for 7 at the close of play.