Quinton de Kock was unbeaten on 44 at the Tea interval.
Bavuma looked well-set until a straight drive from De Kock deflected off the boot of bowler Fernando and onto the stumps with the batsman out of his crease at the non-striker's end.
Sri Lanka head to South Africa on the back of two terrible performances in Australia in which they struggled badly with the bat after failing to deal with the fast tracks and pace bowling which saw them manage just 647 runs from four innings.
Steyn had become South Africa's highest Test wicket taker, eclipsing the previous record set by Shaun Pollock with 421 wickets.
The Proteas will be looking to continue their good form in Test matches after their series whitewash of Pakistan when they face Sri Lanka at Kingsmead in Durban on Wednesday.
Vishwa Fernando (4/62), playing just his fourth Test, and Kasun Rajitha (3/68), playing his fifth, were the chief destroyers for the visitors.
Bavuma's wicket was followed by two more, as Vernon Philander and Keshav Maharaj followed him back into the pavilion, leaving De Kock and the tail to guide the hosts to a decent total. "We will have to figure out how we go about getting wickets".
Hashim Amla was given not out by umpire Aleem Dar following an appeal for leg before wicket, two balls after left-arm opening bowler Vishwa Fernando had Dean Elgar caught behind in just the second over.
Temba Bavuma and Du Plessis rode out the early storm and gradually started to change the momentum in a 72-run partnership which was ended in the last over before lunch when Du Plessis was caught down the legside by wicketkeeper Niroshan Dickwella off Rajitha for 35. Replays showed that the ball pitched in line and was crashing into the stumps and that Sri Lanka had in fact asked for a review when only 13 seconds had passed.
There was more drama as Sri Lanka were denied a review for extending the 15-second limit for claiming one.
Amla, however, wasn't able to use the reprieve and was caught behind for 3 in the seventh over. "Sri Lanka have shown some good intent within their batsmen and will look to score quickly".