One of the nation's most prominent African American executives, Frazier did not mention Trump by name.
But Trump had been criticized for not explicitly condemning white supremacists after violent clashes with counter-protestors turned deadly in in the Virginia college town on Saturday.
That prompted the president to mock Frazier on Twitter in which he said Frazier will now have more time to "lower ripoff drug prices".
It follows the resignations from the same group on Monday of Kenneth Frazier, the black chief executive of pharmaceutical giant Merck, and Kevin Plank, the CEO of Under Armour.
Brian Krzanich said he was leaving the President's manufacturing council because he was concerned about the harm "our divided political climate" is causing to issues such as equality.
Just minutes before, Trump used the microblogging site to proclaim: "For every CEO that drops out of the Manufacturing Council, I have many to take their place".
Among those who have left are the chief executives of Merck, Under Armour and Intel and the president of the Alliance for American Manufacturing.
"Wal-Mart chief executive Doug McMillon joined the chorus, saying in a note to employees: "(We) too felt that he missed a critical opportunity to help bring our country together by unequivocally rejecting the appalling actions of white supremacists". Several top officials, including Vice President Mike Pence and White House advisor Ivanka Trump, have spoken forcefully against hate groups.
A White House official downplayed the importance of the manufacturing council and a separate policy and strategy forum featuring corporate leaders.
Trump criticized Frazier nearly immediately Monday over drug prices, and again Tuesday for having factories overseas.
"It's certainly a sign that Trump's more controversial stuff isn't playing well with companies selling to middle America", he said.
Tesla chief executive Elon Musk and Walt Disney chief executive Robert Iger quit the Trump administration's strategic and policy forum in June after the President announced the United States would withdraw from the Paris climate accord.
Trump formed the manufacturing council a week after his inauguration as part of a push to create jobs.
"Our commitment to diversity and inclusion is unwavering, and we will remain active champions for these efforts", said a spokesman for Campbell Soup for CEO Denise Morrison. And if they do, are new CEOs going to be eager to step in?