Many in the crowd gathered in the capital's Plaza de Colon, waving Spanish flags.
Around 45,000 people attended, officials said.
Former French prime minister Manuel Valls, who was born in Barcelona and is running for mayor of Spain's second largest city, and Peru's Mario Vargas Llosa, a Nobel Prize-winning author who became a Spanish citizen in the 1990s and has spoken out in favour of conservative Spanish causes, were among those who attended the Madrid protest. But Sanchez's government broke off negotiations on Friday, when Vice-President Carmen Calvo said the separatists wouldn't budge from their demand for an independence referendum.
Protesters hold a banner reading "Sanchez resignation".
Speaking at a Socialist party event in northern Spain, Sanchez reminded his political opponents that when he was an opposition leader, he stood by Rajoy on the situation in Catalonia even after separatist regional officials staged an October 2017 independence referendum in defiance of Spanish courts.
Organisers said over 200,000 people took part on the rally held under the slogan "For a united Spain".
"The government is giving many things to supporters of Catalan independence and is going to break Spain apart", said Raquel García, 76, who was carrying a Spanish flag.
The conservative opposition Popular Party and the centre-right Citizens party organised the rally, which was also backed by the upstart far-right Vox and other marginal far-right parties.
"The government of Spain is working for the unity of Spain", he told a party rally in the northern city of Santander.
Despite the government's concession, Catalan separatist parties still refuse to back the government's 2019 budget.
Mr Sanchez is trying to cobble together support to pass a national budget and will need votes from the Catalan separatists to pass it.
Failure by parliament to approve the budget bill could prompt a snap election before the next scheduled vote in 2020.
The protest took place two days before the start of the trial of 12 Catalan independence leaders, who face up to 25 years in prison on charges of rebellion and misuse of public funds for their role in a failed secession bid from Spain that they are accused of spearheading in 2017.