Sports Direct said it was unable to agree "reasonable terms" with landlord Intu.
"There's a lot going on at the centre and our plans mean that we have a number of exciting new brands either now open or currently in negotiations to take their first stores at intu Victoria Centre, whilst others are looking at their existing stores to make sure they are making the most of their presence in the centre".
The four stores are at Lakeside, east of London, Nottingham, Norwich and at the Metrocentre in Gateshead.
The company, which bought the struggling department store chain out of administration in August, said it had held talks over 14 weeks with Intu Properties but had been unable to reach agreement.
"Despite our best efforts we have been unable to agree reasonable terms for these stores to continue trading".
Chapelfield operator Intu said it had been told the Norwich store would close "next year" as part of a "national stores review".
Sports Direct is now consulting staff on the closures. "I urge other institutional landlords to be more proactive to help save the HoF stores in their schemes", he said.
After a local campaign House of Fraser has reprieved the historic Beatties store in Wolverhampton city centre, which faced closure early next year. On Wednesday morning the company said it had given the bidder, led by Intu's deputy chairman John Whittaker with backing from Saudi and U.S. investors, extra time to make a firm offer. It has also struck deals with the landlords of the former Rackhams store in Birmingham and the House of Fraser store in Telford.
It is understood that numerous company's landlords are reluctant to agree to the low rents Ashley is demanding.
Following his acquisition of the chain in August, Mr Ashley pledged to save around 47 of House of Fraser's 59 outlets.
It will be one of four House of Fraser stores to shut in the latest tranche of closures announced on Wednesday.
Intu has been contacted for comment.