The crash Friday was Taiwan's worst rail accident in seven decades.
Taiwan's last major rail crash occurred back in October 2018, per the AP, when an express train derailed while going around a tight corner. The unmanned truck's emergency brake was not properly engaged, according to the government's disaster relief center.
The train, with nearly 500 people aboard, was traveling from Taipei, the capital, to Taitung on the east coast when it derailed in a tunnel just north of the city of Hualien.
More than 188 people were reported injured, and dozens were admitted to a hospital.
On Saturday, focus shifted to removing carriages now blocking one half of the sole train line down Taiwan's remote and mountainous eastern coastline.
He could face charges of causing death by negligence and of forging documents and he said he was co-operating with investigators.
The Hualien district court initially allowed Lee to post bail of 500,000 new Taiwan dollars (US$17,516), but that decision was reversed Sunday when a higher court in Hualien rescinded the lower court's decision to allow bail, Taiwan's Central News Agency reported.
The court said that while the truck's fall into the path of the train possibly resulted from negligence, there was "no possibility of conspiracy".
Yu Hsiu-duan, head of the Hualien prosecutors' office, said they were not pleased with the decision.
Lee's court-appointed lawyer declined to comment to reporters as he left the court.
"Prosecutors will certainly step up investigation and understanding of the crimes or of other suspects involved in the case", department of prosecutorial affairs director-general Lin Jinn-tsun said.
The youngest person confirmed to have died was a six-year- old girl, the oldest a 79-year-old man, according to a government-issued casualty list.
Lee, 49, was part of a team who regularly inspected Taiwan's mountainous eastern train line for landslides and other risks. With rescuers anticipating the discovery of more bodies inside the wreckage, officials have suggested the death toll may still rise.
"We will continue to do everything we can to ensure their safety in the wake of this heartbreaking incident", he added.
Officials investigating a train crash in Taiwan that killed 51 people are focusing on a maintenance lorry that slid onto the track.
President Tsai Ing-wen visited a hospital in Hualien on Saturday to "express my sympathy to the families of the victims", she said.
Media published on the site showed a yellow flatbed truck on its side, near the train.
The de facto French Embassy in Taipei confirmed that one of its citizens had died in the crash.
In a rare sign of goodwill from China, which claims Taiwan as its own territory, President Xi Jinping expressed his condolences over the crash, state news agency Xinhua said.
The accident occurred on the first day of the annual grave-cleaning festival, which is a long public weekend when roads and railways in Taiwan are usually crowded.
Train travel is popular during Taiwan's four-day Tomb Sweeping holiday, when families often return to hometowns to pay respects at the gravesites of their elders.