In stage four, Gaviria, riding for Quick-Step, launched his first attack far from the finish line, but secured the win with a second, powerful burst of speed to beat veterans Peter Sagan of Slovakia and Germany's expert sprinter Andre Greipel.
When it came to the business end though his lead out train was infiltrated by another rider, he got out of position and could not respond when the eventual victor Fernando Gaviria launched his attack pursued by Peter Sagan and Andre Greipel.
Thought to be the purest sprint specialist in the game, Cavendish has had to watch Marcel Kittel and Peter Sagan steal the early limelight on the 2018 Tour.
Yellow jersey Greg Van Avermaet (BMC) finished the 195km stage from La Baule safely in the peloton to retain the race lead, but a number of riders lost time after a crash with 5km to go caused a split in the field.
Movistar lost 53 seconds, which leaves Colombian Nairo Quintana, twice runner-up, 1:13 behind Froome - a massive amount of time after only three days of racing.
Today, though, it was back to the more typical fare of the Tour's opening week.
"Tomorrow (stage five) is a hard stage but it is more of a positioning battle".
Sagan took the yellow jersey on Sunday and was gutted to have to give up the ghost on Monday in the team time-trial, where he took his foot off the pedal at 25km and rolled in three minutes off the pace.
Guillaume Van Keirsbulck, who was involved in the breakaway, won the day's intermediate sprint.
Sky finished second and Quick-Step Floors came third, seven seconds behind. Sagan (Bora-Hansgrohe) was next over the line, consolidating his grip on the green jersey. The only King of the Mountains point went to Anthony Perez, who was first up the St-Jean-la-Poterie.
Mark Cavendish was in with the frontrunners to win the stage, but as has been the case so far this Tour, the sprint ace's legs were not up to scratch this time.
The size of that gap clearly anxious the peloton, with QuickStep Floors among those redoubling their efforts on the front of the bunch, but the advantage was still above a minute with 10km to race.
Less fortunate was the Katusha-Alpecin overall hope, Ilnur Zakarin, who rolled over the line nearly a minute behind the front group.
By the finish line he had dropped a minute to the rest of the GC contenders.