England had opened the score early in the second half thanks to a penalty goal by captain Harry Kane. With their first choice XI back, they should perform better than in their last match, but Colombia will provide them with a stern test.
England had to deal with some rough tactics from Colombia, who gave away 23 fouls to England's 13, and Southgate admitted the scene in the dressing room resembled a scene from M*A*S*H, the 1970s American comedy-drama about an army hospital.
Pickford succeeded where Peter Shilton, David Seaman and Paul Robinson failed as the 1990, 1998 and 2006 World Cup campaigns ended in shootout losses.
At EURO 2004, England reached the quarters but couldn't get past Portugal in regulation or extra time.
The final match in the round of 16 at the 2018 World Cup sees Colombia take on England in Moscow.
He limped out of Colombia's last game due to a recurring calf injury, which has limited him to just 152 out of 270 minutes during this World Cup. Henderson's effort was saved by Ospina, but that was made level when Uribe rattled the bar.
Everton star Pickford's save gave Eric Dier the chance to slot home the winning kick. The entire Colombia team including Ospina piled into the area, but the ball found the head of center back Yerry Mina who headed the ball into the turf and bouncing high off the head of Trippier on the line and into the net. "Through small calls, he was pushing us [toward] our goal, that was clear for me".
England will next play Sweden in the quarter-finals on Saturday.
If Southgate wanted this World Cup to be a learning curve for his players - a finishing school at elite level - this bruising, argumentative, niggly match was perfectly designed for the goal.
The winner of Colombia-England gets the Sweden-Switzerland victor in the quarter-finals, and then a semi-final looms against Croatia or Russian Federation.
Controversy struck in the 38th minute when, while jockeying for position for a free kick near the Colombia box, Wilmar Barrios appeared to throw his head into England midfielder Jordan Henderson's chest, with the Liverpool man throwing himself to the ground and staying down for a number of minutes after the contact.
Uruguay vs. France (3 pm BST) and Brazil vs. Belgium (7 pm BST) is set for Friday, while Russian Federation vs. Croatia (7 pm BST) and Sweden vs. Colombia/England (3 pm BST) takes place on Saturday.
And on that basis, Southgate and his joyous England squad can reflect on a job well done.