Kim Yo Jong, the first member of the North's ruling dynasty to visit the south since the end of the Korean War in 1953, smiled broadly as they shook hands.
Asian media outlets quickly circulated photos of sprawling messages that Kim Yo Jong and Kim Yong Nam left in the Blue House's guest book. The daily said the two sides "discussed a range of issues concerning the two Koreas".
Pence had said all week that if he met with the North Koreans, he would deliver a tough message.
Pence had kept open the possibility for some contact with the North Koreans in South Korea while reiterating Washington's insistence that denuclearization by North Korea is a necessary condition for permanent peace.
Mr Moon took a commemorative photo with the North Korean guests prior to their talks.
After more than of year of accelerated North Korean ballistic missile and nuclear tests, Moon helped bring about a pause in provocations by negotiating North Korea's participation in the Olympics.
Seoul's Unification Ministry said Friday that the Mangyongbong-92 will return to the North on Saturday.
The figures from the two Koreas shared kimchi (Korean pickled cabbage) and soju rice liquor, and spoke for three hours.
A giant dove lit up the sky, famous South Korean singers sang John Lennon's Imagine and, most significantly of all, athletes from North and South Korea walked in together for the first time in 11 years.
The Puerto Rico Olympic Committee then withdrew its recognition of its Winter Sports Federation, making athletes intending to represent Puerto Rico ineligible to compete.
Pence is viewing the short-track competition, in which eight Americans are participating.
That said, these officials said that if the North Koreans had approached Pence with pleasantries, he would have responded in kind.
Vice President Mike Pence did not deliberately snub North Koreans at an Olympic reception, according to U.S. officials who pushed back on South Korean reports that Pence deliberately came late to a VIP gathering Friday evening and then snubbed officials from Pyongyang.
The athletes elicited a standing ovation by most leaders in the VIP box, including Moon and Kim's sister. He also gained US support for postponing joint military exercises until after the Olympics and Paralympics end in late March.
Ms Kim and the North's ceremonial head of state Kim Yong-nam made up the most senior delegation from the North to visit the South since the Korean War in the 1950s.
When asked for a reaction to North Korea's invitation to Moon, the vice president's spokesperson said "The Vice President is grateful that President Moon reaffirmed his strong commitment to the global maximum pressure campaign and for his support for continued sanctions".
"Kim Jong Un is clearly serious about reviving talks with the South to improve relations".
She has been demonstrating a very different demeanour than she does in the North, said Korean Peninsula Future Forum analyst Duyeon Kim.
But North Korea's bad record on human rights and the growing threat from its nuclear weapons program appeared out of mind as Moon warmly greeted Kim Yo Jong, the sister of dictator Kim Jong Un, and Kim Yong Nam, the country's 90-year-old nominal head of state.
The officials toasted with the Korean liquor "soju" and were served buckwheat and potato pancakes, dried pollock soup, grilled halibut and other dishes.
Mr Pence stayed seated when the athletes of the host nation marched into the arena alongside those from the North. Kim Yo Jong and Kim Yong Nam were seated a row behind.
Athletes from North Korea and South Korea marched alongside each other at the 2018 Winter Olympics opening ceremony on Friday.
The North Korean delegation also includes 24 coaches and 21 media representatives.
It's unclear whether the occasion could be used to set up bigger meetings between the Koreas to ease tensions.