The IOC unveiled its first team of refugees last year in an effort to raise awareness of the issue and it was one of the feel-good stories of the 2016 Olympics.
The 10-member team from Syria, Congo, Ethiopia and South Sudan hogged the spotlight after marching as the penultimate team before host nation Brazil in the Opening Ceremony at the Olympic stadium.
The athletes took part in athletics, swimming and judo.
"We are already discussing a potential refugee team for Tokyo 2020," IOC spokesman Mark Adams told reporters on Sunday.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"We want to strengthen our efforts with the United Nations."
Adams said it was too early to talk about size of the team given the complex selection process, with the refugees for Rio being located at camps scattered across the world.
More than a million refugees streamed into Europe in 2016 alone as they fled fighting in the Middle East and elsewhere.
Millions more are housed in camps in countries across the world, having escaped wars or armed conflicts in their home nations.
(Editing by Ed Osmond)
