Japanese authorities told the Olympic and Paralympic committees it was "highly unlikely" that entry to the country could be guaranteed.
Organisers said the move now gives "clarity" to ticket holders and helps ensure "a safe and secure Games for all participants and the Japanese public".
The Games are due to begin on 23 July.
The Paralympics follow the Olympics a month later, from 24 August.
Organisers said the "challenging" Covid-19 situation in Japan and many other countries, global travel restrictions and emergence of variant strains of the virus had led to the decision and that ticket holders would be refunded.
The Olympics were postponed by a year in March last year because of the growing spread of coronavirus across the world.
It is the first time in the event's history it has been postponed, with more than 11,000 athletes from about 200 countries scheduled to take part in 2020.
Thomas Bach, president of the International Olympic Committee, said the move is a "great sacrifice for everybody".