Social distancing curbs were loosened for most of the country on May 14 as new infections fell, but the government has kept Tokyo and four other prefectures under watch.
Shinzo Abe made the comments at the start of a meeting between the two leaders in Tokyo.
Japan said the two leaders will meet on December 20.
The stadium, built at a cost of 156.9 billion yen ($1.44 billion), will have a capacity of 68,000 when it hosts the Olympic Games opening ceremony on July 24.
A visit next week would have allowed him to attend the enthronement ceremony of Japanese Emperor Naruhito scheduled for October 22-24.
A ceremony formally recognizing his ascension to the post will be held on October 22, with the attendance of many world leaders.
Larijani did not elaborate on what those changes would be.
Chief Cabinet Secretary Yoshihide Suga also saidthat Abe and U.S. President Donald Trump had spoken on the telephone earlier on Tuesday to discuss Iran, among other topics.
The trip comes amid escalating tension between Iran and the United States and a year after the United States pulled out of a deal between Iran and global powers to curb its nuclear programme.