Yoshiro Mori, 83, was quoted as saying women talk too much and that meetings with many female board directors would "take a lot of time".
The remarks set off a firestorm of protest and Mr Mori apologised at the time but said he would not resign.
But on Friday he apologised for his "inappropriate statement".
"What is important is to hold the Olympics from July. It must not be the case that my presence becomes an obstacle to that," he said at a special committee meeting on Friday, where he also announced his resignation.
It is not immediately clear who will replace Mr Mori, a former Japanese prime minister. He had initially selected well-known sports administrator Saburo Kawabuchi, 84, to replace him but this also led to protests.
The resignation came as pressure mounted on him over the past week. Major sponsors came forward to criticise his comments, including Toyota, one of the biggest Olympic backers.
Toyota's president Akio Toyoda said the company was "disappointed".
On Tuesday, a group of female lawmakers wore white in a protest against his remarks, with some men doing the same in solidarity.
And Tokyo Governor Yuriko Koike said she would not attend a meeting of high-level Olympic officials in protest.
According to local media, almost 400 people have also withdrawn applications to volunteer at the Olympic games, which are scheduled for later this year.