The 56-year-old Mr Zucker said in a memo to colleagues that he was "wrong" to not report the relationship as required.
The relationship was discovered during an investigation into the conduct of fired CNN anchor Chris Cuomo.
Mr Zucker is considered one of the most powerful media executives in the US.
In the message sent on Wednesday to the network's staff and shared online by CNN media correspondent Brian Stelter, Mr Zucker said that he was asked about his consensual relationship with a colleague during an investigation into Mr Cuomo's time at CNN.
Mr Cuomo was being investigated over efforts to help his politician brother, former New York governor Andrew Cuomo, fight allegations of sexual harassment.
Mr Zucker said that he had worked with the colleague for two decades and that the relationship had "evolved in recent years".
"I was required to disclose it when it began but I didn't," he said. "I was wrong. As a result, I am resigning today."
Mr Zucker's memo did not identify the colleague with whom he had a relationship but CNN named her as Allison Gollust, the network's executive vice president and chief marketing officer.
"Jeff and I have been close friends and professional partners for over 20 years," she said in a statement quoted by Mr Stelter on air. "Recently, our relationship changed during Covid. I regret that we didn't disclose it at the right time."
Ms Gollust was an executive at NBC Universal during Mr Zucker's time at that network as CEO.
In 2012, Ms Gollust was named as communicators director for Andrew Cuomo. She joined CNN the following year after only four months in that role.
In her statement, Ms Gollust added that she intends to stay at the network. Both she and Mr Zucker are divorced, the New York Times reported.
Mr Zucker joined CNN in 2013. Under his leadership, it broke viewership records in 2020 and saw the most viewers in its history during January 2021 - partly as a result of the US Capitol riot on 6 January.
Since then viewership has gone down, falling nearly 90% this January compared to the same time last year, according to Nielsen ratings.
At CNN, Mr Zucker faced criticism from those who believed the network gave too much coverage to the former president, especially during the Republican presidential primary. Mr Trump has also criticised what he saw as unfair coverage.
During his time at NBC, he approved Donald Trump's reality TV show "The Apprentice".