Called it "the hardest decision I've ever made," an emotional Mr Key said "I don't know what I'll do next".
Deputy PM Bill English is likely to take over until the National Party holds a caucus to choose a new PM.
Mr Key, a popular leader, is stepping down at the request of his wife Bronagh, the New Zealand Herald reports.
He won a third term for the National Party at elections in September 2014. He said he would not be contesting the 2017 election.
Mr Key made the announcement during his weekly press conference, citing family reasons for the surprise decision. He set a date of 12 December for the formal resignation.
He said his job required great sacrifices "from those who are dearest to me" and that his children had coped with "an extraordinary level of intrusion...All I can say is that I gave it everything I had. I have left nothing in the tank."
Meanwhile, Italian Prime Minister Matteo Renzi has resigned after suffering a heavy defeat in a referendum over his plan to reform the constitution.
In a late-night news conference, he said he took responsibility for the outcome. He said the No camp must now make clear proposals.
An exit poll for state broadcaster RAI suggests 42-46% voted to back reform, compared with 54-58% voting No.