Fines of €100 (£85) will be imposed at monthly intervals from mid-January on those who refuse, Prime Minister Kyriakos Mitsotakis said.
The money will go towards the Greek health system, which is struggling with a surge in hospital admissions.
About 63% of Greece's 11-million population is fully vaccinated, but data show more than 520,000 people over 60 are yet to get the jab.
"Greeks over the age of 60... must book their appointment for a first jab by January 16," the premier said in a statement to the cabinet.
"Their vaccination is henceforth compulsory."
The measure is still to be put to a parliamentary vote, he said, but lawmakers are widely expected to approve it.
While countries have made vaccines mandatory for health workers and other high-risk workers, Greece will become the first in the EU to target a specific age group.
Austria has announced that compulsory Covid vaccinations will start in February.
Syriza, Greece's main opposition party, called the new measures punitive and financially excessive.
Mr Mitsotakis said the decision had "tortured" him but he felt a "heavy responsibility in standing next to those most vulnerable, even if it might fleetingly displease them".
The virus has claimed more than 18,000 lives in Greece.