Mr Mahathir's shock resignation comes amid rumours that he may form a new coalition without his designated successor, Anwar Ibrahim.
The world's oldest prime minister came into power in 2018.
In a surprise victory, he ousted then-prime minister Najib Razak, who has been linked to a multibillion-dollar scandal involving a government fund.
The Prime Minister's Office said the letter of resignation was submitted at 13:00 local time (05:00 GMT). No other details were included in the statement.
It is not clear who will be the next prime minister - or if new elections will be called.
The 94-year-old has also resigned from Bersatu, the political party of which he was chairman.
Bersatu had been part of the ruling opposition alliance Pakatan Harapan, which Mr Mahathir had joined in 2018 together with Mr Anwar.
On Sunday, Mr Anwar accused Mr Mahathir and his party of being "traitors", saying they had plans to form a new government, presumably without him.
But on Monday, Mr Mahathir submitted his resignation.
Despite this, the BBC's South East Asia correspondent Jonathan Head says Mr Mahathir might not actually be ready to step down.
"At 94, Mahathir Mohammad is still the master manipulator of Malaysian politics, and few believe he is ready to step down, despite the dramatic offer of resignation he's just sent to the country's constitutional monarch," he said.
"It is possible, even likely, that Mr Mahathir will remain as prime minister."
The on-again off-again feud between Mr Anwar and Mr Mahathir is one that has dominated Malaysian politics for decades.
Mr Mahathir was prime minister from 1981 to 2003, and was part of the long-ruling party Barisan Nasional (BN).
Mr Anwar was his deputy but the relationship soured when Mr Anwar was sacked in 1998 after a leadership dispute.
He was later jailed on corruption and sodomy charges, which were widely regarded as politically motivated.
But in 2018, Mr Mahathir shocked the country when he announced that he was teaming up with Mr Anwar and joining Pakatan Harapan.
He said he was doing so to oust the government of Mr Najib, who had become embroiled in the 1MDB corruption scandal.
Mr Mahathir and Mr Anwar's alliance won - and Mr Mahathir agreed to eventually hand power over to Anwar.
But Mr Mahathir repeatedly refused to say when he would transfer power - stoking tensions within the opposition alliance. BBC