The fine, which amounts to double MTN's profit last year, was for failing to cut off unregistered mobile users.
The firm was given until 16 November, just two weeks, to pay the penalty.
The company is in talks to reduce the fine.
"Due to the most unfortunate prevailing circumstances occurring at MTN Nigeria, I, in the interest of the company and its shareholders, have tendered my resignation with immediate effect," said Mr Dabengwa in a statement.
He had been in the post since 2011.
Non-executive chairman Phuthuma Nhleko has been appointed executive chairman for a maximum of six months while the company looks for a successor.
It is thought that the Nigerian government imposed the fine after the kidnapping of a former finance minister, Chief Olu Falae. The kidnappers used an unregistered Sim card from MTN to demand a ransom.
Nigeria is MTN's biggest market, with 28.5 million subscribers, followed by Iran and South Africa.
How much is $5bn?
The $5.2bn (£2.7bn) fine MTN is facing is the same as:
of Nigeria's annual budget
Double MTN's annual group profits ($2.69bn after tax)
$80 for every single MTN customer in Nigeria
1.5x Niger ia's annual capital expenditure
95% of Nigeria's annual spend on servicing its national debt
PWC / CIA factbook / MTN
Analysis: Matthew Davies, Johannesberg
The resignation of Sifiso Dabengwa has not come as a surprise to analysts and market watchers.
His job was on the line ever since the authorities in Nigeria slapped MTN with a $5.2bn fine for failing to cut off mobile phone users who were using unregistered Sim cards.
Mr Dabengwa had been in the top post at MTN for four years, but had served as the company's chief operating office and as head of its Nigerian operations before that.
The ratings agencies Moody's and Fitch lowered MTN's credit rating last month because of the Nigeria fine, the size of which the company is still thought to be trying to negotiate lower.