Mr Magufuli picked up rubbish from the street outside State House as part of the scheme, which he had ordered to replace independence day celebrations.
The move is being seen as symbolic of the president’s promise to tackle corruption, our correspondent says.
Thousands of people across Tanzania are reported to have joined the clean-up.
Last month, Mr Magufuli cancelled traditional independence day celebrations, which usually include a military parade and concert, saying it would be “shameful” to spend huge sums of money while the country was facing a serious cholera outbreak.
“Let us work together to keep our country, cities, homes and workplaces clean, safe and healthy,” Mr Magufuli told a crowd of surprised onlookers as he picked up rubbish off the street, AFP news agency reports.
“Tanzania has changed – this is a new Tanzania,” one resident told AFP, on a break from cleaning up the city’s public beach.
Since coming to power in October, Mr Magufuli, nicknamed “The Bulldozer” for his no-nonsense approach, has announced a range of cost-cutting measures, including a ban on unnecessary foreign travel by government officials.
The hashtag #WhatWouldMagufuliDo, coined after the initial cancellation of festivities last month, has been trending on social media in Tanzania and across East Africa, as people come up with their own money-saving initiatives:
Last month, Mr Magufuli ordered the cost of a party to inaugurate the new parliament to be slashed from $100,000 (£66,000) to $7,000.
He also sacked the head of the main state hospital after finding patients sleeping on the floor during a surprise visit to the facility.
Gal Despachos Aduaneiros en Maputo | Clearing Agent in Mozambique | Top Website | http://www.galclearing.com
The Leader in Oil & Gas Industry Mozambique | Hydrocarbon Logistics Centre | Yukon Logistics Lda | The Pemba Logistics Base | Nigerian company Orlean Invest | Portos de Cabo Delgado (PCD) | Top News Website | Mozambique latest news | Mozambique Biggest English Portal