Some of the government workers are accused of falsifying exam certificates while others are said to have passed relatives' qualifications off as their own.
A government investigation begun last October found 9,932 people it said had faked their secondary school qualifications, a number equivalent to two percent of the country's civil service.
After receiving the final report on Friday, Magufuli said the named workers must resign or face prosecution, which could result in sentences of up to seven years in jail.
"The 9,932 employees recruited on the basis of false diplomas will not receive their salary for this month of April and have to leave their positions immediately," Magufuli said.
He added that the posts would be filled by "qualified persons" as soon as possible and called on local media to publish the names of those fired.
Since taking office in 2015 Magufuli has displayed a no-nonsense approach to corruption but has also been accused of authoritarianism after bridling at criticism and cracking down on the media.
