Muntari was shown the yellow card for dissent by referee Daniele Minelli in the 90th minute and walked off the pitch in protest during stoppage time, leaving his team to play the last few minutes with 10 men.
The 32-year-old told television reporters that a group of fans had been insulting him from the start of the match, including a group of children.
He said he offered the children his shirt to set an example but the insults continued in the second half.
"He told me I should not talk to the crowd. I asked him if had heard the insults. I insisted that he must have the courage to stop the game," Muntari said.
"The referee should not just stay on the field and blow the whistle, he must do everything. He should be aware of these things and set an example," he added.
Italy has struggled to stop racist chanting in its stadiums and there have been a number of incidents over the years.
In one of the most serious in 2013, the AC Milan team abandoned the pitch during a friendly in the town of Busto Arsizio after home fans insulted midfielder Kevin-Prince Boateng.
Under guidelines introduced following that incident, the referee is supposed to alert the fourth official who should in turn report the incident to the public security department responsible for policing at games.
Pescara coach Zdenek Zeman said he sympathised with Muntari but said the player should not have walked off.
"Muntari has abandoned the pitch because of racist chants but we should not take justice into our own hands," he said.
"We have been talking about racism for years now but nothing happens. Today, this has happened to Muntari who has been playing in Italy for several years now. We need a change of mentality."
(Writing by Brian Homewood in Berne; Editing by Clare Fallon)
Share
