"My guys are telling me, and this is what I'll be taking back to Canberra, that he needs to step down as leader of the National party and take a step back into the back bench."
However, Nationals deputy leader Bridget McKenzie said in a statement: "There is no stronger advocate for regional Australia than Barnaby Joyce."
She said there would not be a full party room scheduled for Monday because senators would be involved in estimates hearings.
If Mr Broad wants a full party room meeting he will have to request it from whip Michelle Landry, who is likely to grant permission but will back Mr Joyce retaining the leadership.
Cabinet minister Nigel Scullion says he "absolutely" supports Mr Joyce, who he believes will be deputy prime minister at the end of Monday.
"If I had three votes they'd all go to him but unfortunately I've only got one," Mr Scullion told ABC radio.
"I don't walk away from my mates because they get a bit of a jolt."
Mr Broad said the final straw was Mr Joyce and pregnant former staffer Vikki Campion giving their first interview as a couple to Fairfax Media.
Mr Joyce is on a week's personal leave after weeks of scrutiny over his affair with Ms Campion and controversy over political jobs she was given.
Mr Broad said Mr Joyce was not fit to step up as Australia's acting prime minister.
"He's meant to be taking a break and he's clearly playing to the media," he said.
"This is an issue we should have let quietly die and let's get on with the core job but he's not prepared to do that."
Mr Broad said Mr Joyce had told him on the phone he would not stand down.
The Mallee MP's call came days after West Australian Nationals state leader Mia Davies publicly urged Mr Joyce to stand aside and hours after Queensland Opposition Leader Deb Frecklington refused to say whether Mr Joyce had the support of the Liberal National Party.
"It is a matter for the federal Nationals and I'm really going to leave it at that," Ms Frecklington told reporters.
Acting Prime Minister Mathias Cormann said he had been in touch with the deputy prime minister, who would ordinarily be in the position when Malcolm Turnbull was overseas.
"We had a very good conversation. He was cooking spaghetti at the time we talked," he told reporters in Canberra.
"Yes it's a distraction for the government but there's some human beings involved."
Labor leader Bill Shorten said Mr Joyce's private life was "not really fit for open commentary" but there were legitimate concerns around whether he used his public position for private benefit.
Mr Turnbull and Mr Joyce have been in public conflict since the prime minister said his deputy had committed a "shocking error of judgment" over his affair.
Nats who can decide Joyce's future
Nationals leader Barnaby Joyce appears to have more friends than enemies in the party room ahead of a meeting on Monday.
Victorian Nationals MP Andrew Broad wants Barnaby Joyce benched, after revelations about his affair with an office staffer. But what does the party room think?
Joyce insists he isn't moving, labelling calls to oust him a "witch-hunt" and urging the media to "move on".
The senator who became Joyce's deputy in December is sticking by her leader, offering a "rolled gold guarantee" he will remain in the position.
The first-term MP, appointed agriculture minister at Joyce's insistence in December, has urged those agitating against the leader to "put up or shut up".
The NT senator, who was interim parliamentary leader when Joyce was contesting the by-election, has backed the leader to remain and says reports about Joyce's affair are the biggest beat-up he's seen.
The former chief-of-staff to Joyce, Canavan was also caught up in the revelations after he employed Ms Campion.
In a car-crash television interview, the NSW Nationals MP and leadership contender refused six times to support Joyce as leader, relying on the phrase "at the moment" before saying "of course" he supported him.
The former infrastructure minister was dumped in the December reshuffle, losing his portfolio to Joyce, seen by some as a rebuke from the leader.
The now-backbencher from Queensland was also dumped from his trade responsibilities in the reshuffle and has been laying low since the Joyce reports surfaced.
The outspoken MP and former party whip has backed his leader and says he continues to have the support of the party room.
Drum has also slotted in behind his leader and insists he understood Joyce's affair with Ms Campion was over when she moved to work in his office.
The outspoken Queenslander supports Joyce, saying he's one of the best politicians the country has had in his lifetime. Do you "shoot your best horse because he jumped the fence and was found in the neighbour's paddock".
The Victorian MP says Joyce must go to the backbench and will take a motion to colleagues on Monday when parliament resumes.
The Queensland backbencher believes the ball is in Joyce's court, but something needs to be resolved.
The fellow Queenslander admits there are obviously people concerned about the crisis but just as many, if not more, are supportive of Joyce.
The party's whip has been publicly standing by her leader and won't support a change.
One of the first to warn any revelations about the misuse of public money would mean trouble for Joyce, but has expressed confidence in him.
The NSW MP says he hasn't been approached by anyone talking about a Joyce challenge.
The former minister believes Joyce needs to work through his issues, but the coalition can move on with him at the helm.
The backbencher, who is facing his own High Court fight about questions over his eligibility, says he supports Joyce.
The NSW MP has no doubt Joyce is feeling very bruised, but has talked up the job he has done leading the party.
The NSW MP has said he has "no comment" on Joyce's future.
Issued a statement saying Joyce had lost their support because of the "ongoing damage" he is doing to the party.