The fund has already been knocked down once by Labor, which takes issue with the government's plan to take cash for it away from an existing infrastructure kitty.
The Government's Senate Leader Mathias Cormann said the legislation was a test for Labor.
"This is the time for the Labor party to show leadership and show who's side they are really on," Senator Cormann told reports on Monday.
Labor Leader Anthony Albanese said there was no rush to pass the legislation.
"It won't flow this week, it won't flow this month, it won't even flow this year. It won't flow this financial year," Mr Albanese said on Monday.
Labor wants the money to come from general revenue, rather than the Building Australia fund which was set up by Labor to fund road and rail projects.
Drought Minister David Littleproud said the Building Australia fund had failed to deliver a dividend in five years.
"It's done three parts of bugger all," he said.
The government also wants to pass laws to deal with farm invaders and those deemed to be encouraging such actions, which could result in up to five years' imprisonment.
Labor wants to see a final Senate committee report on the laws before it decides whether to back them.
The Opposition supports the intention of the bill - to protect farmers from trespassers - but is worried it may have unintended consequences.
"Questions have been raised about the significant overlap between this bill and existing state criminal laws, and whether this bill in fact improves protections for farmers," a spokesman for Labor legal affairs spokesman Mark Dreyfus told The Australian.
The Greens said there are already laws to deal with such crimes.
"The concern here is that we've seen this government take us closer to a police state with passage of a range of laws that hand over more power to authorities with a whole range of unintended consequences," Richard Di Natale told ABC Insiders on Sunday.