Dr John Gee, an expert on chemical weapons, worked with US-led weapons hunter, the Iraq Survey Group, after the war and wrote the critical six-page letter when he decided to resign, Fairfax newspapers report.
In the letter he warned the federal government the hunt was "fundamentally flawed" and that there was a "distinct reluctance on the part of many here and in Washington to face the facts" that Iraq had no weapons of mass destruction.
Dr Gee recorded in an email soon after that "Downer has issued instructions it (my letter) is not to be distributed to anyone".
A spokesman for Mr Downer said the minister "did not recall" receiving Dr Gee's letter but said he would check.
But he described as "a conspiracy theory" material showing that the letter had not been given to the head of the Defence Department.
Mr Downer has previously acknowledged he was personally briefed by Dr Gee on the expert's return from Iraq, but has never revealed the contents of that briefing.