But the AU said it was holding out hope that dissident factions of the two groups would accept the Darfur Peace Agreement (DPA) and said the AU Peace and Security Council would meet soon to discuss possible options.
The groups ignored the latest deadline of May 31 for them to sign the deal that was last month approved by the main faction of the rebel Sudan Liberation Army.
A spokesman for Mr Annan said the UN chief believes the peace deal is the only way to address the devastating conflict.
"The secretary general deeply regrets this failure and will remain in touch with the African Union leadership on the possible follow up measures," Mr Annan's spokesman said.
AU commission chief Alpha Oumar Konare said the refusal of the Justice and Equality Movement (JEM) and a camp of the Sudan Liberation Movement (SLM) is disappointing.
Mr Konare "notes with deep regret that despite all the efforts deployed ... Mr Abdulwahid of the SLM and Khalil Ibrahim of JEM have failed to sign the DPA," he said in a statement.
But he also said he is pleased that some members of the two holdout groups had expressed interest in signing the deal, which was signed on May 5 by Khartoum and the main SLM faction.
Shortly after the statement was released, a group of 27 Darfur rebels claiming to represent splinter factions of holdout groups met with AU Peace and Security Council commission Said Djinnit, in a bid to work out an arrangement.
The peace deal aims to end three years of conflict in Darfur, which has left around 300,000 people dead and 2.4 million homeless.
It calls for a more equitable distribution of power and wealth, the disarming of the pro-government Janjaweed militias and a referendum on the future of Darfur.
But the holdouts say it fails to address their concerns and have refused to sign on unless substantial changes are made.