The accident happened approximately 200 metres from the Great Mosque, Islam’s holiest shrine.
Rescue workers used heavy lifting machinery and sound detecting equipment to search for survivors trapped in the rubble of the multi-storey building.
After inspecting the site the deputy governor of Mecca, Dawoud al-Fayez, said it was too early to provide a final death toll.
A health ministry spokesman, Khaled al-Mirghalani said "at least 20 were killed and 59 wounded."
But witnesses have said that at least 23 people have died.
Major General Alwani Jedawi, head of civil defence for the pilgrimage, said that the hostel, known as Luluat Al-Kheir (Pearl of Grace) was old.
Emergency services had to hold back anguished survivors who tried to search for relatives amongst the wreckage themselves.
"My two brothers are inside," Tunisian pilgrim Aiysha bin Jaber, 66, begged security personnel who barred her from the ruin.
Collapse followed fire
Witnesses have said the hostel collapsed after a fire broke out in the building.
French pilgrim Abderrahmane Ghoul said that a firefighting helicopter tried to tackle the initial blaze.
"It started with a fire in the building. A helicopter started to sprinkle water to put out the fire. Afterwards, the building collapsed," he said.
Talhah al-Mazi, 40, said the cave-in happened as pilgrims finished midday prayers in the square outside.
“I saw people rushing out, crying and screaming for help," he said.
The cause of the fire has not been established yet but in previous years, camp fires have been known to spark infernos in pilgrim encampments.
Prayers continue
Despite the tragedy, hours after the collapse hundreds of thousands of pilgrims continued their rituals of faith, gathering at the nearby Great Mosque for sunset prayers.
An estimated 2.5 million pilgrims are expected to converge on Mecca for the Haj, one of the five pillars of Islam, a pilgrimage Muslims are required to make at least once in their lifetime.
Saudi authorities had set a midnight Wednesday deadline for the last pilgrims to arrive in the kingdom.
A force of 60,000 security, firefighting and civil defence personnel has been deployed to prevent deadly stampedes and structural failures that have happened in previous years.
Stampedes killed 251 people in 2003 and 1,426 in 1990.