Amnesty International's report is based on interviews with more than 100 Rohingya migrants who reached Indonesia.
The testimony quotes survivors who say they saw crew members kill people whose families did not pay the demanded ransom.
Amnesty International said the testimonies provide evidence that survivors of the Rohingya exodus from Myanmar this year may have resulted in "hundreds, maybe thousands" of deaths at sea compared to initial estimate of 370.
One 15-year old Rohingya boy described the abuse he experienced.
"In the morning you were hit three times," he said. "In the afternoon you were hit three times. At night you were hit nine times.”
Some migrants were said to be shot or thrown overboard, while others died of neglect.
“The shocking truth is that those we spoke to are the ‘lucky’ ones who made it to shore – countless others perished at sea or were trafficked into forced labour situations," said refugee researcher at Amnesty International, Anna Shea.
"Governments must do more to prevent this human tragedy from recurring.”
The UN estimates that at least 370 people lost their lives between January and June 2015 after Thailand's crackdown on human trafficking.
But Amnesty International said it believes the actual death toll to be much higher.
Australia criticised
The report also criticised the aid and resettlement assistance offered by Australia.
It said despite the UN launching an appeal for $A18 million, only 20 per cent of the funds have been raised.
"Although Australia gave $US749,000 ($A1.04 million) to UNHCR’s appeal, it did not deploy any of its considerable naval resources to assist with search and rescue," Amnesty International's report said.
"Furthermore, Australia refused to resettle any of the Rohingya who arrived in Indonesia in May 2015 or reverse its policy that hadended the resettlement of all refugees who registered with UNHCR-Indonesia after 1 July 2014."
A persecuted minority
Many of the more than 4,000 migrants who have landed in Indonesia, Malaysia, Thailand and Myanmar since the Thai government launched a crackdown on people-smuggling gangs are Rohingya who say they are escaping persecution.
Most of Myanmar's 1.1 million Rohingya, an ethnic minority living in western Myanmar, are stateless and live in apartheid-like conditions.
Almost 140,000 were displaced in deadly clashes with Buddhists in the state of Rakhine in 2012. They are denied citizenship and have long complained of state-sanctioned discrimination.
- with Reuters
