The adults, who have intellectual and psychosocial disabilities, launched the class action in the Federal Court on Tuesday against Adrian Powell, who ran the Grand Western Lodge boarding house, his employer and the NSW government.
They allege that between 2000 and 2011, Mr Powell participated in and encouraged physical assaults on residents at the lodge in Millthorpe, near Orange.
It is also alleged that residents were confined as a form of punishment and that unprescribed quantities of psychotropic medication was used to sedate some of them.
Maurice Blackburn Lawyers, which is representing the residents, said they were seeking compensation for financial losses, injuries and false imprisonment.
"These residents lived in depraved circumstances that you wouldn't wish on your own worst enemy," Maurice Blackburn's NSW managing principal, BenSlade, told a press conference at the launch of the action.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"They were paying 100 per cent of their pensions for the pleasure of living in this house of horrors."
The class action, which also includes allegations that residents weren't given enough food and clothing, comes after allegations of abuse at the boarding house first surfaced in 2002.
Despite this, Mr Slade said the state government failed to act.
"These allegations are particularly distressing as they suggest that some of the most vulnerable people in our community have suffered so much for far too long," he said in a statement.
As part of the claim, it is alleged the NSW Department of Ageing Disability and Home Care was negligent in monitoring its licensees and that it knew there was a risk of harm to residents but failed to act.
Matthew Bowden, co-CEO of People with Disability Australia, told SBS Radio the class action exposed serious human rights abuses.
"People have spoken to us about not having any freedom of movement or freedom of association," Mr Bowden says.
"People talked to us about living in fear daily and that this fear extended to fears around their personal and physical saftely. People also have made allegations that they were forcibly drugged, that they recieved medications to shut them up."
NSW Minister for Ageing and Disability Services John Ajaka would not comment as the matter is before the courts.
