Australian Federation of Disability Organisations chief executive Ross Joyce said the number of infections was a growing concern.
"People living with disabilities are very vulnerable [to COVID-19], due to their underlying health conditions," Mr Joyce told SBS News.
"We are deeply concerned about people living in congregate settings such as aged care.
"When [COVID-19] first started, they weren't allowed any visitors and weren't allowed to connect with their circle of support."
Mr Joyce said the lack of contact was a worry.
"It leaves people with disabilities very vulnerable, especially in an enclosed environment where they may be left open to abuse or neglect or exploitation," he said.
"Individual advocates are not allowed access to those premises as they are not treated as a worker, visitor or a priority role ... even support workers can't get in, in a lot of cases."
He also expressed concern about people with disability who are working within Australian Disability Enterprises (ADE's).
"A lot of people with intellectual disabilities are still working in ADE's packing and sorting," Mr Joyce said.
"We don't think any ADE should be operating and we think it's imperative on management to really seriously look at the health and safety requirements of their workers.
"All people with disabilities should be at home with online support."
Mr Joyce - who provided primary health care during the 2009 swine flu epidemic - said he was "quite amazed at the lack of understanding and training on infection control."
"10 years later we haven't learned lessons from the swine flu experience," he said. "Even knowing how to put on masks properly and when to dispose of them or how to clean."
"It needs to be part of a thorough induction for everyone working [in the sector] to understand the risk."
Women with Disabilities Victoria chief executive Leah Van Poppel said there is a "strong sense of caution and concern" about the potential for further outbreaks in group homes.
"That's a concern because they are closed environments and there's challenges about staffing in those environments," she told SBS News.
"If someone is really routine-bound or has particular access requirements it is difficult to move them around."
With self-isolation a major safeguard against the virus, Ms Van Poppel is also worried about a lack of accountability.
"As we become less visible, there might be a rise in violence toward people with disability," she said.
"It's not just the risk of the virus and its complications, but also the increased risk of violence because more people living with disabilities might not have a choice about how isolated they are.
"It's difficult to have people visit and check-in and we know particularly women with disability report a higher rate of violence than men with disability."
Premier Daniel Andrews says a government partnership has been set up to coordinate the state's measures in the disability care sector, similar to steps taken in aged care.
"Any cases amongst vulnerable groups are concerning but I think it's important to note it is a really important partnership and one that has served us and will continue to when it comes to those settings," he said.
Metropolitan Melbourne residents are subject to Stage 4 restrictions and must comply with a curfew between the hours of 8pm and 5am.
During the curfew, people in Melbourne can only leave their house for work, and essential health, care or safety reasons. Between 5am and 8pm, people in Melbourne can leave the home for exercise, to shop for necessary goods and services, for work, for health care, or to care for a sick or elderly relative.
All Victorians must wear a face covering when they leave home, no matter where they live.
People in Australia must stay at least 1.5 metres away from others. Check your state’s restrictions on gathering limits.
If you are experiencing cold or flu symptoms, stay home and arrange a test by calling your doctor or contact the Coronavirus Health Information Hotline on 1800 020 080.
News and information is available in 63 languages at sbs.com.au/coronavirus