The 3,000 residents across the nine towers, many of whom are from multicultural backgrounds, are not being allowed to leave their homes for any reason.
They are being monitored by 500 police officers per shift - a ratio of around one officer per six residents.
Why is this happening?
Health authorities have detected at least 27 COVID-19 cases across the North Melbourne and Flemington towers.
The towers are crowded and contain many communal spaces, meaning the risk of community transmission is high.
Mr Andrews acknowledged on Sunday the lockdowns would not be a pleasant experience.
"This is not going to be a pleasant experience for those residents, but I have a message for those residents: this is not about punishment, but protection," he told reporters.
"If we have a large group of people who already have pre-existing and underlying health challenges infected with this virus then people will die. It is as simple as that."
How is it going to work?
A number of logistical challenges will be encountered in keeping 3,000 people inside for five days.
The operation will see public health workers go door-to-door on every floor of the towers to test every resident that has not already returned a positive test.
In order to support residents, Mr Andrews said the state government will provide food, mental health support, domestic violence support and other healthcare for tower residents.
Deliveries of activity boxes for kids have begun, and a hotline has been established "to make sure help is available when and where it’s needed".
Mr Andrews said drug therapies, such as methadone, were also being added to a "comprehensive list of needs" for tower residents that was being updated “every hour”.
"There has been a lot of work overnight to make sure we can connect those people with the services that are very important to them, just as mental health support, just like basic food … right across the board," he said.
Tenants who are employed will also receive a $1,500 hardship payment to compensate for missing work, while tenants who are unemployed will receive a $750 hardship payment. Affected tenants will also not be charged rent for the next two weeks.
The state government said it has deployed translators to the buildings, who will be doorknocking to help explain directions and understand individual assistance tenants might need.
What are the residents saying?
Residents and community groups reported seeing police swarm the estates shortly after the announcement on Saturday, leading to panic and confusion.
The restrictions on tenants of the affected buildings are much tighter than the stay-at-home orders on 12 other Melbourne postcodes, and came without notice.
Anisa Ali, who lives in an estate in North Melbourne with her mum and 10-year-old brother, told SBS News she feels people in the towers are being made scapegoats.
"We are literally trapped in our flats as if it were a prison cell. It's quite inhumane and it's quite unfair," she said.
Hana Ibrahim, who lives in one of the Flemington towers with her elderly parents and her two-year-old son, said there is lots of confusion about what is going on.
"Why are there so many police here and why are they guarding us?" she said.
"We were not given enough time, not given enough information. Until today no-one has come to knock on the door and tell [us] what is going on. It's just TV and some people don’t have TVs in their apartments."
Father-of-five Abdirahman Ibrahim said the hyper-localised lockdown is "not fair".
"What the Victorian government is doing is discriminating [against] the people who don't have a voice. We are not different from the rest of the Victorian community, which is in lockdown now," he told AAP.
Some residents of the towers say while the lockdowns came as a shock, they are happy something is being done.
"We have been scared, and we have thought that the government had forgotten us," Flemington resident Awatif Taha wrote in The Guardian.
What other concerns are there?
Legal and community groups have voiced concerns about the possibility of racial profiling by police, given many tower residents are from multicultural backgrounds.
The Flemington Kensington Community Legal Centre's Dan Nguyen said the group was was "very concerned" about what Mr Andrews called a hard lockdown.
"Residents have been reporting a lot of shock. There’s a lot of anxiety and a lot of fear with very little information. The information that came by was almost instantaneous ... and it came from the police on their doorstep," he said.
"Policing seems to be the centralised approach here in Victoria and it just can’t be. We need a public health approach."
Victorian Council of Social Service CEO Emma King agreed, saying nurses and social workers need to be allowed to lead the response alongside police.
"A number of people in the estate come from communities where there’s been war and from highly-traumatic experiences. So having numbers of police turn up on their doorstep is really traumatic experience," she said on Sunday.
"It's our responsibility in the community to support people who are living in the estate. We don't need lights and sirens, we need nurses and social workers.”
Others have questioned how the operation will be safely and hygienically carried out, given there is an inquiry underway into Victoria’s hotel quarantine program that has allegedly seen under-trained staff not using personal protective equipment.
Asked by reporters how equipped authorities deployed to the housing estates are, Mr Andrews said he has confidence that all staff have been "appropriately trained and briefed".
How has the community reacted?
Many Victorians have rallied behind the affected residents, however given police are not letting anyone in or out of the buildings, it is unclear how much community support will make it inside.
Victorian Housing Minister Richard Wynne said he was "overwhelmed by the compassion" of people who wanted to provide support to the towers in lockdown.
"Presently my phone is running off the hook with people saying, 'how can we help?'," he told reporters on Sunday.
"From the Lord Mayor of Melbourne to the City of Moonee Valley [and] Trades Hall, many numbers of people have said 'we want to be able to help'."
A crowdfunding campaign for residents, organised by the Victorian Trades Hall Council, had raised more than $250,000 by Sunday afternoon.
On Saturday, tenants in the buildings reported being unable to access information about the lockdowns in languages other than English, prompting community members to translate messaging in-language.
Members of Melbourne's Sikh community on Sunday posted photos on social media of volunteers set up outside of some of the housing estates to provide free meals.
People in Australia must stay at least 1.5 metres away from others. Check your state’s restrictions on gathering limits.
Testing for coronavirus is widely available across Australia. If you are experiencing cold or flu symptoms, arrange a test by calling your doctor or contact the Coronavirus Health Information Hotline on 1800 020 080.
The federal government's coronavirus tracing app COVIDSafe is available for download from your phone's app store.
SBS is committed to informing Australia’s diverse communities about the latest COVID-19 developments. News and information is available in 63 languages at sbs.com.au/coronavirus
Additional reporting by Jarni Blakkarly, AAP.