The creation of the panel comes days after two people in their early 20s died, three more revellers were taken to hospital in a critical condition and hundreds of others fell ill at the weekend's Defqon.1 music festival in Penrith.
Premier Gladys Berejiklian on Tuesday said the expert panel would consider increased penalties for drug dealers and how festival promoters could improve safety.
However, it won't consider pill testing, despite the Australian Medical Association and other experts calling for a trial.
"We do not support pill testing," Ms Berejiklian told reporters in Sydney.
"We do not support a culture which says it's OK to take illegal drugs."
AMA president Tony Bartone on Monday said: "proper co-ordinated clinical trials" were needed to see if pill testing did have a role to play.
Kieran Palmer from the Noffs Foundation told ABC TV evidence from overseas and Australian trials showed pill testing "works to reduce overall drug consumption at festivals, it reduces the harm associated with drugs and by extension it reduces deaths".
That's partly because revellers are often made aware their pills don't contain what they think they did.
Ms Berejiklian on Sunday vowed to shut down the Defqon.1 festival but she was sounding more conciliatory on Tuesday.
"The event over the weekend, I believe, did not meet the safety requirements expected of such events," the premier said.
"Which is why the event, in its current form, needs to improve its safety requirements and I'll be getting advice from this panel (on that)."
The premier said the festival had now had four deaths in five years.
"I want to make sure that event changes, or steps up in its safety, otherwise it shouldn't proceed."