Authorities said the remains were found in two graves in a Catholic cemetery in the western town of Menden by investigators working on evidence from wartime witnesses.
Ulrich Maas, the state prosecutor in the city of Dortmund reported that in one of the graves, the remains of 22 children were found piled on top of each other along with two adults.
The other contained the bodies of men and women.
Mr Maas said it was suspected that the dead, particularly the children, were disabled people who were subjected to euthanasia by the Nazis during World War II.
The Dortmund state prosecution is trying to prove the theory. Results of DNA tests to be carried out on the remains are due around the end of the month.
The German press agency DPA cited a historical expert saying that the euthanasia theory was difficult to prove.
The Nazis used killing methods, such as poisoning and starvation that would leave no trace.
Under Nazi rule, Hitler's personal doctor Karl Brand, who was in charge of the regime's euthanasia programme, had a hospital built near the cemetery.
More than 70,000 people who were disabled or judged physically weak died under the Nazis, most of them killed by injections of gas or poison liquids.
The contemporary witnesses said as many of 200 people killed by the Nazis may be buried in the Menden cemetery. A funeral service is to be held after the excavations are completed.
The remains will then be re-interred in the cemetery.