"And indeed, statements issued in Pyongyang have indicated suspicion of the fact that the United States is referring to human rights."
Mr Kirby chaired a United Nations probe into human rights abuses in North Korea, which wrapped up in 2014.
"There will never be peace on the Korean peninsula while there are grave human rights abuses occurring in North Korea," he said.
His comments come as US President Donald Trump hinted there would be imminent news about three Americans detained in North Korea.
"The past Administration has long been asking for three hostages to be released from a North Korean Labor camp, but to no avail. Stay tuned!" Trump wrote on Twitter.
But Mr Kirby said "I do hope that the US will not be concentrating only on the human rights of its own citizens in North Korea".
"Much more important, in terms of numbers, are the issues that were revealed in the reports of the Commission of Inquiry of the United Nations, which I chaired, and that is about the human rights of millions of people in North Korea."
He said this should be a "very important card that will have to be played in negotiations between (Donald Trump and Kim Jong-un) when they meet".
Mr Kirby's Commission of Inquiry on Human Rights in the Democratic People's Republic of Korea found the North had committed "crimes against humanity".
It said this included "extermination, murder, enslavement, torture, imprisonment, rape, forced abortions and other sexual violence (and) persecution on political, religious, racial and gender grounds".
"Crimes against humanity are ongoing in the Democratic People's Republic of Korea because the policies, institutions and patterns of impunity that lie at their heart remain in place," it said.