Indian Prime Minister Manmohan Singh paid his condolences of the passing of the great South African leader.
"This is as much India’s loss as South Africa's," he wrote on Twitter.
"He was a true Gandhian. His life and work will remain a source of eternal inspiration for generations to come. I join all those who are praying for his soul."
Meanwhile, Indian President Pranab Mukherjee hailed Mr Mandela as “a statesman, world leader and icon of inspiration of humanity”.
Mr Mandela, a follower of Mahatma Gandhi’s teachings, made his first trip to India in 1990, only a few months after being released from 27 years in jail.
In the same year Mr Mandela was the second non-citizen to receive the Bharat Ratna, India’s highest civilian award.
To honour his release from jail, a residential street in the southern city of Kochi was named after him.
The nation’s capital also had a road named after the leader, illustrating the influence he has had on millions of Indians.
Meanwhile, thousands of Indians have taken to social media to pay their respects to Mr Mandela.
Ramachandra Guha, author of 'India after Gandhi', tweeted: "Nelson Mandela was, next only to Mahatma Gandhi, the greatest peron of the 20th century."
Mr Mandela frequently praised Gandhi for his principles of ‘Satya and Ahimsa’ – the belief that truth must pervade all considerations of politics and that the concept of nonviolence must be upheld.
"I could never reach the standard of morality, simplicity and love for the poor set by the Mahatma," Mr Mandela said in a speech in 1995 in western city of Ahmedabad.
"While Gandhi was a human without weaknesses, I am a man of many weaknesses."
Indian women’s activist, Ilmana Fasih, told SBS she believed Mr Mandela was deeply inspired by Gandhi.
"Gandhi stood for non-violence and truth in the same way Mandela’s truth and reconciliation was a non-violent resolution of Apartheid mess," she said.
"His resolve to not be bitter and forgive was also Gandhian as he too preached forgiveness and compassion. I think like Martin Luther King, Mandela also considered Gandhi as an icon."
Other politicians also paid their respect, drawing parallels between the two great men.
"Mandela believed in Gandhian methods and philosophy and became an iconic figure who gave up his own freedom to ensure freedom for his people," Rajnath Singh, National President of the Bharatiya Janata Party (BJP), tweeted.
