Most of the 56 people were citizens of Nigeria, Senegal, Morocco or Bangladesh, the border guard said. Some had forged documents.
They included more than a dozen people who were removed from Kaliningrad-bound trains long after the last World Cup game was played in the city on Thursday, a border guard statement said.
All of them had the Fan ID, which is issued to World Cup ticket holders and provides visa-free travel in Russia and Belarus but not in Lithuania.
All trains between Russia's isolated Kaliningrad enclave and Belarus, Russia's ally, pass through Lithuania, a member of NATO and the European Union since 2004, which requires proper paperwork for the 235-km (150 mile) stretch of railway on its territory.
About 100 Nigerian soccer fans were stranded in Kaliningrad following their team's loss to Croatia in their World Cup opener after what was described as a visa gaffe.
Most of the supporters had planned to take the train from Kaliningrad back to Moscow following the Super Eagles 2-0 loss but did not have the necessary visas to pass through Lithuania.
Russian officials stepped in to help the marooned fans by arranging flights, but they were not free.
