But fast-forward a few weeks, and many are changing their minds.
There has been no violence to speak of, Russians have surprised visitors with the warmth of their welcome and their penchant for a party and, speak it softly, the England team are going strong.
Only 2,500 English fans watched England's first game in Volgograd, and only a few more were in Moscow on Tuesday to see their side win a World Cup penalty shootout for the first time ever, holding their nerve to beat Colombia for a first place in the quarter-finals since 2006.
England goalkeeper Jordan Pickford said the English supporters had been so outnumbered and out-shouted by Colombians that the tie had felt like an away match.
Londoner Matthew Jordan was there, though, and said he had quickly spent 275 pounds to travel to Samara with three nights' accommodation for Saturday's match against Sweden.
"It's not every day you have the opportunity to see your country in a World Cup ," he said. "There's a real buzz around the team at the moment, and I'm sure most fans will do their utmost to be in Samara on Saturday."
Back in England, 25-year-old Kezsa Mitchell said she was trying to book time off work with her fiance to travel to Russia after being assured by other supporters there had been no issues with safety or fan violence.
"We only decided to go over last night in the excitement of winning penalties," she said. "We went to the Euros (in France) two years ago but saw the violence between Russian fans and England fans, so that put us off going to the World Cup."
A spokesman for the British Embassy in Moscow said authorities now expected a higher turnout for Saturday's match, even though it takes place in Samara, more than 850 km (530 miles) south-east of Moscow.
Ryan, a 25-year-old from Bedfordshire, said he had hoped to watch England play Belgium in their final group match last week, but family and work commitments meant he couldn't travel at short notice.
"Obviously you want an environment where you are going to feel safe and not sort-of worried about getting beaten up and into trouble," he said. "But I think the success of the team is the main thing that would prompt me to want to come out."
"I'm sort of wishing I could have been out there now. It looks like it would have been a good one."
(Editing by Kevin Liffey)
