About 1,000 people were gathered outside the capital's main international railway station on Tuesday morning, with many heard shouting "Germany, Germany" - where they are hoping to travel.
Before being blocked from access to trains, thousands of refugees made it to Vienna and Munichover the past few days.
Police in Vienna said 3,650 refugees had reached the city on Monday, representing a new daily record.
'Germany, thank you!'
Many of the refugees arriving in Vienna's railway station on Monday evening immediately raced to board trains heading on to Germany, as policemen looked on passively, preferring not to intervene, witnesses said.
Refugees cheered and chanted "Germany, thank you!" as they saw a welcome sign held up by local people at Munich Central Station late on Monday.
Police said some 500 refugees, mainly from the Middle East, were on the train. They were to be transported by bus to camps where they will be registered.
Hungary, the gateway to the EU from Eastern Europe, is hoping to finish erecting a 175km razor-wire fence along its border with Serbia to stop refugees from crossing into the country.
The move was heavily criticised by France, whose foreign minister said his country would not even use such a barrier for animals.
EU countries will hold an emergency meeting on September 14 to discuss the refugee crisis.
The number of refugees reaching the EU's borders reached nearly 340,000 during the first seven months of the year, up from 123,500 during the same period in 2014, according to the bloc's border agency Frontex.
The majority of these refugees were fleeing the raging conflicts in the Middle East.
Source: Agencies
