Ratings agency Standard & Poor's said there were "important considerations and uncertainties" that factored into the creditworthiness of banks should Scotland vote to end its 307-year union with England in September.
Britain's three main political parties have been campaigning fiercely to keep the union intact, arguing that both countries are better off together.
But Scotland's nationalists believe a split would give them the economic freedom to prosper.
In a report, S&P said it counted the existence of a Scottish central bank, the Scottish government's attitude towards helping struggling banks, changes to financial regulation and independent Scotland's currency among crucial factors that could impact its ratings on the country's banks.
It added that its ratings on British banks currently assumed that the UK government would provide extraordinary support to systemically important banks under stress.
"The willingness and ability of the Scottish government to support its banking sector appears challenging," S&P said, highlighting that the Scottish banking system's assets are currently a high 1,254% of Scotland's GDP.
This compares with 880% for Iceland in 2007, just before its banking system collapsed, the ratings agency said. It currently classifies Iceland as "support uncertain," factoring no extraordinary government help into domestic bank ratings.
"We note a possible parallel here with Iceland, where in 2008 the national deposit insurance scheme could not honour claims when the country's outsized banking system failed."
S&P said if Scotland chose to give up the British pound with the Bank of England's safety net, it would need credible deposit insurance arrangements on standby to bail out too-big-to-fail banks and instil depositor confidence.
The agency added that, if Scotland chose to leave the pound, creating a separate banking market would mean higher costs for all banks operating in the country.
New York-based money manager BlackRock said last month that Scotland would be better off launching its own currency rather than keeping the British pound or joining the euro.
The report follows a slew of warnings from other financial services companies, including RBS, Standard Life and Barclays. The bosses of oil majors BP and Royal Dutch Shell have also voiced arguments against a "yes" vote for independence.