Cardinals gathered in Rome will vote again tomorrow on a new Pope after failing to reach agreement in the first ballot.
Black smoke from the roof of the Sistine Chapel just after 7.00pm signalled that the cardinals had failed to elect a new Pope.
More than 10,000 people watching from Saint Peter's Square initially broke into applause, thinking the election had been successful because the first puffs of smoke appeared white, before the narrow chimney on the chapel roof began billowing black fumes.
The 115 cardinals who will decide Pope John Paul II’s successor gathered in the Vatican's Sistine Chapel this afternoon to begin their deliberations.
The cardinals will be sealed off from the outside world until a new Pontiff is elected.
Earlier, the Dean of the College of Cardinals said a new Pope who would defend traditional teachings and reject attempts to modernise doctrine had to be chosen.
Cardinal Joseph Ratzinger made his comments in front of other cardinals at a mass in St Peter's Basilica.
The 78-year-old German cardinal is regarded as a favourite to succeed Pope John Paul.
A successful candidate must get the backing of 77 cardinals (two-thirds) to reach the required quota.
The Archbishop of Honduras, 62-year-old Oscar Rodriguez Maradiaga is also a favourite for the papacy, but other frontrunners include Argentina's Jorge Mario Bergoglio, Ivan Dias of India and Austria's Christoph Schoenborn.
For the duration of the conclave, the prelates will lodge at the purpose-built Santa Marta residence in the Vatican.
The conclave can choose whether to hold a first vote this afternoon or to continue its reflection and discussion before voting tomorrow morning.
An announcement is not expected until Wednesday or Thursday. White smoke and the tolling of St Peter's bells will mark the election of a new Pope.