WASHINGTON DC (Reuters) - When the U.S. Supreme Court talks about religion, all hell breaks loose.

A dispute over an upstate New York town's prayer before council meetings produced an unusually testy oral-argument session on Wednesday that recalled the decades of difficulty Supreme Court justices have had drawing the line between church and state.

Court decisions involving freedom of religion tend to be closely decided with many separate opinions rather than clear-cut majority statements. The case of Town of Greece v. Galloway appears to be headed that way.

In the case brought by two Greece residents who objected to the overwhelmingly Christian prayers at meetings, the justices appeared likely to allow legislative prayer to continue but not ready to offer new guidance for when government might have gone too far in favoring, for example, Christianity over other faiths.

The more liberal of the nine justices appeared sympathetic to the challenge while the conservatives who control the court's majority seemed ready to back the town - but not with a single obvious rationale.

At one point during the hourlong session, Justice Stephen Breyer referred to the challenge of setting constitutional rules so people of different religions live "harmoniously together." Not soon after, Justice Elena Kagan asserted that, "Every time the court gets involved in things like this, it seems to make the problem worse rather than better."

Overall, the justices' remarks were more pessimistic than positive regarding a possible consensus. They voiced frustration with the lawyers who appeared before them and with each other as well.

When Breyer asked the town's lawyer if officials could take more steps to invite non-Christians and even people who are not religious to offer the equivalent of a prayer, Justice Antonin Scalia mockingly asked what sort of invocation "somebody who is not religious" could offer. As lawyer Thomas Hungar, representing the town of Greece, began to suggest a chant "of guidance and wisdom," Breyer interjected with some annoyance toward Scalia, "Perhaps he's asking me that question and I can answer it later."

ENDURING DILEMMAS

Such have been the tensions over one of America's most enduring dilemmas. The U.S. Constitution guarantees the free exercise of religion while requiring the separation of church and state.

Neither dictate is absolute and the court has struggled with religious protections, often leaving lower courts with confusing standards, particularly on the mandate at issue on Wednesday - that government "make no law respecting an establishment of religion."

Retired Justice Sandra Day O'Connor, often a deciding vote on religion during her 1981-2006 tenure, attended the session in the ornate courtroom. She sat, expressionless, in a front-row spectator seat.

The religious makeup of the bench has changed since O'Connor's time. The court has historically been dominated by Protestants but it is now has six Catholic and three Jewish justices. The religious character of the justices, however, is unlikely to control their views. Among the Catholics, for example, are Scalia, a conservative who has voted repeatedly for prayer in public settings, and Sonia Sotomayor, a liberal who seemed more supportive of those challenging the prayer policy in Greece, a suburb of Rochester.

The high court heard the case after the U.S. Court of Appeals for the 2nd Circuit ruled that Greece officials relied on overwhelmingly Christian clergy, appearing to endorse their beliefs. The court noted that prayers at the town's legislative sessions regularly mentioned Jesus Christ and the Holy Spirit. The appeals court differentiated the Greece dispute from a 1983 Supreme Court case allowing non-sectarian prayers in the Nebraska legislature. In that decision, Marsh v. Chambers, the justices relied on the historic nature of legislative prayer.

In key Supreme Court cases since 1983, the justices have focused on whether government might be endorsing a particular religion or coercing people to participate in prayer. In a 1989 case, they rejected a Pennsylvania county's courthouse crèche display and in 1992 they struck down a Rhode Island public school district's prayer at graduations.

In Greece's appeal, Hungar urged the justices to consider America's history of legislative prayer dating to its 18th century founding. The Obama administration filed a brief in support of the town.

"KEEPING GOVERNMENT NEUTRAL"

Douglas Laycock, a lawyer representing town resident Susan Galloway and other challengers to the Greece prayer sessions, countered that the town's practice "coerces" people who attend meetings to join Christian invocations. Laycock said he was not arguing against non-denominational, inclusive prayer.

"Give me an example of a prayer that would be acceptable to Christians, Jews, Muslims, Buddhists, Hindus," Justice Samuel Alito insisted. As Laycock hesitated, Alito kept repeating, "Give me an example. ... Give me an example."

Chief Justice John Roberts also was skeptical of Laycock's rationale. "What exactly is coercive in this environment?" he asked. "Having to sit and listen to the prayer?"

To sharp questioning by Justice Anthony Kennedy, who has succeeded O'Connor as a pivotal vote among the nine, Laycock insisted the court could write a rule that would ensure legislative prayers were not exclusively Christian but that did not require excessive screening of prayers by officials.

Laycock disagreed with Kagan's remark that the court seems to be making the law worse rather than better.

"I don't think that's true," Laycock said. "There are people who distort your decisions. There are people who misunderstand your decisions honestly ... But keeping government neutral as between religions has not been a controversial proposition in this court."

(Reporting by Joan Biskupic; Editing by Howard Goller, Amy Stevens and Bill Trott)