Supreme Court Project
The Supreme Court Project attempts to build on and extend empirical work explaining the United States Supreme Court’s decision making and its review of the 13 United States Courts of Appeals. We have started our inquiry by looking at the question of how to measure the Supreme Court’s reversals of the United States Courts of Appeals (Phase I) and expect to continue our analyses by building empirical models explaining the variation in reversal rates across the circuits and predicting Supreme Court Justice voting and Court decisions (Phase II).
Phase I. Initially, the Project examines whether there is a better way to track how well the Courts of Appeals do before the Supreme Court, conventionally measured through reversal and affirmance rates. Traditionally, the reversal rate for a circuit court is an outcome-based, score-card test: For any given term of the Supreme Court, a reversal rate can be established by dividing the number of appeals where the Court reversed the circuit by the total number of appeals decided by the Supreme Court for that circuit. While straightforward, this traditional method suffers from two fundamental weaknesses: (1) it measures outcomes rather than legal reasoning; and (2) it fails to account for decisions implicitly reversed or affirmed by the Supreme Court, even though not directly before the Supreme Court on appeal.
Rule 10(a) of the Supreme Court Rules explains that a reason for granting certiorari is that a Court of Appeals “has entered a decision in conflict with the decisions of another United States court of appeals.” Our analysis confirms this as, for the six terms of the Roberts Court (2005-2011), nearly half of the Supreme Court’s annual merit decisions address substantive legal issues for which there is a circuit split. For each of these cases, the Supreme Court ruled on a legal issue addressed not just in the case on appeal, but also one or more “shadow decisions” (i.e., court of appeals decisions that have ruled on a legal issue that comprises the circuit split). Any measure of reversals and affirmances that does not account for these shadow decisions is therefore incomplete and potentially misleading.
We, therefore, have established a “full” measure of the circuit courts’ reversal and affirmance rates, taking into account both decisions on direct appeal and these shadow decisions. This measure, we believe, provides a more robust and more accurate view of the relationship between the United States Courts of Appeals and the United States Supreme Court.
We discuss our preliminary results in an article, “Towards a Better Measure and Understanding of U.S. Supreme Court Review of Courts of Appeals Decisions,” published in BNA Weekly. Generally, we conclude:
- the Supreme Court reverses the Courts of Appeals less often than is commonly thought;
- the full measure of reversals identifies different Courts of Appeals as least reversed than those identified under the traditional method;
- in resolving circuit splits, the Supreme Court affirms the majority approach of the circuit approximately 50% of the time;
- our analysis permits us to construct a concordance table showing the degree to which the various Courts of Appeals agree with each other, much like the concordance tables showing agreement among justices' voting.
Overall, we believe this analysis, which takes into account both cases directly on appeal and shadow decisions, provides a more robust and more accurate view of the Supreme Court’s review of the Courts of Appeals.
For those interested in more detailed Tables and Charts regarding this analysis, they are available here.
Phase II. While our Phase I work identified variations in circuit court reversal rates, it did not attempt to explain the reasons for variations. In Phase II we will apply econometric techniques to test various hypotheses that explain these variations.
One hypothesis, for example, is that the political party (or appointing president) of a justice and the members of a circuit court panel may, in part, explain the justice’s decision to affirm or reverse; a justice and a judge appointed by the same president or a president from the same party may, all else equal, lead to a lower probability of reversal. Other hypotheses include: whether a justice is less likely to reverse (1) a court of appeals over which s/he is the circuit justice; (2) a judge who attended the same law school as the justice, or (3) a judge who is the same gender as the justice.
Clearly, a few caveats are in order. First, a statistical correlation is only just that and needs to be supplemented by an underlying rationale; a statistical finding that left-handed justices reverse left-handed court of appeals judges more often than right-handed ones, for example, would likely be spurious. Nevertheless, data driven inquiries such as these (and others) are important in revealing patterns that actually occur. We welcome interested readers to email us with suggestions of plausible theories that could be tested with available data.
Second, the tests of these hypotheses should be part of a broader statistical model of Supreme Court Justice decision making. In that way, we anticipate that this work will build on the pioneering work of, among others, the Washington University Supreme Court Forecasting Project, which applied statistical techniques to predict Supreme Court Justice voting behavior and court decisions. We hope to extend that work by, among other things, updating their results for the Roberts Court and employing additional explanatory variables (including those relating to the Courts of Appeals).
We have reviewed all of the merits decisions in the seven terms since Chief Justice Roberts joined the Court in 2005. Although there have been several changes in the composition of the Court during that time (Justices Alito, Sotomayor, and Kagen joined, as Justices Stevens, O’Connor, and Souter left), the six-year period was chosen because it provided a coherent block of recent years to study.
The goal of the “full” method for assessing reversal and affirmance rates of the courts of appeals is to measure how often the legal standards applied by the circuit courts are reversed or affirmed by the Supreme Court. To this end, rather than measuring case outcomes – which is limited to the cases appealed to the Supreme Court each term – we measure the Supreme Court’s assessment of the legal standard applied by the lower courts, whether on direct appeal or involved in a relevant circuit split. This difference in measurement creates occasional differences in results when using the traditional method and the full method. For example, if the Supreme Court affirms a circuit court decision on alternate grounds, the traditional methodology assigns that circuit court an affirmance whereas the full methodology assigns it a reversal. See, e.g., Union Pacific R. Co. v. Brotherhood of Locomotive Engineers and Trainmen General Committee of Adjustment, Cent. Region, 130 S.Ct. 584 (2009) (affirming Seventh Circuit Court of Appeals on alternate grounds). The same is true of the converse: if the Supreme Court adopts the legal standard of the circuit court below but reverses for a separate reason, the traditional methodology assigns that circuit court a reversal whereas the full methodology assigns it an affirmance.
This analysis involved a two-step process which requires identifying circuit splits and then determining whether or not each of the shadow decisions involved in the circuit split has been affirmed or reversed. To do so, we employed the following process:
In identifying circuit splits, we relied on the Supreme Court to provide guidance. In many instances, the Supreme Court references a circuit split on its own, as well as the cases involved in the split. On rare occasions, the Supreme Court would identify a circuit split, but refer to a circuit court opinion cataloguing the circuits and cases involved in that split. See, e.g., DePierre v. U.S., 131 S.Ct. 2225 (2011) (referring to circuit split discussed in First Circuit opinion below). In those instances, and only those instances, we looked beyond the Supreme Court to identify the opinions involved in the circuit splits.
Once a circuit split was identified, we determined whether the shadow decisions had been affirmed or reversed. In doing so, we applied a rigorous test: a circuit court’s legal standard would only be “affirmed” if the Supreme Court wholly adopted its reasoning. Even cases in which there was close congruence between the shadow decision’s legal standard and the standard adopted by the Supreme Court would be assigned a “reversal”. For example, in the Court’s 2010 term, it decided Fowler v. United States, a case analyzing 18 U. S. C. §1512(a)(1)(C), which makes it a crime to “kill another person, with intent . . . to prevent the communication by any person to a [federal] law enforcement officer” of “information relating to the . . . possible commission of a Federal offense.” In adopting its own standard to assess this statute, the Court vacated and remanded the Eleventh Circuit’s opinion below, as well as implicitly reversing shadow decisions from the Second, Third, Fourth, Fifth, and Eighth Circuits. Although these circuits had applied standards similar to the Supreme Court’s ultimate ruling, because the Supreme Court did not specifically adopt the ruling of any of the courts below, they all received a mark for a reversal.
In rare circumstances, the Supreme Court granted certiorari in a case that involved more than one substantive legal issue. For such cases, we made a judgment as to which was the more substantive legal issue. If there was a circuit split on that issue, the shadow decisions were assessed. For example, in Milavetz, Gallop & Milavetz, P.A. v. U.S., 130 S.Ct. 1324 (2010), the court rules on two issues: (1) a statutory interpretation of the meaning of a provision of the Bankruptcy Abuse Prevention and Consumer Protection Act, and (2) a constitutional interpretation of whether that provision violated the First Amendment. Because the constitutional issue was more substantive (in our minds), we analyzed only that issue. For cases in which a circuit split was identified, but not central to the legal issue before the court (i.e., dicta circuit splits), that circuit split was not assessed.
Finally, because the “full” method analyzes the Supreme Court’s assessment of the substantive legal standards applied by the circuit courts rather than outcomes, we removed from our data set cases in which the court did not opine on a substantive legal standard from a court below. Between 2005 and 2010, there were 35 such cases. These cases fell into the following categories: original jurisdiction cases; appeals directly from district courts; cases in which the court determined that certiorari had been improvidently granted; cases in which the court dismissed for lack of jurisdiction; cases involving applications for stays; and cases the court vacated as moot.
“A Sixth Sense: Sixth Circuit has Surpassed the Ninth as the Most Reversed Appeals Court,” ABA Journal, December 2012 (written by Mark Walsh)
"The 'Full' Method of Measuring the Court's Review of Decisions by the Courts of Appeals," SCOTUSblog, October 23, 2012
"Towards a Better Measure and Understanding of US Supreme Court Review of Courts of Appeals Decisions," BNA's The United States Law Week, September 27, 2011
"The Third Circuit's Reversal Rate: A Success Story," The Legal Intelligencer, November 10, 2011
"First Circuit Reversal Rate Not What It Seemed," Massachusetts Lawyers Weekly, January 30, 2012
John S. Summers is a shareholder at Hangley Aronchick Segal Pudlin & Schiller. He received a BA from Wesleyan University in 1980 and a JD from the University of Pennsylvania in 1984.
Michael J. Newman is an associate at Hangley Aronchick Segal Pudlin & Schiller. He received a BA from the University of Pennsylvania in 2002 and a JD from Columbia Law School in 2006. He is a member of the Supreme Court of the United States Bar.
Michael Cliff is Vice President at Analysis Group. He has a BS from Virginia Tech and a PhD in finance from the University of North Carolina at Chapel Hill.
David Klein is an Analyst at Analysis Group. He has a BA in Economics and Political Economy from Washington University in St. Louis.
Sharon Weiss is an assistant at Hangley Aronchick Segal Pudlin & Schiller.
Danielle Acker Susanj earned her law degree from the University of Pennsylvania Law School in 2013. She attended Wheaton College and earned a BA in history. She was born in Philadelphia and raised in central Pennsylvania.
Jonathan Conigliari was also a member of the University of Pennsylvania Law School Class of 2013. He received a BA in French and an MA in French studies from New York University. He is originally from Flagstaff, Arizona.
Sarah Gignoux-Wolfsohn is working toward her PhD at Northeastern University. She earned her BA in biology and French, with honors in biology, from Wesleyan University.
Gabrielle J. Niu earned her MA in East Asian languages and cultures from the University of Pennsylvania in 2012. She is a graduate of Bowdoin College, where she majored in Asian studies and minored in chemistry.
Ben Jackal is a member of Temple University School of Law Class of 2014. He attended Cornell University and earned a BA in history. He was born and raised in Philadelphia.
Colleen Daniels is currently a Crafts major attending the University of the Arts in Philadelphia with a concentration in woodworking. She will graduate in 2015 with a minor in creative writing as well. Colleen was born and raised in the Philadelphia suburbs.
Ellen Boyer graduated from Temple University in 2013 with a Bachelors in English and Political Science. She grew up in Hatboro, Pennsylvania.
David Huppert is a George Washington University student who will graduate in 2015 with a BS in Economics and a minor in English. David was born in Philadelphia and raised in the Philadelphia suburbs.