United States Court of Appeals
For the First Circuit
No. 14-1952
PEDRO LOPEZ, individually and on behalf of a class of
individuals similarly situated; ABEL CANO, individually and on
behalf of a class of individuals similarly situated; KEVIN
SLEDGE, individually and on behalf of a class of individuals
similarly situated; CHARLES DE JESÚS, individually and on behalf
of a class of individuals similarly situated; RICHARD BROOKS,
individually and on behalf of a class of individuals similarly
situated; MASSACHUSETTS HISPANIC LAW ENFORCEMENT ASSOCIATION,
individually and on behalf of a class of individuals similarly
situated; ROBERT ALVAREZ, individually and on behalf of a class
of individuals similarly situated; SPENCER TATUM, individually
and on behalf of a class of individuals similarly situated;
SHUMEAND BENFOLD, individually and on behalf of a class of
individuals similarly situated; ANGELA WILLIAMS-MITCHELL,
individually and on behalf of a class of individuals similarly
situated; GWENDOLYN BROWN, individually and on behalf of a class
of individuals similarly situated; LYNETTE PRAILEAU,
individually and on behalf of a class of individuals similarly
situated; TYRONE SMITH, individually and on behalf of a class of
individuals similarly situated; EDDY CHRISPIN, individually and
on behalf of a class of individuals similarly situated; DAVID E.
MELVIN, individually and on behalf of a class of individuals
similarly situated; STEVEN MORGAN, individually and on behalf of
a class of individuals similarly situated; WILLIAM E. IRAOLO,
individually and on behalf of a class of individuals similarly
situated; JOSÉ LOZANO, individually and on behalf of a class of
individuals similarly situated; COURTNEY A. POWELL, individually
and on behalf of a class of individuals similarly situated;
JAMES L. BROWN, individually and on behalf of a class of
individuals similarly situated; GEORGE CARDOZA, individually and
on behalf of a class of individuals similarly situated; LARRY
ELLISON, individually and on behalf of a class of individuals
similarly situated; DAVID SINGLETARY, individually and on behalf
of a class of individuals similarly situated; CHARISSE BRITTLE
POWELL, individually and on behalf of a class of individuals
similarly situated; CATHENIA D. COOPER-PATERSON, individually
and on behalf of a class of individuals similarly situated;
MOLWYN SHAW, individually and on behalf of a class of
individuals similarly situated; LAMONT ANDERSON, individually
and on behalf of a class of individuals similarly situated;
GLORIA KINKEAD, individually and on behalf of a class of
individuals similarly situated; KENNETH GAINES, individually and
on behalf of a class of individuals similarly situated; MURPHY
GREGORY, individually and on behalf of a class of individuals
similarly situated; JULIAN TURNER, individually and on behalf of
a class of individuals similarly situated; NEVA GRICE,
individually and on behalf of a class of individuals similarly
situated; DELORES E. FACEY, individually and on behalf of a
class of individuals similarly situated; LISA VENUS,
individually and on behalf of a class of individuals similarly
situated; RODNEY O. BEST, individually and on behalf of a class
of individuals similarly situated; KAREN VANDYKE, individually
and on behalf of a class of individuals similarly situated;
ROBERT C. YOUNG, individually and on behalf of a class of
individuals similarly situated; ROYLINE LAMB, individually and
on behalf of a class of individuals similarly situated; LYNN
DAVIS, individually and on behalf of a class of individuals
similarly situated; JAMES A. JACKSON, individually and on behalf
of a class of individuals similarly situated; JUAN ROSARIO,
individually and on behalf of a class of individuals similarly
situated; LOUIS ROSARIO, JR., individually and on behalf of a
class of individuals similarly situated; OBED ALMEYDA,
individually and on behalf of a class of individuals similarly
situated; DEVON WILLIAMS, individually and on behalf of a class
of individuals similarly situated; JULIO M. TOLEDO, individually
and on behalf of a class of individuals similarly situated,
Plaintiffs, Appellants,
MARISOL NOBREGA, individually and on behalf of a class of
individuals similarly situated,
Plaintiff,
v.
CITY OF LAWRENCE, MASSACHUSETTS; CITY OF METHUEN, MASSACHUSETTS;
JOHN MICHAEL SULLIVAN, in his capacity as Mayor of the City of
Lawrence, Massachusetts; WILLIAM MANZI, III, in his capacity as
Mayor of the City of Methuen, Massachusetts; CITY OF LOWELL,
MASSACHUSETTS; CITY OF WORCESTER, MASSACHUSETTS; APPOINTING
AUTHORITY FOR THE CITY OF LOWELL, MASSACHUSETTS; MICHAEL
O’BRIEN, in his capacity as City Manager of the City of
Worcester, Massachusetts; CITY OF BOSTON, MASSACHUSETTS; CITY OF
SPRINGFIELD, MASSACHUSETTS; DOMENIC J. SARNO, JR., in his
capacity as Mayor of the City of Springfield, Massachusetts;
MASSACHUSETTS BAY TRANSPORTATION AUTHORITY; DANIEL GRABAUSKAS,
in his capacity as General Manager, Massachusetts Bay
Transportation Authority; BOARD OF TRUSTEES OF THE MASSACHUSETTS
BAY TRANSPORTATION AUTHORITY,
Defendants, Appellees,
WILLIAM F. MARTIN, in his capacity as Mayor of the City of
Lowell, Massachusetts; KONSTANTINA B. LUKES, in her capacity as
Mayor of the City of Worcester, Massachusetts; COMMONWEALTH OF
MASSACHUSETTS; PAUL DIETL, in his capacity as Personnel
Administrator for the Commonwealth of Massachusetts,
Defendants.
APPEAL FROM THE UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF MASSACHUSETTS
[Hon. George A. O'Toole, Jr., U.S. District Judge]
Before
Torruella, Lynch, and Kayatta,
Circuit Judges.
Harold L. Lichten and Stephen S. Churchill, with whom Benjamin
Weber, Lichten & Liss-Riordan, P.C., and Fair Work, P.C., were on
brief, for appellants.
Bonnie I. Robin-Vergeer, Attorney, Department of Justice,
Civil Rights Division, Appellate Section, with whom Sharon M.
McGowan, Attorney, Civil Rights Division, Vanita Gupta, Acting
Assistant Attorney General, P. David López, General Counsel, and
Carolyn L. Wheeler, Acting Associate General Counsel, Appellate
Services, Equal Employment Opportunity Commission, were on brief
for amicus the United States of America.
Kay H. Hodge, with whom John M. Simon, Geoffrey R. Bok,
Stoneman, Chandler & Miller LLP, Susan M. Weise, Attorney, City of
Boston Law Department, and Lisa Skehill Maki, Attorney, City of
Boston Law Department, were on brief, for appellee City of Boston,
Massachusetts.
James F. Kavanaugh, Jr., with whom Christopher K. Sweeney,
and Conn Kavanaugh Rosenthal Peisch & Ford, LLP, were on brief,
for appellees Massachusetts Bay Transportation Authority, Daniel
Grabauskas, and the Board of Trustees of the Massachusetts Bay
Transportation Authority.
Rachel M. Brown, Assistant City Solicitor, City of Lowell Law
Department, with whom Christine Patricia O'Connor, City Solicitor,
City of Lowell Law Department, was on brief for appellees City of
Lowell, Massachusetts, and Appointing Authority for the City of
Lowell, Massachusetts.
Tim D. Norris, with whom Joshua R. Coleman, and Collins,
Loughran & Peloquin, P.C., were on brief, for appellees City of
Worcester, Massachusetts, Michael O'Brien, City Manager of
Worcester, and Konstantina B. Lukes, Mayor of the City of
Worcester.
Anthony I. Wilson, Associate City Solicitor, City of
Springfield Law Department, with whom Edward M. Pikula, City
Solicitor, and John T. Liebel, Associate City Solicitor, were on
brief, for appellees City of Springfield, Massachusetts, and Mayor
Domenic J. Sarno, Jr.
Raquel D. Ruano, Attorney, Office of the City Attorney, City
of Lawrence, Massachusetts, and Charles D. Boddy, Jr., Attorney,
Office of the City Attorney, City of Lawrence, Massachusetts, on
brief for appellees City of Lawrence, Massachusetts, and Mayor
John Michael Sullivan.
Kerry Regan Jenness, Attorney, Office of the City Solicitor,
City of Methuen, on brief for appellees City of Methuen,
Massachusetts, and Mayor William M. Manzi, III.
Michael L. Foreman, Civil Rights Appellate Clinic, Dickinson
School of Law, Pennsylvania State University, on amicus brief of
National Urban League and the National Association for the
Advancement of Colored People.
Gary Klein, Kevin Costello, Corinne Reed, Klein Kavanagh
Costello, LLP, Mark S. Brodin, Professor, Boston College Law
School, and Ray McClain, Director, Employment Discrimination
Project, Lawyers' Committee for Civil Rights Under Law, on amicus
brief of Massachusetts Association of Minority Law Enforcement
Officers, New England Area Conference of the National Association
for the Advancement of Colored People, Urban League of Eastern
Massachusetts, and Professor Mark S. Brodin.
Christopher L. Brown, Christopher J. Petrini, and Petrini &
Associates, P.C., on amicus brief of International Municipal
Lawyers Association, Massachusetts Municipal Lawyers Association,
National Public Employer Labor Relations Association,
Massachusetts Chiefs of Police Association, Inc., and Fire Chiefs
Association of Massachusetts, Inc.
May 18, 2016
KAYATTA, Circuit Judge. In selecting police officers
for promotion to the position of sergeant in 2005 and 2008, the
City of Boston and several other Massachusetts communities and
state employers adapted a test developed by a Massachusetts state
agency ("HRD")1 charged under state law with creating a selection
tool that "fairly test[s] the knowledge, skills and abilities which
can be practically and reliably measured and which are actually
required" by the job in question. Mass. Gen. Laws ch. 31, § 16.
There is no claim in this case that defendants intentionally
selected the test in order to disadvantage any group of applicants.
To the contrary, the evidence is that the test was the product of
a long-running effort to eliminate the use of race or other
improper considerations in public employment decisions.
The percentage of Black and Hispanic applicants selected
for promotion using the results of this test nevertheless fell
significantly below the percentage of Caucasian applicants
selected. Some of those Black and Hispanic applicants who were
not selected for promotion sued, claiming that the use of the test
resulted in an unjustified "disparate impact" in violation of
Title VII notwithstanding the absence of any intent to discriminate
on the basis of race. 42 U.S.C. § 2000e-2(k)(1)(A)(i). After an
1 This agency is the Human Resources Division of the
Massachusetts Executive Office of Administration and Finance.
Lopez v. City of Lawrence, No. 07-11693-GAO, 2014 U.S. Dist. LEXIS
124139, at *7 n.1 (D. Mass. Sept. 5, 2014).
- 6 -
eighteen-day bench trial, the district court determined, among
other things, that the use of the test did have a disparate impact
on promotions in the City of Boston, but that the test was a valid
selection tool that helped the City select sergeants based on
merit. Lopez v. City of Lawrence, No. 07-11693-GAO, 2014 U.S.
Dist. LEXIS 124139, at *37, *60–62 (D. Mass. Sept. 5, 2014). The
court further found that the plaintiffs failed to prove that there
was an alternative selection tool that was available, that was as
(or more) valid than the test used, and that would have resulted
in the promotion of a higher percentage of Black and Hispanic
officers. Id. at *60–79. Finding that the district court applied
the correct rules of law and that its factual findings were not
clearly erroneous, we affirm.
I. Background
The plaintiffs in this suit (the "Officers") sought
promotion in the police departments operated by the Massachusetts
municipalities or state agencies sued in this case. Id. at *7–8.
All parties agree that affirmance of the judgment in favor of
Boston would result in affirmance of the judgment in favor of the
other defendants as well, so we focus our discussion for
simplicity's sake on the evidence concerning Boston. Because this
is an appeal of fact-finding and application of law to fact
following a trial on the merits, we describe the facts in a manner
that assumes conflicting evidence was resolved in favor of the
- 7 -
prevailing party unless there is particular reason to do otherwise.
Wainwright Bank & Tr. Co. v. Boulos, 89 F.3d 17, 19 (1st Cir. 1996)
("We summarize the facts in the light most favorable to the
verdict-winner [ ], consistent with record support.").
A. Development of the Exams Over Time
In 1971, Congress noted that the United States
Commission on Civil Rights ("USCCR") found racial discrimination
in municipal employment "more pervasive than in the private
sector." H.R. Rep. No. 92-238, at 17 (1971). According to the
USCCR, nepotism and political patronage helped perpetuate pre-
existing racial hierarchies. U.S. Comm'n on Civil Rights, For All
the People, By All the People: A Report on Equal Opportunity in
State and Local Government Employment, 63–65, 119 (1969),
reprinted in 118 Cong. Rec. 1817 (1972). Police and fire
departments served as particularly extreme examples of this
practice. See, e.g., Wesley MacNeil Oliver, The Neglected History
of Criminal Procedure, 1850–1940, 62 Rutgers L. Rev. 447, 473
(2010) ("Officers who delivered payments to their superiors were
practically assured of retention and even promotion, regardless of
their transgressions."); Nirej S. Sekhon, Redistributive Policing,
101 J. Crim. L. & Criminology 1171, 1191 (2011) ("Police
departments were prime sources of patronage jobs.").
Boston's police department was no exception: As far
back as the nineteenth century, a subjective hiring scheme that
- 8 -
hinged on an applicant's perceived political influence and the
hiring officer's subjective familiarity with the candidate (or the
candidate's last name) was seen as the primary culprit behind a
corrupt, inept, and racially exclusive police force. See, e.g.,
George H. McCaffrey, Boston Police Department, 2 J. Am. Inst. Crim.
L. & Criminology 672, 672 (1912) ("This system worked very
unsatisfactorily, however, because places on the police force were
invariably bestowed as a reward for partisan activity.").
At both the state and local levels, Massachusetts
officials eventually gravitated toward competitive exams as a tool
to accomplish an important public policy of moving away from
nepotism, patronage, and racism in the hiring and promoting of
police. Boston Chapter, N.A.A.C.P., Inc. v. Beecher, 504 F.2d
1017, 1022 (1st Cir. 1974) ("[C]ivil service tests were instituted
to replace the evils of a subjective hiring process . . . ."); see
generally League of Women Voters of Mass., The Merit System in
Massachusetts: A Study of Public Personnel Administration in the
Commonwealth 3–5 (1961). At the statewide level, this movement
resulted in legislation and regulations aimed at ensuring that
employees in civil service positions are "recruit[ed], select[ed]
and advanc[ed] . . . on the basis of their relative ability,
knowledge and skills" and "without regard to political
affiliation, race, color, age, national origin, sex, marital
status, handicap, or religion." Mass. Gen. Laws ch. 31, § 1.
- 9 -
B. The 2005 and 2008 Exams
Born of these purposes and shaped by decades of Title
VII litigation,2 the examinations at issue in this case allowed no
room for the subjective grading of applications. The total score
of a test-taker who sat for the promotional examination in 2005 or
2008 was determined by two components: an 80-question written
examination scored on a 100-point scale and an "education and
experience" ("E&E") rating, also scored on a 100-point scale. The
written examination counted for 80% of an applicant's final score
and the E&E rating comprised the remaining 20%. Applicants needed
an overall score of seventy to be considered for promotion. On
top of the raw score from these two components, Massachusetts law
affords special consideration for certain military veterans, id.
§ 26, and individuals who have long records of service with the
state, id. § 59.
The subject matter tested on the 2005 and 2008
examinations can be traced back to a 1991 "validation study" or
"job analysis report" performed by the state agency responsible
for compiling the exam.3 See 29 C.F.R. § 1607.14 (technical
2 The district court offered a detailed summary of this
litigious history. See Lopez, 2014 U.S. Dist. LEXIS 124139, at
*24–27.
3 The Officers argue that Boston misrepresented its reliance
on the 1991 report and that the City, in fact, used only a less-
thorough report conducted in 2000. The Officers' evidence for
this consists of a comparison, in a footnote in their appellate
- 10 -
requirements for a content validity study under the Uniform
Guidelines on Employee Selection Procedures); see also Watson v.
Fort Worth Bank & Tr., 487 U.S. 977, 991 (1988) (opinion of
O'Connor, J.) ("Standardized tests and criteria . . . can often be
justified through formal 'validation studies,' which seek to
determine whether discrete selection criteria predict actual on-
the-job performance.").
That 1991 report was prepared by the Massachusetts
Department of Personnel Administration ("DPA"), the predecessor to
HRD. In preparing the report, DPA surveyed police officers in
thirty-four jurisdictions nationwide, issuing a questionnaire that
sought to ascertain the kinds of "knowledge[], skills, abilities
and personnel characteristics" that police officers across the
country deemed critical to the performance of a police sergeant's
responsibilities. The report's authors distilled the initial
results from this survey and their own knowledge regarding
professional best practices into a list of critical police
supervisory traits. They then distributed this list in a second
survey to high-ranking police officers in Massachusetts, who were
asked to rank these traits according to how important they felt
brief, between three tested skill areas out of fifteen total areas
on the 2008 outline of exam questions and similar language from
the 2000 job analysis. We decline to find that this perfunctory,
post-judgment sampling demonstrates that the district court
committed clear error.
- 11 -
each was to a Massachusetts police sergeant's performance of her
duties. DPA further refined the ranking of key skills and traits
through focused small-group discussions with police sergeants and
conducted a "testability analysis" of which skills could likely be
measured through the written examination or the E&E component. In
2000, HRD engaged outside consultants to refresh the findings of
the 1991 examination through a process similar to, though less
thorough than, DPA's approach in 1991.
The written question and answer component of the
examination consisted of multiple choice questions that covered
many topic areas, including the rules governing custodial
interrogation, juvenile issues, community policing, and firearm
issues, to name a few.4 The text of individual questions was often
closely drawn from the text of materials identified in a reading
list provided by the Boston Police Department ("BPD") to test-
takers in advance of the exams.
For example, one question on the 2008 promotional exam
asked applicants to accurately complete the following statement:
According to [a criminal investigations
textbook on the reading list], a warrantless
search and seizure is acceptable:
A. after stopping a vehicle for a traffic
violation and writing a citation.
4 Boston supplemented the HRD-produced examination with
additional jurisdiction-specific questions that sought to probe a
candidate's knowledge of Boston-specific rules, orders, and
regulations.
- 12 -
B. after obtaining the consent of the
person, regardless of whether obtained
voluntarily or nonvoluntarily.
C. when possible loss or destruction of
evidence exists.
D. when a quick search of the trunk of a
motor vehicle is desired.
In addition to completing the question and answer
component of the examination, applicants listed on the E&E rating
sheet their relevant work experience, their degrees and
certifications in certain areas, their teaching experience, and
any licenses they held.5 Points were assigned based on the listed
education and experience. For example, applicants could receive
up to fifteen points in recognition of their educational
attainment, with an associate's degree providing up to three points
and a doctorate providing up to twelve.
After collecting and scoring the exams, HRD provided the
municipalities with a list of passing test-takers eligible for
promotion, ranked in order of their test scores. Mass. Gen. Laws
ch. 31, § 25. Each of the municipal defendants in this case
selected candidates in strict rank order based on the list they
received from HRD.6
5 The Officers point out that the same E&E sheet was used to
identify candidates for promotion among Massachusetts firefighters
in 2010.
6 State law permitted a certain amount of flexibility for
- 13 -
Because many officers achieved at least the minimum
passing score of seventy and there were relatively few openings
for promotion to sergeant, all of those who were promoted scored
well above the minimum in both 2005 and 2008. In 2005, 9 of the
224 Black and Hispanic candidates who took the exam were promoted,
whereas 57 of the 401 other candidates were promoted. In 2008, 1
of the 213 Black and Hispanic test-takers was promoted, whereas 25
of the 291 other candidates were promoted. The average scores for
those who the statisticians called "minority test takers" fell
below the average scores for the "non-minority test takers" by 6.4
points in 2005 and 6.6 points in 2008.
II. Analysis
We recently described in another suit against Boston the
elements of a disparate impact claim. Jones v. City of Boston,
752 F.3d 38, 46, 54 (1st Cir. 2014). In a nutshell, litigation of
such a claim in a case challenging hiring or promotion decisions
municipalities to "bypass" a candidate who had the next-highest
score on the ranked list. Mass. Gen. Laws ch. 31, § 27. The
municipality could be held accountable to the bypassed employee
and, if challenged, would have to articulate a defensible reason
for skipping him or her over. See City of Cambridge v. Civil Serv.
Comm’n, 682 N.E.2d 923, 925 (Mass. App. Ct. 1997). No
justification "inconsistent with basic merit principles, can[] be
used to justify a bypass," including a candidate's race. Mass.
Ass'n of Minority Law Enf't Officers v. Abban, 748 N.E.2d 455, 462
(Mass. 2001). The Massachusetts Bay Transit Authority ("MBTA"),
a state agency and a defendant, behaved slightly differently during
the relevant years by treating all the candidates on HRD's list as
having scored equally and narrowing down their pool of candidates
by using oral interviews.
- 14 -
focuses on three questions: Do the plaintiffs show by competent
evidence that the employer is utilizing an employment practice
that causes a disparate impact on the basis of race; If so, does
the employer show that the challenged employment practice creating
this disparate result is nevertheless job-related for the position
in question and consistent with business necessity; If so, do the
plaintiffs show that the employer has refused to adopt an
alternative practice that equally or better serves the employer's
legitimate business needs, yet has a lesser disparate impact? Id.
To prevail, plaintiffs require a "yes" answer to the first
question, and either a "no" to the second question or a "yes" to
the third question. See id.
In this case, all parties agree that, using competent
statistical analysis, the Officers have proven that Boston's use
of the challenged exam in 2005 and 2008 did indeed have a marked
disparate impact because the selection rates of Black and Hispanic
officers for promotion to sergeant were so much lower than the
selection rates of the other officers that we can fairly exclude
random chance as the explanation for the difference.7
7 The other defendants did not concede that the statistical
analyses applied to the outcomes among their smaller groups of
applicants established a disparate impact, and the district court
agreed with the defendants. Our disposition of this appeal does
not require us to assess the correctness of that ruling.
- 15 -
A. Validity
The focus of the trial thus turned to the second
question: Did Boston use a "practice [that was] 'job related for
the position in question and consistent with business necessity.'"
Ricci v. DeStefano, 557 U.S. 557, 578 (2009) (quoting 42 U.S.C.
§ 2000e–2(k)(1)(A)(i)). The parties agree that, in the context of
hiring or promotion decisions, this inquiry trains on whether the
selection practice--here, the use of the exam--is "valid." In
simple terms, a selection practice is valid if it materially
enhances the employer's ability to pick individuals who are more
likely to perform better than those not picked.
In this case, Boston sought to carry its burden of
proving the validity of its exams by demonstrating what the Equal
Employment Opportunity Commission ("EEOC") refers to as "content
validity" under the Uniform Guidelines on Employee Selection
Procedures ("Guidelines"). See 29 C.F.R. § 1607.16(D). The
parties agree generally that establishing content validity in this
context requires a "showing that the content of the selection
procedure is representative of important aspects of performance on
the job for which the candidates are to be evaluated." Id.
§ 1607.5(B). This means that the "behavior(s) demonstrated in the
selection procedure are a representative sample of the behavior(s)
of the job in question or that the selection procedure provides a
representative sample of the work product of the job." Id.
- 16 -
§ 1607.14(C)(4). Work behavior(s) selected for measurement should
either be "critical work behavior(s)" or "important work
behavior(s) constituting most of the job," or both. Id.
§ 1607.14(C)(2).
Much of the evidence at trial and many of the arguments
in the briefs on appeal focus on the Guidelines' technical testing
standards. The Officers' briefs treat the Guidelines as if they
were inflexible and binding legal standards that must be rigorously
applied in ascertaining whether an employment selection device
significantly advances the employer's business needs. For two
reasons, this is not so.
First, "[b]ecause 'Congress, in enacting Title VII, did
not confer upon the EEOC authority to promulgate rules and
regulations,' the agency's guidelines receive weight only to the
extent of their 'power to persuade.'" Jones, 752 F.3d at 50 n.14
(quoting E.E.O.C. v. Arabian Am. Oil Co., 499 U.S. 244, 257
(1991)). In Jones itself, we rejected the Guidelines' view that
plaintiffs need carry the burden of proving "practical
significance" in order to establish a prima facie case of disparate
impact. Id. at 50–53. And in Ricci, the Supreme Court's most
recent disparate impact decision, the Court found New Haven's
firefighter promotional exam job-related without mentioning the
Guidelines' extensive technical criteria for assessing job-
relatedness. See Ricci, 557 U.S. at 87–89.
- 17 -
Second, even on their own terms, the Guidelines poorly
serve the controlling role assigned to them by the Officers in
challenging the district court's findings. The Guidelines quite
understandably provide no quantitative measure for drawing the
line between "representative," 29 C.F.R. § 1607.5(B), and
nonrepresentative samples of job performance and behaviors.
Rather, the Guidelines point to the qualitative understandings of
these concepts generally accepted by professionals who evaluate
"standardized tests and other selection procedures, such as those
described in the Standards for Educational and Psychological Tests
prepared by a joint committee of the American Psychological
Association." Id. § 1607.5(C).
All that being said, Boston did not shy away from seeking
to show that its process for selecting new police sergeants in
2005 and 2008 satisfied the technical requirements of the
Guidelines. To make such a showing, the City presented the
testimony of Dr. James Outtz. Outtz is an industrial
organizational psychologist with twenty years of experience
testing and measuring employee selection systems. He has served
as a consultant to numerous American municipalities and federal
agencies and has assisted in the development of employment
selection devices used by many public employers. Outtz has
published approximately twenty academic publications in the field
of industrial organizational psychology. He has worked for both
- 18 -
plaintiffs and defendants in challenges to the validity of exams.
In Ricci, for example, Outtz co-authored an amicus brief brought
on behalf of industrial psychologists arguing that the New Haven
Fire Department promotional examinations for captain and
lieutenant were flawed and invalid. See Br. of Industrial-
Organizational Psychologists as Amici Curiae at 3, Ricci, 557 U.S.
557 (Nos. 07–1428, 08–328), 2009 WL 796281, at *3.
Outtz reviewed the development, application, substance,
and results of the exams at issue in this case. He opined that
the exams were based on job analyses that validly identified the
critical skills used by actual police sergeants and that the tests
covered a "representative sample" of the content of the job. Id.
§ 1607.14(C)(4). In support of this conclusion, Outtz testified
that the two job validity reports relied on in composing the 2005
and 2008 exams were not too stale to serve as useful starting
points for the test-makers, nor were the reports otherwise infirm
from a technical standpoint. While the reports
--particularly the 1991 report--were somewhat dated, Outtz
testified that the relative stability of a police sergeant's
responsibilities over time, combined with the presence of the 2000
study, cured any defect introduced by the passage of time.8
8
The district court was entitled to rely on this conclusion,
despite the Officers' various quibbles with the methodologies used
to compile the 1991 and 2000 reports.
- 19 -
Outtz went on to opine that the written question and
answer portion of the exam, standing alone, nevertheless did not
pass muster under the Guidelines because it fell short of testing
a "representative sample" of the key qualities and attributes that
were identified by the two validation reports. Id. In Outtz's
opinion, however, the addition of the E&E component effectively
pushed the selection device as a whole across the finish line to
show validity. It did this, according to Outtz, because the level
and extent of work and educational experience and accomplishments
listed by each applicant served as a useful, if imperfect, proxy
for the kinds of qualities that were deemed to be important to a
sergeant's daily responsibilities, yet were insufficiently tested
by the examination's question and answer component. Outtz
recognized that the gain in validity from the E&E component was,
on its own, only marginal or "incremental." As the Officers
stress, many of the attributes for which the E&E assigned points
(e.g., previous service as a police officer) were shared by all or
most applicants. Thus, while the E&E score range for the 2005
exam was 0–100, the actual score distribution approximated 40–94.
And when weighted to provide only 20% of the combined final score,
it accounted for a range of only about 5% to 7% of a candidate's
total score.9 Nevertheless, we cannot see how a rational
9 The Officers place this variation slightly lower, at 1% to
- 20 -
factfinder could ignore the impact of the E&E, small or not, in
evaluating the exam overall.
Outtz concluded that "at the end of the day" the combined
"package" of the written examination and the E&E as administered
tested a "representative sample" of the key supervisory skills
identified by the 1991 and 2000 reports and was "minimally valid"
or "acceptable" under the Guidelines. Id. He testified that the
representativeness of the skills tested by the two components and
the linkage of these skills to the validation reports were in line
with what was contemplated by the Guidelines' technical standards
for constructing a content-valid selection device. See id.
§§ 1607.5(B); 1607.14(C)(4).
This is not to say that Outtz's testimony trumpeted a
wholehearted endorsement of the scheme used by Boston to identify
candidates for promotion. He agreed with the Officers that the
validity of the Boston examination could have been improved,
perhaps by incorporating a "well-developed assessment center" to
evaluate an officer's interpersonal skills through observed social
interaction, or some kind of device for measuring an applicant's
oral communication skills. Outtz was clear that his opinion solely
4%, relying on testimony suggesting that no candidate could reach
the ceiling of the potential boost offered by the E&E. Unguided
by fact-finding on this narrow question, we note only the absence
of any evidence that Outtz's opinion turned on a plainly erroneous
calculation of the precise percentage.
- 21 -
concerned the selection device's compliance with his profession's
minimum standards as translated into the EEOC's Guidelines.
The Officers challenged Outtz's conclusions on cross-
examination, arguing that his testimony fell short of the mark in
several respects that we will discuss, and presented the contrary
opinions of their own expert, Dr. James Wiesen. Altogether, the
trial testimony of these competing experts consumed the better
part of nine days of the eighteen-day trial.
The district court judge who listened to these experts
testify concluded that Outtz was correct: "After consideration of
the evidence as a whole, I find and conclude that Dr. Outtz’s
opinion rests on adequate grounds and is therefore correct: the
exams in question were minimally valid." Lopez, 2014 U.S. Dist.
LEXIS 124139, at *60–61. Supporting this conclusion, the court
found that the examinations tested a representative sample of
skills that were identified by the 1991 and 2000 reports, which
were themselves valid under the Guidelines. Id. at *61. Finding
that Boston employed valid examinations that reliably achieved the
City’s stated business need, the court ruled in Boston’s favor.
Id. at *78.
On appeal, the Officers now ask us to set aside the
district court's finding that the 2005 and 2008 exams were valid.
In considering such a request, we ask whether the district court
applied the correct legal standards and whether the record
- 22 -
contained sufficient support for its findings. See, e.g., Beecher,
504 F.2d at 1022 (affirming a finding of invalidity as "supported
by the record"). Since our decision in Beecher, all circuit courts
that have addressed the question have reviewed a district court's
determination that a selection method was or was not valid for
clear error. See M.O.C.H.A. Soc'y, Inc. v. City of Buffalo, 689
F.3d 263, 275 (2d Cir. 2012); Ass'n of Mex.-Am. Educators v.
California, 231 F.3d 572, 584–85 (9th Cir. 2000) (en banc) ("The
question whether a test has been validated properly is primarily
a factual question, which depends on underlying factual
determinations regarding the content and reliability of the
validation studies that a defendant utilized."); Melendez v. Ill.
Bell Tel. Co., 79 F.3d 661, 669 (7th Cir. 1996); Hamer v. City of
Atlanta, 872 F.2d 1521, 1526 (11th Cir. 1989); Bernard v. Gulf Oil
Corp., 890 F.2d 735, 739 (5th Cir. 1989).
With this standard in mind, we consider the Officers'
critique of the district court's reliance on Outtz's opinion in
finding the examinations valid. Repeatedly, the Officers suggest
that Outtz's own characterization of the exams as "minimally valid"
should render his opinion legally insufficient to carry the City's
burden. Implicitly, the Officers ask us to read "minimally valid"
as meaning, in effect, "not valid enough." Read in context,
however, Outtz was plainly testifying that he found the exams to
be valid, albeit not by much. Indeed, elsewhere in his testimony
- 23 -
he made clear that the exams were "valid" and, in his view,
complied with the technical requirements of the Guidelines.
Moving more aptly from debating adverbs to discussing
the law, the Officers (with the support of the United States as
amicus curiae) argue that the district court misconstrued the law
in finding Outtz's testimony sufficient. Specifically, they say
that the district court did not reach its finding of content
validity in accord with the Guidelines' statement that evidence of
an exam's content validity should "consist of data showing that
the content of the selection procedure is representative of
important aspects of performance on the job for which the
candidates are to be evaluated." 29 C.F.R. § 1607.5(B). Instead,
argue the United States and the Officers, the district court simply
counted up the knowledge, skills and abilities ("KSAs") called for
by the job without qualitatively considering their importance.
It is true that the district court observed that "more
than half of the KSAs identified as pertinent to the job of
sergeant were tested," and that "this was sufficient to meet the
'representative sample' requirement of the Uniform Guidelines."
Lopez, 2014 U.S. Dist. LEXIS 124139 at *54 (quoting 29 C.F.R.
§ 1607.14(C)(4)). The district court made this statement, though,
only after first expressly citing the Guidelines standard, id. at
*15–17, and after undertaking an examination of the tested KSAs
"to ensure that there is a link between the selection procedure
- 24 -
and the critical KSAs necessary for successful performance of the
job," id. at *16. The court then made clear that its examination
of the manner in which the exams tested KSAs trained on "the
knowledge, skills and abilities which can be practically and
reliably measured and which are actually required to perform the
primary or dominant duties of the position for which the
examination is held." Id. at *50–51 (quoting Mass. Gen. Laws ch.
31, § 16). The district court also cited to testimony establishing
that knowledge of the constitutional and regulatory law applicable
to police work is "critical to a police sergeant's ability to
effectively perform as a supervisor" and to evidence that a written
job knowledge test is "[a]n effective way" of testing whether a
candidate possesses such critical knowledge. Id. at *51–52.
Similarly, the district court found that the 1991 job analysis
upon which the exams were based identified "the frequent and
critical tasks and duties" and the "important [KSAs] required at
the time of appointment."10 Id. at *52. In short, in referring
10 Joined by the United States as amicus curiae, the Officers
further dispute the "linkage" between these validation reports--
both the 1991 and 2000 reports--and the examinations themselves.
Their chief challenge on this front revolves around a "testability
analysis" document prepared in connection with the 1991 report
that evaluates which key skills could, in theory, be tested on a
future examination but does not directly link the skills identified
to actual examination or E&E content. The defect with the
Officers' reliance on this document is that it asks us to pretend
that it was the only relevant evidence the district court could
rely on in drawing a connection between the validation reports and
- 25 -
to the KSAs identified as pertinent to the job of sergeant, the
district court was plainly referring to the "critical" and
"important" KSAs that it found to have been identified in the job
analysis upon which the exams were predicated.
The district court's qualitative focus on the importance
of the factors that the exam tested was further highlighted by the
court's agreement with Outtz that the written job knowledge portion
of the test was not alone valid "because it could not measure some
skills and abilities (as distinguished from knowledge) essential
to the position." Id. at *60. After then agreeing with Outtz
that the E&E component of the exams adequately, albeit minimally,
filled in this gap, the district court expressly found that the
exams "were based on job analyses that considered the important
tasks necessary to the successful performance of the job." Id. at
*61. The district court's opinion as a whole thus makes clear
that the court trained its focus on critical and important
knowledge, skills, and abilities called for by the job, and it did
not clearly err by finding that a test that measured a large
the examinations as administered. This was hardly the case. The
district court weighed the testimony of Dr. Wiesen and Dr. Outtz,
both of whom had analyzed the examinations as well as the reports,
reviewed the testability analysis, applied their scientific
expertise, and formed their own (differing) judgments as to whether
the examinations tested the skills identified by the reports. In
crediting Outtz's testimony, the district court did not clearly
err.
- 26 -
percentage of such critical and important KSAs was a test that was
sufficiently "representative of important aspects of performance
on the job." 29 C.F.R. § 1107.5(B).11 Our conclusion to this
effect finds further support in the absence of any quantitative
measure of "representativeness" provided by the law. Rather, the
relevant aim of the law, when a disparate impact occurs, is to
ensure that the practice causing that impact serves an important
need of the employer, in which case it can be used unless there is
another way to meet that need with lesser disparate impact. We
cannot see how it is an error of law to find that an exam that
helps determine whether an applicant possesses a large number of
critical and necessary attributes for a job serves an important
need of the employer.
The Officers and the United States also contend that our
1974 opinion in Beecher, 504 F.2d 1017, mandates our reversal of
this conclusion. Their reliance on Beecher fits this case
awkwardly because of the burdens we have already detailed. In
Beecher, the central question was whether the record supported the
district court's finding of fact that a hiring exam given to would-
11In the district court's observation that "more than half
of the KSAs identified as pertinent to the job were tested," Lopez,
2014 U.S. Dist. LEXIS 124139, at *54, the Officers see clear error,
pointing out that the 1991 testability analysis only identified 70
out of a total 156 critical KSAs (i.e., not quite half) that could
be tested on the exam. We decline the Officers' invitation to
find this difference to be so material as to constitute clear
error.
- 27 -
be firefighters was not valid. See id. at 1022–23. To affirm, we
needed only to find that the record did not compel a contrary
finding. Id. at 1022. Here, by contrast, the Officers ask us to
find that this record compels a finding contrary to that reached
by the district court.
The Officers and the United States nevertheless seem to
find much significance in one analogy we drew in Beecher. In
assessing an exam for the position of firefighter, we compared
knowledge of firefighting terminology to knowledge of baseball
vocabulary possessed by a potential recruit for the Boston Red Sox
"who could not bat, pitch or catch." Id. at 1023. Here, in
reviewing an exam for the supervisory position of sergeant, the
more apt baseball analogy would be the hiring of a coach, who must
certainly have an extensive knowledge of the rules that must be
followed by those being managed. At trial, former Boston Police
Commissioner Edward Davis testified that a "sergeant really has to
have a strong basis of knowledge of all the rules and regulations
and constitutional protections that are afforded the citizens of
the Commonwealth to do the job properly," because when officers in
the field "get confused and don't understand something, the first
thing they do is call the sergeant." This "fundamental
understanding" of "how things work," was a "critical component" of
a sergeant's responsibilities, according to Commissioner Davis.
And, the court supportably found, those skillsets were tested by
- 28 -
the exam.
The Officers' reliance on Beecher is further undermined
by the different approach taken in that case towards validation of
the exam. We stated that for an exam to be valid, the court must
be satisfied that "it demonstrably selects people who will perform
better the required on-the-job behaviors after they have been hired
and trained." Id. at 1021–22. We observed that "[t]he crucial fit
is not between test and job lexicon, but between the test and job
performance." Id. at 1022. This approach resembles what the
Guidelines, adopted four years after Beecher, call "criterion-
related validity." 29 C.F.R. § 1607.5(B) ("Evidence of the
validity of a test or other selection procedure by a criterion-
related validity study should consist of empirical data
demonstrating that the selection procedure is predictive of or
significantly correlated with important elements of job
performance."). Because in this case, as we have discussed, we
assess validity for the most part under the separate "content
validity" framework, Beecher's relevance is further limited.
None of the remaining arguments advanced by the Officers
seriously support any claim that the exams are not materially
better predictors of success than would be achieved by the random
selection of those officers to be promoted to sergeant. The
parties' arguments, instead, focus on how much better the exams
were. Do they test enough skills and knowledge? Do they weigh
- 29 -
the answers in an appropriate, valid way? In finding Outtz
persuasive on these points, the district court as factfinder did
not clearly err.12
B. Rank-Order Selection
When officials at the BPD received the results of the
2005 and 2008 sergeant promotional examinations from HRD, they
selected as many police officers for promotion as there were
vacancies currently available, beginning with the highest-scoring
name at the top of the list and moving down the list, one at a
time, in order of the score each candidate received. The Officers
argue that this method of selection--quite independently from the
written examination itself--led to a disparate impact and the
district court was obligated to conduct a separate analysis of its
validity under Title VII. We review the legal sufficiency of the
district court's holding on this point de novo and its subsidiary
fact-finding for clear error. E.E.O.C. v. Steamship Clerks Union,
Local 1066, 48 F.3d 594, 603 (1st Cir. 1995).
The Officers first argue that the district court failed
altogether to wrestle with the consequences of rank-order
selection. This is clearly not the case. Citing section
12 The Officers did not move to strike any portion of Outtz's
testimony under Daubert v. Merrell Dow Pharmaceuticals, Inc., 509
U.S. 579 (1993). Hence, even if we had thought that any part of
Outtz's opinion was unreliable or unsupported, we would have had
to employ plain error review. See United States v. Diaz, 300 F.3d
66, 74 (1st Cir. 2002).
- 30 -
1607.14(C)(9) of the Guidelines, the district court noted in its
exegesis of the law that "[t]he use of a ranking device requires
a separate demonstration that there is a relationship between
higher scores and better job performance." Lopez, 2014 U.S. Dist.
LEXIS 124139, at *16–17. The court went on to find that Boston's
selection method "reliably predicts a candidate’s suitability for
the job, such that persons who perform better under the test method
are likely to perform better on the job." Id. at *61.
This finding by the district court, to the Officers, is
"not enough." Based on their reading of the Guidelines, something
more is required. The Officers argue that the use of the results
of an examination that is "minimally valid" insofar as it tests
job-related skills may not necessarily be valid if used to select
candidates solely according to their scores on that exam.
Two provisions of the Guidelines discuss an employer's
appropriate use of a rank-ordering selection method. In the
section of the Guidelines establishing "General Principles," the
EEOC has advised the following:
The evidence of both the validity and utility
of a selection procedure should support the
method the user chooses for operational use of
the procedure, if that method of use has a
greater adverse impact than another method of
use. Evidence which may be sufficient to
support the use of a selection procedure on a
pass/fail (screening) basis may be
insufficient to support the use of the same
procedure on a ranking basis under these
guidelines. Thus, if a user decides to use a
- 31 -
selection procedure on a ranking basis, and
that method of use has a greater adverse
impact than use on an appropriate pass/fail
basis (see section 5H below), the user should
have sufficient evidence of validity and
utility to support the use on a ranking basis.
29 C.F.R. § 1607.5(G) (emphasis supplied). The Guidelines also
contain a refinement of this principle specific to the use of
content validity studies in the "Technical Standards" section:
If a user can show, by a job analysis or
otherwise, that a higher score on a content
valid selection procedure is likely to result
in better job performance, the results may be
used to rank persons who score above minimum
levels. Where a selection procedure supported
solely or primarily by content validity is
used to rank job candidates, the selection
procedure should measure those aspects of
performance which differentiate among levels
of job performance.
Id. § 1607.14(C)(9).
These two statements evidence some inconsistency.
Section 1607.5(G) clearly indicates that an employer need have
sufficient evidence of validity to support use of the exam on a
ranking basis "if . . . that method of use has a greater adverse
impact than use on an appropriate pass/fail basis" (emphasis
supplied). Under this guidance, if an exam is valid, one may use
it on a rank-order basis unless the use of rank ordering creates
or adds to a disparate impact. One can read section 1607.14(C)(9),
however, as requiring that, to defend rank ordering, the employer
must first show that "a higher score on a content valid selection
- 32 -
procedure is likely to result in better job performance"; i.e.,
one must validate the use of ranking itself if the exam as a whole
produces a disparate impact. Other provisions of the Guidelines
support this latter reading, albeit without acknowledging the
inconsistency. Compare, e.g., id. § 1607.5(G) ("[I]f a user
decides to use a selection procedure on a ranking basis, and that
method of use has a greater adverse impact than use on an
appropriate pass/fail basis . . . , the user should have sufficient
evidence of validity and utility to support the use on a ranking
basis." (emphasis supplied)), with Adoption of Questions and
Answers to Clarify and Provide a Common Interpretation of the
Uniform Guidelines on Employee Selection Procedures, 44 Fed. Reg.
11,996, 12,005, Question and Answer n. 62 (1979) ("Use of a
selection procedure on a ranking basis may be supported by content
validity if there is evidence from job analysis or other empirical
data that what is measured by the selection procedure is associated
with differences in levels of job performance.").
Several courts have seemed to approach this issue by
requiring more scrutiny of the validation evidence as a whole when
rank ordering is used, particularly when the exams in question
have led to closely bunched scores. See Johnson v. City of
Memphis, 770 F.3d 464, 479–81 (6th Cir. 2014), cert. denied, 136
S. Ct. 81 (2015); Police Officers for Equal Rights v. City of
Columbus, 916 F.2d 1092, 1102–03 (6th Cir. 1990); Guardians Ass'n
- 33 -
of N.Y.C. Police Dep't, Inc. v. Civil Serv. Comm'n of City of N.Y.,
630 F.2d 79, 100–05 (2d Cir. 1980).
The district court in this case expressly adopted the
approach most favorable to the Officers, citing 29 C.F.R.
§ 1607.14(C)(9), for the proposition that "[t]he use of a ranking
device requires a separate demonstration that there is a
relationship between higher scores and better job performance."
Lopez, 2014 U.S. Dist. LEXIS 124139, at *16–17. As we have noted,
supra, and as the Officers seem to ignore, the court then
specifically found that it was "satisfied on the evidence that
Boston carried its burden of showing" "that persons who perform
better under the test method are likely to perform better on the
job." Id. at *61–62. As a predicate to this finding, the district
court observed that a group of incumbent sergeants who took an
exam in 2005 that contained 53 of the questions asked of applicants
on the sergeant's exam had a materially higher passing rate on
those common questions than did the job applicants. Id. at *56–
57. The district court viewed this evidence as showing that "those
questions were related to the sergeants' actual performance of
their jobs." Id. at *57. The Officers' only reply is to say that
this evidence only shows that people who previously did well on
the exam (and thus became sergeants) still did well on it. But
the Officers point to no evidence that these incumbent sergeants
in 2005 somehow achieved their positions by previously taking the
- 34 -
same, or more or less the same, exam that was first offered in
2005.
Even accepting the district court's opinion that added
scrutiny was called for because rank ordering was used, whatever
added scrutiny one need apply here certainly falls short of the
added scrutiny one would apply if rank ordering had been a material
contributor to the disparate impact. Although they belatedly offer
on appeal, without citation to the record, counsel's own
calculations that "banding" in lieu of rank order selection would
have caused more Black and Hispanic applicants to be "reachable"
for selection by subjective "performance" criteria, the Officers
made no effort to demonstrate that an increased number of Black
and Hispanic applicants likely would have been selected under such
an alternative approach. Rank ordering furthers the City's
interest in eliminating patronage and intentional racism under the
guise of subjective selection criteria. Such a goal is itself a
reasonable enough business need so as to provide some weight
against a challenge that is unaccompanied by any showing that rank
order selection itself caused any disparate impact in this case.13
13
Given the absence of any showing that an equally or more
valid alternative to rank-order selection would have reduced
disparate impact, we need not address the Officers' arguments that
any state law favoring rank order selection is unlawful or
preempted.
- 35 -
None of this is to suggest that Boston could not have
come up with an exam that did a better job of furthering its goal
of selecting the best candidates for promotion to the position of
sergeant. The Officers argue persuasively that Boston could have
made the exam more valid. Indeed, Outtz agreed and so, too, it
would appear, does the City, which, counsel tells us, has since
2008 developed a new exam that it now uses.
The point, instead, is that the record contains
detailed, professionally buttressed and elaborately explained
support for the district court's finding "that persons who perform
better under the test method are likely to perform better on the
job." Id. at *61. Given that plainly supported finding, it makes
little sense to debate in the abstract how much better the exam
might have been. Instead, it makes more sense to move to the next
step of the inquiry to see if there is any alternative selection
test that would have had less adverse impact. If so, then the
court will have a meaningful gauge of validity by comparing the
two tests. And if the alternative test with less adverse impact
has equal or greater validity, it makes no difference how valid
the employer's actual test is; the employee wins. Ricci, 557 U.S.
at 578 (citing 42 U.S.C. §§ 2000e–2(k)(1)(A)(ii) and (C)).
Conversely, absent proof of an equally or more valid test that has
less adverse impact, there is no reason to force the employer to
promote randomly if the employer has a tool that will do
- 36 -
meaningfully better than that. For this reason, once a court
concludes that a selection device is materially more job-related
than random selection would be, it makes sense to turn the focus
sooner rather than later to the question of whether there is any
alterative option that is as good or better, yet has less adverse
impact. Otherwise, courts and litigants are left to engage in
unpredictable qualitative assessments without any meaningful gauge
as to what is enough. We therefore turn next to that question.
C. The Officers' Alternatives
So, the pivotal question on appellate review is whether
the evidence compelled a finding "that the employer refuse[d] to
adopt an available alternative employment practice that has less
disparate impact and serves the employer's legitimate needs." Id.
To carry this burden, plaintiffs must "demonstrate a viable
alternative and give the employer an opportunity to adopt it."
Allen v. City of Chicago, 351 F.3d 306, 313 (7th Cir. 2003).
Outtz explained that he thought the Officers would be
unlikely to carry this burden due to the very large number of
applicants for relatively few open positions in Boston. On the
2008 exam, for example, where the disparate impact was much greater
than in 2005, there were only 26 openings for 504 applicants. He
explained that his experience is that:
[I]n dealing with adverse impact[,] the ball
game is played, for the most part, in terms of
selection ratio. If I come to--if an employer
- 37 -
comes to me and says, "Look, I've got five job
openings and I've got 5,000 people that are
applying for those five jobs and I want you to
develop a system that reduces adverse impact,"
I'm just going home.
The Officers' own expert agreed that the selection ratio heavily
influenced the menu of available options, offering his opinion
that the degree of adverse impact caused by a selection process
"depends so much on how many people you appoint."
The Officers baldly assert that the district court did
not find "that Plaintiffs failed to meet their burden of putting
forward a specific less discriminatory alternative." In fact, the
district court precisely so found--twice. Lopez, 2014 U.S. Dist.
LEXIS 124139, at *78 (holding that the Officers' showing was "not
enough to carry their burden on this issue" and did not
"demonstrat[e] by the evidence that there was an alternative
employment practice with equal validity and less adverse impact
that was available and that BPD refused to adopt").
The Officers also contend that "[i]t is undisputed
that . . . adding test components such as an assessment center,
structured oral interview, or performance review to an exam process
increases the validity of an exam while having less adverse impact
on minorities." Yet the Officers failed to offer any evidence
that would have compelled the district court to find that the
deployment of any of these supposedly "undisputed" solutions would
have led to "a smaller racial disparity in outcomes," Jones, 752
- 38 -
F.3d at 55, given the selection ratios facing authorities in
Boston.
Our own review of the record does disclose testimony
convincingly establishing that, as a general matter, incorporation
of selection tools such as use of "hurdles," banding, oral
interviews, so-called assessment centers, and open ended
"situational judgment" questions generally tend to result in less
adverse impact than does a reliance on multiple choice exams. What
is missing, though, is any rebuttal to Outtz's opinion that the
low rates of job openings in the Boston sergeant ranks relative to
the number of applicants made it unlikely that any alternative
selection device would have materially reduced adverse impact in
2005 and 2008.
The Officers did offer evidence that the mean
differentials on the oral portion of an exam Boston used in 2002
were less than the mean differentials on the written portions of
that exam. But the 2002 exam as a whole still had substantially
the same adverse impact as did the exams administered in 2005 and
2008.14 And, again, the Officers provide no analysis of the effect
of the selection ratios in 2005 and 2008.
Additionally, as the district court noted, Boston's
prior attempt to employ assessment centers with situational
14
The adverse promotional impact ratio in 2002 was calculated
to be .32. In 2005, it was .28.
- 39 -
exercises and oral questioning in its 2002 promotional exam
resulted in a cost of $1.2 million to develop the exam and the
required "transporting, housing, and training a substantial number
of police officers from throughout the country who acted as the
assessors," id. at *70, without generating any convincing support
that repeating such an approach in 2005 or 2008 would have reduced
adverse impact, id. at *73. In concluding that the City was not
required to again incur such costs without any demonstration that
adverse impact would be materially reduced, the district court
acted well within its discretion in making the judgments called
for by the applicable law.15 See Watson, 487 U.S. at 998 (opinion
of O'Connor, J.) ("Factors such as the cost or other burdens of
proposed alternative selection devices are relevant in determining
whether they would be equally as effective as the challenged
practice in serving the employer's legitimate business goals.").
Satisfying a plaintiff's burden on this point at trial
"demands evidence that plaintiffs' preferred alternative would
have improved upon the challenged practice," Johnson, 770 F.3d at
477, not just that such practices exist in the abstract.
15
Boston had previously tried other tactics to reduce adverse
impact. In 1992 and 2002 Boston experimented by integrating an
assessment center component into the exam. After the 1992 exam,
the City used its bypass authority to promote several Black
candidates over Caucasian candidates in order to achieve
compliance with a consent decree and the Guidelines. They were
sued and the bypasses were reversed. See Abban, 748 N.E.2d 455.
- 40 -
Furthermore, securing the reversal of a trial court's factual
finding that the Officers' proof on this point was not persuasive
required evidence that is so compelling as to render its rejection
clear error. The Officers' scattershot listing of alternatives
without any developed rejoinder to Outtz's testimony concerning
the challenge posed by the selection ratios in 2005 and 2008 fell
short of this mark.16
III. Conclusion
Given our finding that the district court applied the
correct law and committed no clear error in finding persuasive the
expert evidence tendered by Boston, we affirm the district court's
order finding that the exams Boston used in 2005 and 2008 did not
violate Title VII and we therefore affirm as well the entry of
judgment in favor of all defendants.
16 The Officers' failure to explain how a particular
alternative would have reduced disparate impact in 2005 and 2008
--and by how much--is particularly odd given the obvious mootness
of their claim for injunctive relief. Consequently, had the remedy
phase of trial proceeded as the Officers would have hoped, each
officer would have needed to show that, more likely than not, he
or she would have been promoted had Boston used an equally or more
valid selection tool with less impact. See 42 U.S.C. § 2000e–
5(g)(1) (authorizing "back pay" remedy for Title VII violation);
Azimi v. Jordan's Meats, Inc., 456 F.3d 228, 235 (1st Cir. 2006)
("Injuries allegedly caused by the violation of Title
VII . . . must be proven to the factfinder . . . which may
reasonably find, within the law, that while there has been
[injury], the plaintiff has not been injured in any compensable
way by it."). How any officer could have made such a showing
without first securing a liability finding predicated on a specific
alternative selection tool that would have been equally or more
valid and produced less adverse impact is entirely unclear.
- 41 -
- Opinion Concurring in Part and Dissenting in Part Follows -
- 42 -
TORRUELLA, Circuit Judge, concurring in part and
dissenting in part. I agree with my colleagues in the majority
only to the extent that the challenged tests did have a disparate
impact. There is little doubt in my mind, however, that the
majority's question, whether "the employer[s] show[ed] that the
challenged employment practice creating this disparate result is
nevertheless job-related for the position in question and
consistent with business necessity," supra at 15, cannot be
answered in the affirmative based on this record.17 To my view,
the district court committed clear error in finding that the
challenged tests were valid when placed under the legal prism of
Title VII, 42 U.S.C. § 2000e et seq. M.O.C.H.A. Soc'y, Inc. v.
City of Buffalo, 689 F.3d 263, 275 (2d Cir. 2012); Ass'n of Mex.-Am.
Educators v. California, 231 F.3d 572, 584–85 (9th Cir. 2000) (en
banc); Melendez v. Ill. Bell Tel. Co., 79 F.3d 661, 669 (7th Cir.
1996); Hamer v. City of Atlanta, 872 F.2d 1521, 1526 (11th Cir. 1989);
Bernard v. Gulf Oil Corp., 890 F.2d 735, 739 (5th Cir. 1989).
A review of the record shows that Boston18 did not,
contrary to the district court's finding and the majority's
assertion, "show[] that the content of the selection procedure is
representative of important aspects of performance on the job for
17 I would also have found the Officers established a prima
facie case as to all defendants, but, as the majority does not
address this question, supra at 7, I will focus on test validity.
18 Like the majority, supra at 7, I will refer primarily to
Boston for the sake of simplicity.
- 43 -
which the candidates are to be evaluated." Supra at 16 (quoting
29 C.F.R. § 1607.5(B)); see also 29 C.F.R. § 1607.5(A). Because
there is ample precedent on which to draw, see, e.g., Bos. Chapter,
NAACP, Inc. v. Beecher, 504 F.2d 1017 (1st Cir. 1974), I need not
engage the majority's emphasis on the non-binding nature of EEOC
Guidelines, supra at 17-18, nor rest my objection on what I would
consider the Guidelines' rather overwhelming persuasiveness vis-
à-vis this case. Id. at 17 (citing Jones v. City of Bos., 752
F.3d 38, 50 n.14 (1st Cir. 2014)). It is enough to say that, based
on our precedent and this record, there is a solid legal basis to
find that the district court's acceptance of Boston's case for
content validity is clearly erroneous.
The most significant flaws in Boston's case for validity
should each independently have been fatal to it: Boston failed to
demonstrate (1) that the 1991 Validation Report and 2000 job
analysis were applicable and reliable19 and (2) that the exams
tested "representative" and critical knowledge, skills, and
abilities ("KSAs") necessary to quality for the position of police
sergeant.
This first flaw stems from "the way in which the
validation study was performed" and its effect on test validity.
19 As I would find neither sufficed to support the exams'
validity, it does not matter which Boston relied upon for each
test, the 2000 job analysis or 1991 Validation Report. See supra
at 10 n.3.
- 44 -
Beecher, 504 F.2d at 1025. The Validation Report and job analysis
were defective. The district court acknowledged the "rule of
thumb" that a job analysis should typically have been performed
within the last five to eight years to be reliable. López v. City
of Lawrence, No. 07-11693-GAO, 2014 U.S. Dist. LEXIS 124139, at
*51 (D. Mass. Sept. 5, 2014). Yet, the 1991 job analysis and
resultant Validation Report predate the first of the contested
exams by fourteen years. Neither of the two conditions noted by
the district court as potentially saving an older analysis from
obsolescence -- lack of change in job requirements or a later
review updating the analysis -- rescue the Report. Id.; cf. 29
C.F.R. § 1607.5(K) (explaining totality of circumstances should be
considered in determining whether a validation study is outdated).
The Officers bolstered the presumption that a test more
than eight years old is not reliable, and the common sense
conclusion that a position changes over time, by pointing to
specific evidence that defendants' police departments changed
practices since the Report and analysis were performed: testimony
from Commissioner Edward F. Davis that Lowell implemented a
community policing model and a 2002 Boston Commissioner's memo
referring to changes in policing policy and practice. While the
district court was entitled to rely on Dr. Outtz's testimony as to
the unchanging nature of the position of sergeant, it clearly erred
in doing so for the proposition it drew from his testimony, that
- 45 -
the position of police sergeant in the defendant departments had
not changed, as Dr. Outtz based his statement on "[his] experience
generally" regarding the position in other municipalities,
including those in other states.
The subsequent job analysis completed in 2000, within
the time range to be presumed reliable, is unreliable by virtue of
the way it was performed. The 2000 job analysis suggests that the
eleven subject matter experts ("SMEs"), sergeants and detective
sergeants, relied upon by the testing firm to evaluate KSAs and
tasks for inclusion in the exam, were to do so individually; the
analysis details procedures for reconciling disparate results to
determine which items should make the cut. For example, "[f]or a
KSA to be included as a [sic] important component of the Police
Sergeant position, the KSA had to be rated by nine . . . of the
eleven . . . SMEs" in a certain way across all five categories.
Yet the eleven SMEs evaluating 160 KSAs each rated all 160 KSAs'
five attributes -- job relatedness, time for learning, length of
learning, differential value to performance, and necessity20 -- in
exactly the same way, although there were 72 possible ways to rate
20 Job relatedness could be answered "[y]es" or "[n]o"; time
for learning, "[b]efore assignment" or "[a]fter assignment";
length of learning, "[l]onger than brief orientation" or "[b]rief
orientation"; differential value to performance, "[h]igh,"
"[m]oderate," or "[l]ow"; and necessity, "[r]equired,"
"[d]esirable," or "[n]ot required."
- 46 -
each KSA. The same was true of task ratings, wherein each SME was
supposed to rate each of 218 tasks' frequency, importance,
necessity, relationship to performance, and dimensions,21 despite
the fact that each of 218 tasks could be rated in 1,674 ways. I
will not speculate as to how and why this total agreement occurred
but only observe that an analysis that generates a result so
unfathomably inconsistent with its proposed methods is not
reliable.22 As such, it was clear error to find the 2000 job
analysis supports the exams' validity. Beecher, 504 F.2d at 1025.
Beyond these threshold issues, the resultant exams did
not test a representative portion of KSAs. See 29 C.F.R.
§ 1607.5(B). Nor did they test critically important KSAs "in
proportion to their relative importance on the job." Guardians
Ass'n of N.Y.C. Police Dep't, Inc. v. Civil Serv. Comm'n of N.Y.C.,
633 F.2d 232, 243-44 (2d Cir. 1980) (citation omitted); see also
21 Frequency could be rated "[r]egular[]," "[p]eriodic[],"
or "[o]ccasional[]"; importance, "[v]ery important,"
"[i]mportant," or "[n]ot important"; necessity, "[n]ecessary upon
entry" or "[n]ot necessary"; and relationship to performance,
"this task clearly separates the best workers," "better workers
seem to perform this better than poor or marginal workers," or
"[m]ost perform this task equally well." Dimensions could be
answered using any combination of "[o]ral [c]ommunication,"
"[i]nterpersonal [s]kills," "[p]roblem ID & [a]nalysis,"
"[j]udgment," and "[p]lanning and [o]rganizing" or "all."
22 A second suspect aspect of this analysis, one that further
clarifies how troubling the purported across-the-board agreement
is, is in how the SMEs rated certain KSAs and tasks. For example,
all eleven SMEs -- including two assigned to administrative roles,
-- responded that "[s]et[ting] up command posts at scenes
of[]robberies, homicides, fires, etc.," was a "daily" task.
- 47 -
Beecher, 504 F.2d at 1024 (noting district court did not err in
finding that two significant correlations between exam and job
performance components did not make "'convincing' evidence of job
relatedness" (citation omitted)); see also 29 C.F.R.
§ 1607.14(C)(2) (an exam should measure "critical work behavior(s)
and/or important work behavior(s) constituting most of the job").
The 2000 job analysis identified 163 "important tasks"
and 155 "important" KSAs. The district court acknowledged that
the eighty-point multiple-choice portion of the exams tested
primarily the "K" of the KSAs, knowledge, and failed to measure
key skills and abilities, and thus would not be independently
valid. López, 2014 U.S. Dist. LEXIS 124139, at *60–61. The E&E
component that purportedly compensated for the "SA" deficit,
edging the exams into the realm of validity, consisted of a single
sheet requiring candidates to bubble in responses as to length of
work experience in departmental positions by rank, educational
background, and teaching experience. As the majority concedes,
this component had a minimal effect on score. Supra at 20–21.
The conclusion that more than half, López, 2014 U.S.
Dist. LEXIS 124139, at *54, or nearly half, supra at 27 n.11, of
applicable KSAs were or could be tested by the exams overestimates
the number of KSAs tested by the E&E component. But even if that
estimate were correct, relying upon this quantitative measure
misses that representativeness is partly qualitative.
- 48 -
It is quite a stretch to conclude that the E&E's bubbles
incorporated measures of the majority of key skills and abilities.
It is even more difficult to conclude from the record that the
skills and abilities measured received representative weight.
Supra at 21. How, exactly, could this worksheet test, as the
testability analysis suggests, "[k]nowledge of the various
communities within the Department's jurisdiction and the factors
which make them unique," "[s]kill in perceiving and reacting to
the needs of others," or "[k]nowledge of the procedures/techniques
when a major disaster occurs,"? And how, if it only affected the
ultimate score by five to seven percent at most, supra at 20, could
it be said that the KSAs for which the E&E ostensibly tested were
adequately represented relative to those KSAs tested on the
multiple-choice component?
The exam's failure to include particularly significant
KSAs also precludes representativeness. See Gillespie v.
Wisconsin, 771 F.2d 1035, 1044 (7th Cir. 1985) ("To be
representative for Title VII purposes, an employment test must
neither: (1) focus exclusively on a minor aspect of the position;
nor (2) fail to test a significant skill required by the position."
(emphasis added)); Guardians, 630 F.2d at 99. The exams here may
have tested the knowledge a supervisor must have but omitted any
meaningful test of supervisory skill, which is unquestionably
essential to the position of police sergeant. López, 2014 U.S.
- 49 -
Dist. LEXIS 124139, at *51. Written tests of supervisory skill
have been found by other courts to be altogether inadequate to
evaluate that attribute. See Vulcan Pioneers, Inc. v. N.J. Dep't
of Civil Serv., 625 F. Supp. 527, 547 (D.N.J. 1985), aff'd on other
grounds, 832 F.2d 811, 815-16 (3d Cir. 1987); see also Firefighters
Inst. for Racial Equal. v. City of St. Louis, 549 F.2d 506, 513
(8th Cir. 1977).
As in Beecher, "[t]here are, in sum, too many problems
with the test . . . to approve it here." 504 F.2d at 1026. It
cannot be anything but clear error, supra at 23, to find valid
exams based on an outdated validation report and a facially flawed
job analysis, exams that are not only unrepresentative but also
omit critical KSAs for the position of police sergeant. To endorse
the means by which these exams were created and the exams
themselves here establishes a perilous precedent that all but
encourages corner-cutting when it comes to Title VII.
On these grounds, I respectfully dissent.
- 50 -