*652OPINION OF THE COURT
SCIRICA, Chief Judge.William Houston retired from his position with the Easton Area School District in July 1999. Houston brought suit under Title VII of the Civil Rights Act of 1964, as amended, 42 U.S.C. §§ 2000e to 2000e-17, alleging the School District discriminated against him based on his race (African American) when he received payment for only 25% of the value of his unused sick days upon his retirement, while four comparable white employees received payment for 100% of their unused sick days upon their respective retirements. The District Court granted in part the School District’s motion in limine, excluding evidence of the retirement packages of three of the four alleged comparators on the basis that they were not similarly situated to Houston because their positions were not covered under a Pennsylvania statute commonly known as Act 93, 24 P.S. § 11-1164. Houston challenges the court’s exclusion of comparator evidence from the trial. Because we find Act 93 status should not have been determinative as to whether the employees were similarly situated to Houston, we will vacate the judgment of the District Court.1
I.
At the end of his 34-year career with the School District, Houston was employed as the Director of Support Services. In this position, Houston was a member of the superintendent’s cabinet, which was an unofficial term used by the School District to designate the members of the superintendent’s administrative team. Houston was appointed to the cabinet by Superintendent Joseph Piazza in 1995 or 1996. Houston served on Superintendent Piazza’s cabinet contemporaneously with employees Louis Ciccarelli, Business Manager; Karl Hettel, Director of Personnel; and Roger Wrazien, Director of Elementary Education.2
Superintendent Piazza submitted his resignation letter in the spring of 1997, to be effective at the end of the summer. Piazza’s resignation letter specifically requested that he be paid for all his unused sick days. The Easton Board of Education approved his request in April 1997. After Piazza’s retirement, Bernadette Meek replaced him as Superintendent. Houston, Ciccarelli, Hettel, and Wrazien remained in their positions as part of Superintendent Meek’s cabinet.
Houston submitted his resignation letter in June 1998, to be effective the following month. Houston’s letter did not request that he be paid for all his unused sick days.3 Pursuant to School District policy, he was paid 25% of the value of his unused sick days upon retirement, which amounted to a payment of $32,919.04. Ciccarelli, Hettel, and Wrazien all retired within a few months of each other in the summer of 1999. Although the general policy was to pay employees 25% of the value of their unused sick days, Superintendent Meek *653agreed to pay Cieearelli, Hettel, and Wrazien 100% of their unused sick days.4
Act 93, 24 P.S. § 11-1164, part of the Pennsylvania Public School Code of 1949, requires school employers to adopt written compensation plans for certain school administrators whose positions are not included in a bargaining unit. 24 P.S. § 11-1164(d). Act 93 explicitly excludes superintendents, business managers, and personnel directors from covered school administrators. Id. § ll-1164(a). The School District’s Act 93 Compensation Plan at issue applied to school administrators below the rank of superintendent and, consistent with the statute, excluded the positions of business manager and director of personnel. Thus, Piazza (as Superintendent), Cieearelli (as Business Manager), and Hettel (as Director of Personnel) were not covered by the Act 93 Plan, while Houston and Wrazien were covered.
The School District’s Act 93 Plan provided that “[flringe benefits shall be determined by the Superintendent of Schools and/or the Board of Education.”5 Aceording to Hettel, the School District had a policy of treating the business manager and the director of personnel the same as Act 93 employees for purposes of fringe benefits. According to Superintendent Meek, personnel directors and business managers could be remunerated the same as Act 93 employees, but they did not have to be treated the same. She testified that their benefits were left to the discretion of the superintendent, and that she used her discretion to agree to pay Cieearelli (not covered by Act 93), Hettel (not covered by Act 93), and Wrazien (covered by Act 93) 100% of their sick time at retirement.
In an October 3, 2005 Order, the District Court, per Magistrate Judge Rapoport, granted in part the School District’s motion in limine. The Order excluded evidence of the retirement packages of Piazza, Cieearelli, and Hettel on the basis that these individuals were not similarly situated to Houston because their positions were excluded from the Act 93 Plan.6 In 2008, the case was taken out of suspense and was reassigned to Magistrate Judge Per-*654kin. Before trial, Houston filed a motion for reconsideration of the Order on the motion in limine, which Magistrate Judge Perkin denied on October 8, 2005. A one-day bench trial was held on October 15, 2008. In a February 24, 2009 Memorandum Opinion and Verdict, the court entered judgment for the School District.
II.
Houston argues the District Court’s evidentiary decisions constituted clear error and we therefore should remand for a new trial. Houston contends evidence of the retirement packages of the excluded individuals should have been admitted as relevant to his case. He further contends the evidence demonstrated that Act 93 status was not relevant to the payment of sick time upon retirement and that the excluded individuals were similarly situated to him because they served in cabinet positions at the same time, had comparable years of service, and retired within the same time frame as him. The School District contends the District Court correctly determined that Wrazien was the only appropriate comparator because the retirement benefits provided under a specific plan should not be comparable, for purposes of establishing a discrimination claim, to those provided to employees not covered by that plan.
We review the District Court’s decision to exclude evidence on a motion in limine for abuse of discretion. See Bernardsville Bd. of Educ. v. J.H., 42 F.3d 149, 161 (3d Cir.1994). An abuse of discretion may be found when “the district court’s decision rests upon a clearly erroneous finding of fact, an errant conclusion of law, or an improper application of law to fact.” Int’l Union v. Mack Trucks, Inc., 820 F.2d 91, 95 (3d Cir.1987). We will reverse, however, only if we find the District Court’s error was not harmless. Donlin v. Philips Lighting N. Am. Corp., 581 F.3d 73, 80 (3d Cir.2009).
To make a comparison of the plaintiffs treatment to that of an employee outside the plaintiffs protected class for purposes of a Title VII claim, the plaintiff must show that he and the employee are similarly situated in all relevant respects. See Holifield v. Reno, 115 F.3d 1555, 1562 (11th Cir.1997) (collecting cases). The Supreme Court has held that evidence offered in a discrimination case concerning purported comparators with different supervisors is neither per se admissible nor per se inadmissible. See Sprint/United Mgmt. Co. v. Mendelsohn, 552 U.S. 379, 128 S.Ct. 1140, 1143, 170 L.Ed.2d 1 (2008). We can glean from Mendelsohn that whether a factor is relevant for purposes of a similarly situated analysis must be determined by the context of each case.
In disciplinary cases or in the context of personnel actions, for example, the relevant factors often include a “showing that the two employees dealt with the same supervisor, were subject to the same standards, and had engaged in similar conduct without such differentiating or mitigating circumstances as would distinguish their conduct or the employer’s treatment of them.” Radue v. Kimberly-Clark Corp., 219 F.3d 612, 617-18 (7th Cir.2000) (citation omitted); see also Johnson v. Kroger Co., 319 F.3d 858, 867 (6th Cir.2003) (“In the context of personnel actions, the relevant factors for determining whether employees are similarly situated often include the employees’ supervisors, the standards that the employees had to meet, and the employees’ conduct.” (citation omitted)). In a failure to transfer or promote case, the relevant factors may include some of the above, in addition to evidence that the employees “possessed analogous attributes, experience, education, and qualifications relevant to the positions sought.” Radue, 219 F.3d at 618. In a severance or *655retirement benefits case, the relevant factors may include the positions held, policies or plans in effect, the decisionmakers, and the timing of the separations/retirements. See McGuinness v. Lincoln Hall, 263 F.3d 49, 54-55 (2d Cir.2001) (plaintiff established she was similarly situated to a colleague who received more money in severance where the two employees “held positions of roughly equivalent rank (both in the Executive Cabinet), ... were fired at roughly the same time, [and] the decisions with respect to the severance were both made at the highest levels of the company”). Additional factors may include whether the employees “retired under normal conditions, early, or pursuant to a settlement or other special circumstances.” Sellars v. City of Gary, 453 F.3d 848, 851 n. 1 (7th Cir.2006). The plaintiff is not required to show that he is identical to the comparator in each relevant factor, “but he must show substantial similarity.” Radue, 219 F.3d at 618 (citation omitted).
The District Court considered only Act 93 status in its determination of whether Piazza, Cicearelli, Hettel, and Wrazien were similarly situated to Houston. The court, however, did not consider the evidence showing that Act 93 status was not determinative as to the amount of sick time paid to School District employees upon retirement. Depending on the facts and evidence presented, Act 93 status or coverage may or may not be relevant—but in this case it is not dispositive. Because of the heavily fact-driven character of the inquiry, the mere presence of a particular compensation statute or plan implementing that statute is not determinative of whether employees are similarly situated for a discrimination claim. This is especially the case where, as here, (1) the Act 93 Plan provided that fringe benefits were to be determined by the superintendent and/or the Board of Education and did not specify the percentage of unused sick days that would be paid upon retirement; (2) the School District had a general policy of paying all employees, regardless of Act 93 status, 25% of their unused sick days at retirement; and (3) Superintendent Meek retained the discretion to pay both Act 93 and non-Act 93 employees more than 25% of their unused sick days.
We find that under the facts of this case, Act 93 status should not have been the dispositive factor for purposes of determining whether Houston was similarly situated to other School District administrators. Evidence of the retirement packages of Piazza, Cicearelli, and Hettel should not have been excluded based on the fact that their positions were not covered by the School District’s Act 93 Plan. Because the District Court relied solely on Act 93 status, it abused its discretion in excluding evidence of the retirement packages of these individuals.
Furthermore, we cannot say this error was harmless. The similarly situated analysis is for the District Court to determine in the first instance. See Mendelsohn, 128 S.Ct. at 1146 (“[(Questions of relevance and prejudice are for the District Court to determine in the first instance.” (citing U.S. v. Abel, 469 U.S. 45, 54, 105 S.Ct. 465, 83 L.Ed.2d 450 (1984))). But it appears that at least Cicearelli and Hettel may be similarly situated to Houston.7 Houston, Cicearelli, and Hettel were all long-term employees of the School District, held positions of roughly equivalent rank (in the *656superintendent’s cabinet) during the same time period, and retired within a year of each other. Moreover, decisions regarding the fringe benefits each cabinet member received upon retirement were all made by Superintendent Meek. If Houston had been able to present evidence that Ciccarelli received and Hettel was promised 100% of his sick time upon retirement, the burden of production likely would have shifted to the School District to present a legitimate, nondiscriminatory reason for why each employee received more favorable treatment than Houston. See Fuentes v. Perskie, 32 F.3d 759, 763 (3d Cir.1994). Houston, in turn, would have had the opportunity to show that the School District’s stated reasons were a pretext for discrimination. Id. Evidence of additional similarly situated employees outside of Houston’s protected class—each of whom received or was promised 100% of his sick pay upon retirement—likely would have changed the entire complexion of the trial.
Accordingly, we will vacate the judgment in favor of the School District and remand for a new trial. On remand, the District Court should engage in a factual analysis of the relevant factors to determine whether any of the excluded comparators is similarly situated to Houston.8
III.
In summary, and for the reasons set forth above, we will vacate the judgment of the District Court and remand for further proceedings consistent with this opinion.
. The District Court had jurisdiction under 28 U.S.C. § 1331, and we have jurisdiction under 28 U.S.C. § 1291.
. Piazza, Ciccarelli, Hettel, and Wrazien are Caucasian.
. Houston testified that at the time he submitted his resignation letter, it was his understanding he was entitled to receive payment for 25% of his unused sick days. He had heard, however, that some people had received payment for 100% of their unused sick days. As a result, Houston asked Hettel, Director of Personnel, on two or three occasions whether it would be possible for him to receive payment for 100% of his unused days. Houston testified that Hettel told him he was entitled to only 25%.
.Cieearelli was paid the full amount of his sick time soon after his retirement. The School District thereafter initiated litigation against Cieearelli seeking reimbursement for unused sick-day compensation paid to him in excess of 25% of his time. Easton Area Sch. Dist. v. Ciccarelli, No. C-0048-cv-2001009149 (Ct. C.P. Northampton County). Upon retirement, Hettel and Wrazien both arranged for their sick-day payments to be paid in equal installments over four years. They received only the first payment of 25%. Wrazien thereafter initiated litigation against the School District for the remaining payments. See Wrazien v. Easton Area Sch. Dist., 926 A.2d 585, 586-88, 590 (Pa.Cnmmw.Ct.2007) (affirming judgment in favor of Wrazien and finding no error in the trial court's decision that the School District was bound by the terms of Wrazien's retirement package negotiated by Meek and approved by the School Board). Hettel decided not to file suit against the School District and has not received his remaining payments.
. An Act 93 fringe benefits chart introduced at trial provided that all covered administrators would receive compensation for 25% of their unused sick days at retirement. Trial evidence, however, indicated that this chart may have been an outdated version of a worksheet Superintendent Meek used as a guide, not a mandate, to determine what benefits were provided to employees, and that the chart was not part of the formal Act 93 Plan. The chart was not submitted to the court as part of the School District’s motion in limine.
. The Order also excluded evidence of alleged “historical inequities” in compensation and an alleged "secret code” used to request payment for sick time in Ciccarelli's retirement letter. Houston does not challenge the exclusion of the historical inequities evidence. He does challenge the exclusion of the secret code language, which is discussed, infra, at note 8.
. We note that Houston appears to concede that Piazza, as Superintendent, is not an appropriate comparator. In his Reply Brief, Houston contends he is similarly situated to all the cabinet members (Cicearelli, Hettel, and Wrazien) because all the cabinet members were subject to the same standards in employment and they were all responsible for advising the superintendent. Piazza was not a cabinet member—as Superintendent, he supervised the cabinet members.
. The District Court also excluded evidence of alleged "secret code” language used in Ciccarelli's retirement letter because evidence of his retirement was excluded as not comparable to Houston. If the court determines on remand that Ciccarelli is an appropriate comparator, his retirement letter should be admissible in accordance with the court's ruling that similar language in Wrazien’s retirement letter was admissible.