Defendants-Appellants Michael Casper and Jay Bob Klinkerman (collectively “Defendants”) appeal the district court’s denial, without prejudice, of their Fed.R.Civ.P. 12(b)(6) motions to dismiss based on qualified immunity. As the district court ruled only that the evidentiary record was insufficient to rule on the merits of Defendants’ motions, we grant Plaintiffs-Appellees’ motion to dismiss this appeal for lack of appellate jurisdiction.
*1262 Background
Plaintiffs-Appellees Leslie Weise and Alex Young (collectively “Plaintiffs”) brought an action pursuant to Bivens v. Six Unknown Named Agents of Federal Bureau of Narcotics, 403 U.S. 388, 91 S.Ct. 1999, 29 L.Ed.2d 619 (1971), alleging that Defendants and five others, acting under color of law and pursuant to White House policy, violated their First and Fourth Amendment rights by ejecting them from a public appearance by President George W. Bush. Plaintiffs contend that they were ejected because a bumper sticker on their car expressed a viewpoint contrary to that of the President’s.
According to the complaint, on March 21, 2005, the President delivered a speech on Social Security at the Wings Over the Rockies Air and Space Museum, in Denver, Colorado. See Complaint ¶ 9.1 The White House set the policies and procedures as to who could attend the President’s speech, and tickets were made available to the public. See id. at ¶¶ 10-12. The White House also solicited the assistance of staff and volunteers, including Defendants, to carry out its attendance policies at the event. See id. at ¶¶ 11-12; Aplt. Br. at 4.
Plaintiffs obtained tickets to the event and arrived at the event in a vehicle, owned and driven by Plaintiff Weise, which had a bumper sticker that read “No More Blood For Oil.” See id. at ¶¶13, 15-16. After parking, Plaintiffs approached security. See id. at ¶¶ 17, 19. While Plaintiff Young was permitted to enter, Plaintiff Weise was directed to wait with Defendant Klinkerman, who identified himself as a “volunteer” from Colorado, and told Plaintiff Weise that the Secret Service wanted to speak with her. See id. at ¶¶ 19, 21. Soon thereafter, Defendant Casper, who wore a dark blue suit, earpiece and lapel pin approached and Defendant Klinkerman said, “that’s him” or “here he comes.” Id. at ¶ 22. Defendant Casper told Plaintiff Weise that “she had been TD’d’, and that if she had any ill intentions” or “tried any ‘funny stuff [she] would be arrested, but that he was going to let [her] in.” Id. at ¶ 23.
Shortly thereafter, Defendant Casper consulted with one or more of the unnamed defendants and was advised that one or more of them had, at the direction of a White House official, set a policy prohibiting anyone from attending the event if they held a viewpoint contrary to that held by the President. See id. at ¶ 25. A few minutes later, Defendant Casper approached Plaintiffs, who had reached their seats, and directed them to leave the event. See id. at ¶27. Plaintiffs were escorted out of the event and not allowed to reenter. See id. at ¶¶ 27-30. After the event, Secret Service confirmed that Plaintiffs were ejected because of the bumper sticker on their vehicle. See id. at ¶ 32. Plaintiffs claim they had no intention of disrupting the event, but if given the opportunity, one of them would have asked the President a question. See id. at ¶ 18.
Plaintiffs brought a Bivens action against Defendants in their individual capacities, alleging that Defendants acted under “color of federal law” and that “[a]t all times, the policies concerning attendance at the event were set by federal officials acting as federal officials, including some of the Doe defendants,” and that “Defendants Klinkerman and Casper ejected the plaintiffs at the direction of and pursuant to policies of those federal officials.” Id. at ¶¶ 33, 36. Plaintiffs further allege that Defendants “conspired” with and “acted in concert with the Doe defendants who directed the ejection and *1263who established the policies that were being enforced by the ejection.” Id. at ¶ 34.
Each Defendant filed a motion to dismiss the action on qualified immunity grounds. See Aplt. Br. at 2-3. In addition, both sought a protective order prohibiting all discovery until the issue of qualified immunity was resolved. See Motion to Dismiss for Lack of Appellate Jurisdiction of Plaintiffs-Appellees Leslie Wiese and Alex Young, Nov. 30, 2006, at 3. In response, Plaintiffs requested discovery on the issue of whether Defendants were entitled to qualified immunity as private parties alleged to have been acting under color of law. See id. The magistrate granted Defendants’ motions for a protective order and stayed all discovery until the pending motions to dismiss based on qualified immunity were decided. See id.
The district court denied Defendants’ motions to dismiss, without prejudice. See Weise v. Casper, No. 05-02355, 2006 WL 3093133, at *4 (D.Colo. October, 30, 2006). Although the district court found that Plaintiffs sufficiently alleged federal action to support their Bivens claim, it was unclear to the district court whether Defendants could assert a qualified immunity defense. Relying on Richardson v. McKnight, 521 U.S. 399, 117 S.Ct. 2100, 138 L.Ed.2d 540 (1997), the district court explained that because Defendants were private parties, they were entitled to assert a qualified immunity defense only if they were acting as federal officials or as private parties at the direction and close supervision of federal officials. See Weise, 2006 WL 3093133, at *4 (citing Richardson, 521 U.S. at 413, 117 S.Ct. 2100; Rosewood v. Sunflower Diversified Servs., Inc., 413 F.3d 1163, 1167 (10th Cir.2005)). The district court noted that Plaintiffs requested discovery on this “threshold issue,” that Defendants did not address it in their moving papers, and thus the district court found the record to be insufficient for it to rule on the merits of the motions. See id. at *2-*4. Accordingly, the district court ordered that Plaintiffs would be allowed limited discovery on the issue of “Defendants’ status at the time of the conduct at issue, whether Defendants were ‘closely supervised’ by government officials, and whether Defendants are entitled to assert [a] qualified immunity [defense].... ” Id. at *4.2 This appeal followed.
Discussion
Under the Supreme Court’s collateral order doctrine, Cohen v. Beneficial Indus. Loan Corp., 337 U.S. 541, 546, 69 S.Ct. 1221, 93 L.Ed. 1528 (1949), “a district court’s denial of a claim of qualified immunity, to the extent that it turns on an issue of law, is an appealable ‘final decision’ within the meaning of 28 U.S.C. § 1291 notwithstanding the absence of a final judgment.” Mitchell v. Forsyth, 472 U.S. 511, 530, 105 S.Ct. 2806, 86 L.Ed.2d 411 (1985). Accordingly, the Supreme Court has limited appeals of interlocutory decisions denying the defense of qualified immunity “to cases presenting neat abstract *1264issues of law.” Johnson v. Jones, 515 U.S. 304, 317, 115 S.Ct. 2151, 132 L.Ed.2d 238 (1995) (internal quotation omitted); see also Behrens v. Pelletier, 516 U.S. 299, 313, 116 S.Ct. 834, 133 L.Ed.2d 773 (1996). In contrast, pretrial determinations of evi-dentiary sufficiency in qualified immunity cases are not immediately appealable. See Behrens, 516 U.S. at 313, 116 S.Ct. 834; Jones, 515 U.S. at 314-18, 115 S.Ct. 2151. The Supreme Court has provided two rationales to support this distinction. First, evidentiary sufficiency determinations are not separable from a plaintiffs claim and thus do not constitute final decisions under Cohen and Mitchell. See Behrens, 516 U.S. at 313, 116 S.Ct. 834; Jones, 515 U.S. at 314-15, 115 S.Ct. 2151. Second, “considerations of delay, comparative expertise of trial and appellate courts, and wise use of appellate resources argue in favor of limiting interlocutory appeals of ‘qualified immunity’ matters to cases presenting more abstract issues of law.” Jones, 515 U.S. at 317, 115 S.Ct. 2151. These principles apply to interlocutory appeals at either the dismissal or summary judgment stage of a litigation. See Behrens, 516 U.S. at 306-07, 116 S.Ct. 834.
Notwithstanding these principles, Defendants argue that we have appellate jurisdiction over their appeal. We disagree. First, they argue that there can be no factual dispute on a Fed.R.Civ.P. 12(b)(6) motion to dismiss and that the complaint on its face establishes they are entitled to assert qualified immunity. See Defendanb-Appellant’s Response to Motion to Dismiss for Lack of Appellate Jurisdiction of Plaintiff-Appellees Leslie Weise and Alex Young, Dec. 15, 2006 (“Defendants-Appellants’ Response”), at 2 — 6; Aplt. Br. at 9-12; Reply Br. at 2-4. However, it is well-established that denials of qualified immunity based on a motion to dismiss are only immediately appealable to the extent they turn on issues of law. See Prager v. LaFaver, 180 F.3d 1185, 1190 (10th Cir.1999). If a district court cannot rule on the merits of a qualified immunity defense at the dismissal stage because the allegations in the pleadings are insufficient as to some factual matter, the district court’s determination is not immediately appealable. United States v. Yonkers Bd. of Educ., 893 F.2d 498, 502-03 (2d Cir.1990) (denial of motion to dismiss on grounds of immunity not appealable where applicability of immunity cannot be decided from the face of the pleadings and requires discovery); Lawson v. Abrams, 863 F.2d 260, 262-63 (2d Cir.1988) (same); see also Almonte v. City of Long Beach, 478 F.3d 100, 110 (2d Cir.2007).
Second, Defendants argue that the district court should have first analyzed the purely legal issue of whether a constitutional violation occurred based on the facts contained in the complaint and, if so, whether the constitutional right alleged to have been violated was clearly established. See Defendants-Appellants’ Response at 6-9; Aplt. Br. at 8-9; Reply Br. at 2-3. While this would be the ordinary course, see Siegert v. Gilley, 500 U.S. 226, 229-33, 111 S.Ct. 1789, 114 L.Ed.2d 277 (1991), this analysis can only proceed after the court determines that a defendant is entitled to assert qualified immunity in the first instance. See id. at 231, 111 S.Ct. 1789 (“Qualified immunity is a defense that must be pleaded by a defendant official.”); Wyatt v. Cole, 504 U.S. 158, 168, 112 S.Ct. 1827, 118 L.Ed.2d 504 (1992) (private parties who conspire with public officials to violate constitutional rights are not automatically immune from suit). The district court recognized that Defendants are not public officials and decided more inquiry was necessary before engaging in any further aspect of qualified immunity analysis. The district court did not abuse its discretion in ordering limited discovery given its concerns.
*1265Third, Defendants rely on Behrens, 516 U.S. at 305-14, 116 S.Ct. 834, in arguing that the district court’s discovery order forces them to renew their defense on summary judgment, thus depriving them of their right to dispose of the case at the dismissal stage. See Defendant-Appellants’ Response at 6. Behrens does not provide that Defendants are automatically entitled to appeal both the denial of a motion to dismiss and a motion for summary judgment. Rather Behrens rejects the “one-interlocutory-appeal” approach and clarifies that the denial of qualified immunity at the dismissal stage does not preclude a renewal of that defense at summary judgment after further factual development has occurred. Behrens, 516 U.S. at 305-11, 116 S.Ct. 834; see also Walker v. City of Orem, 451 F.3d 1139, 1146 n. 5 (10th Cir.2006). Further, it is well established that limited discovery may be necessary to resolve qualified immunity claims on summary judgment. See Anderson v. Creighton, 483 U.S. 635, 646 n. 6, 107 S.Ct. 3034, 97 L.Ed.2d 523 (1987) (explaining that although qualified immunity should be resolved at the earliest possible stage of litigation, “tailored” discovery may be necessary to resolve factual questions before disposing of the case on summary judgment).
Finally, Defendants argue that some of our sister circuits have exercised appellate jurisdiction in analogous cases. See Defendants-Appellants’ Response at 6-9. We disagree as each of the cases relied upon by Defendants are distinguishable from this case. Defendants principally rely on the Second Circuit’s holdings in Locurto v. Safir, 264 F.3d 154, 164-65 (2d Cir.2001) and X-Men Sec., Inc. v. Pataki, 196 F.3d 56, 66 (2d Cir.1999) and the Fourth Circuit’s holding in McVey v. Stacy, 157 F.3d 271, 275-76 (4th Cir.1998). These cases hold that when a district court denies a qualified immunity motion on the ground that a factual question exists as to whether the defendant’s conduct violated a constitutional right, that denial is reviewable by the court of appeals because the district court’s denial constitutes at least an implicit legal decision that the complaint alleges a constitutional claim on which relief can be granted, and that the constitutional right alleged to have been violated was clearly established at the time of defendant’s conduct. See Locurto, 264 F.3d at 164 (citing X-Men, 196 F.3d at 66); McVey, 157 F.3d at 276. In such cases, the district court must rely only on the facts as alleged in the complaint and assume that those facts, for the purposes of a motion to dismiss, are true. See Locurto, 264 F.3d at 164-65 (citing X-Men, 196 F.3d at 66-67); McVey, 157 F.3d at 276. Indeed, we have recognized that a district court cannot avoid ruling on the merits of a qualified immunity defense when it can resolve the purely legal question of whether a defendant’s conduct, as alleged by plaintiff, violates clearly established law. See Medina v. Cram, 252 F.3d 1124, 1130 (10th Cir.2001) (citing Malik v. Arapahoe County Dep’t of Soc. Servs., 191 F.3d 1306, 1315 (10th Cir.1999)).
The decisions relied upon by Defendants, however, are clearly distinguishable from the instant case. Here, the factual issue involves not whether Defendants’ conduct violated a constitutional right, the factual issue involves the threshold question of whether Defendants are entitled to assert qualified immunity in the first instance. Indeed, even the Second Circuit has recognized this distinction in squaring its Locurto holding with its Lawson holding. See Locurto, 264 F.3d at 165 (citing Lawson, 863 F.2d at 263). In this case, the district court made no legal decision whatsoever, explicit or implicit, on the merits of Defendants’ motions. The district court only determined that the allegations in the complaint even taken as true did not adequately address whether Defen*1266dants were entitled to assert qualified immunity, and thus ordered discovery on that question. Accordingly, the district court’s interlocutory order is not appealable.
The dissent argues we have jurisdiction over this appeal, contending evidentiary issues can never arise on a motion to dismiss and the district court effectively decided the legal issue that the facts, as alleged in the complaint, do not entitle Defendants to qualified immunity.3 We disagree.
First, we reject the dissent’s attempt to suggest a circuit split on the issue presented by this appeal, as the cases it cites by the Fifth and Seventh Circuits are inappo-site to the Second Circuit cases on which we rely. Neither of the cases relied upon by the dissent involved the review of a district court’s determination that the complaint was insufficient on its face to determine whether the defense of qualified immunity was properly before the court.4 In fact, in both cases, the district court denied defendants’ motions to dismiss on their merits, making the exercise of appellate jurisdiction entirely appropriate.5 See Levenstein v. Salafsky, 164 F.3d 345, 347 (7th Cir.1998); Doe ex rel. Doe v. Hillsboro Indep. Sch. Dist. 81 F.3d 1395, 1399 (5th Cir.1996).6 Further, that these cases state the general proposition that the denial of qualified immunity on a motion to dismiss constitutes a legal decision because Rule 12(b)(6) requires a court to accept as true all well-pleaded facts in the complaint is beside the point. Of course this proposition is true, but we emphasize that it is only true when the facts alleged in the complaint are well-pleaded. Here, the district court determined that whether qualified immunity was even available as a defense was unclear because the allegations in the complaint were not well-pleaded, and ordered limited discovery to correct *1267that problem. Surely the district court has discretion to undertake this course, but regardless, its determination clearly does not constitute a decision on an abstract point of law and is not reviewable under the collateral order doctrine.
Second, the dissent’s argument that evi-dentiary issues can never arise on a motion to dismiss is belied by Rule 12(b) itself. As acknowledged by the dissent, Rule 12(b) contemplates the possibility of factual insufficiency at the pleading stage and expressly permits the conversion of a motion to dismiss into one for summary judgment to resolve that problem. Dissent at 3. Further, it is well-settled that a district court has the discretion to effect such a conversion, sua sponte, by requesting information outside the pleadings. See 5C Charles Alan Wright & Arthur P. Miller, Federal Practice and Procedure § 1366. Notwithstanding the apparent position of the dissent that defendants who assert qualified immunity must always be spared from discovery,7 the Federal Rules of Civil Procedure do not operate differently in the context of qualified immunity. See id. (“There is considerable authority to the effect that the conversion provision applies to affirmative defenses raised on a Rule 12(b)(6) motion. As a result, when a defendant’s motion to dismiss raises an affirmative defense that is not apparent on the face of the pleadings and outside matter is presented and accepted, federal courts will generally treat the motion as if it were one for summary judgment.”) (footnotes omitted); see also Anderson, 483 U.S. at 646 n. 6, 107 S.Ct. 3034 (“tailored” discovery may be necessary to resolve factual questions concerning qualified immunity before disposing of the case on summary judgment).
Finally, the dissent claims that Defendants make the legal argument that the issue of whether they were closely supervised by federal officials is “not a necessary predicate to their invocation of qualified immunity” and that our holding denies Defendants the “valuable right to obtain a ruling on this significant legal question, and exposes them to the burden of discovery on an issue that, they contend, should be resolved as a matter of law.” Dissent at 1270. Although Defendants argue the district court’s denial of their motions to dismiss constituted an implicit decision that they are not entitled to invoke qualified immunity, see Defendants-Appellants’ Response at 9, Defendants do not argue that Richardson does not extend to the facts of this case. To the contrary, Defendants explicitly argue that “this case presents the precise situation envisioned by the Supreme Court in Richardson,” and claim that the facts, as alleged in the complaint, demonstrate they were closely supervised. See Aplt. Br. at 9-12; see also Reply Br. at 4-5.
Even if Defendants made this argument (which they do not), it would not change our holding. The district court made no legal ruling whatsoever as to whether Richardson extends to the facts of this case. The district court recognized that Richardson might apply to this case, but failed to rule either way on this issue because the factual predicate necessary to *1268conduct that analysis, namely Defendants’ status at the time their alleged conduct occurred, was unclear from the face of the pleadings. Therefore, the dissent’s claim that the district court’s decision constituted a “legal conclusion about the scope of qualified immunity for private parties” does not seem to have record support.
APPEAL DISMISSED.
. The Complaint can be found at pages 12-20 of Appellants’ Appendix.
. Following the district court's ruling, we allowed Plaintiffs to take depositions of Casper and Klinkerman for the limited purpose of identifying other potential defendants so Plaintiffs could file claims against them within the relevant statute of limitations. See Order, Feb. 15, 2007. As a result of information obtained during those depositions, Plaintiffs now agree that Defendants were closely supervised by public officials and are entitled to assert qualified immunity. See Aplee. Br. at 7-8. However, Plaintiffs’ concession cannot confer appellate jurisdiction on this court. See Garrett v. Stratman, 254 F.3d 946, 951 (10th Cir.2001) (even if a party concedes it, "we have a ‘special obligation to satisfy’ ourselves of appellate jurisdiction” (quoting Bender v. Williamsport Area Sch. Dist., 475 U.S. 534, 541, 106 S.Ct. 1326, 89 L.Ed.2d 501 (1986))). We still must examine whether the district court’s order is one where an interlocutory appeal would lie.
. Notwithstanding its conclusion that we have jurisdiction over this interlocutory appeal, the dissent contends that we should remand this case to the district court because the complaint, on its face, makes it "extremely difficult” for us to analyze the merits of Defendants’ qualified immunity defense. See Dissent at 1268 n. 1. It appears that the dissent concedes that further factual development is necessary to decide the merits of Defendants’ defense, which would support our position that there are times when the allegations contained in the complaint are inadequate to determine whether qualified immunity is even available as a defense. In any event, remand is not the proper course because we do not have appellate jurisdiction in the first place. The only way this case is immediately appealable is if the district court’s ruling turned on a question of law, whereby our merits review would not be "extremely difficult.”
. To our knowledge, the Second Circuit is the only circuit to address whether the denial of immunity at the dismissal stage is immediately appealable when the applicability of that defense could not be determined from the face of the pleadings.
. Indeed, the Second Circuit has exercised appellate jurisdiction in analogous cases. See, e.g., Iqbal v. Hasty, 490 F.3d 143, 152 (2d Cir.2007) ("[A] defendant may appeal a district court’s ruling denying qualified immunity when, if a plaintiff’s allegations are assumed to be true, the only question is whether the alleged conduct violated a clearly established right.”).
. Furthermore, the only dispute in these cases concerned whether defendants’ alleged conduct violated plaintiffs’ constitutional rights and whether those rights were clearly established. See Levenstein, 164 F.3d at 351-53; Hillsboro, 81 F.3d at 1406. Therefore, these cases are distinguishable from the instant case for the same reason we earlier distinguished the Second and Fourth Circuit cases relied upon by Defendants, namely that this type of factual dispute does not preclude appellate review. Supra at 1265-66. For the same reason, we disagree with the dissent’s contention that our decision in this case "would make nearly every denial of a motion to dismiss unappealable.” Dissent at 1270.
. To the contrary, the assertion of qualified immunity does not automatically act as a complete bar to discovery. See Mitchell, 472 U.S. at 526, 105 S.Ct. 2806 ("Indeed, Harlow emphasizes that even such pretrial matters as discovery are to be avoided if possible ....”) (emphasis added). The ordering of narrow discovery in qualified immunity cases is entirely permissible. See Garrett, 254 F.3d at 953 ("[qjualified immunity does not shield government officials from all discovery but only from discovery which is either avoidable or overly broad”) (alteration in the original). More importantly, narrowly tailored discovery orders designed to "uncover only those facts needed to rule on the immunity claim” are not immediately appealable. Id.