concurring in part and dissenting in part:
As the Supreme Court has specifically recognized, the right at stake here, the right to have “a jury’s verdict ... based upon the evidence developed at trial[,] goes to the fundamental integrity of all that is embraced in the constitutional concept of trial by jury.” Turner v. Louisiana, 379 U.S. 466, 472, 85 S.Ct. 546, 549, 13 L.Ed.2d 424 (1965). Unfortunately, the atrocious crimes committed and abundant circumstantial evidence pointing to Sherman as the perpetrator have led the majority to disregard this bedrock principle. Regardless of how much sympathy we have for the victims or how certain we may be of Sherman’s guilt, we must follow the Constitution. Adherence to it does not require that Sherman be released but it does require that he be provided a new trial. Accordingly, although I agree with some of the majority’s conclusions, I must respectfully dissent from its holding.
I.
Most constitutional errors are trial errors and can be harmless, but some “will always invalidate the conviction.” Sullivan v. Louisiana, 508 U.S. 275, 279, 113 S.Ct. 2078, 2081, 124 L.Ed.2d 182 (1993). Such “structural defects in the constitution of the trial mechanism ... defy analysis by ‘harmless-error’ standards.” Arizona v. Fulminante, 499 U.S. 279, 309, 111 S.Ct. 1246, 1265, 113 L.Ed.2d 302 (1991). Structural errors are those that affect “the framework within *1151which the trial proceeds.” Id. The right at issue in this case — the right to be convicted solely on the basis of evidence presented at trial — seems to me to fit squarely within this definition. Thus, an error involving this right, like errors involving the related right to a public trial or the right to be represented by counsel at trial, should be deemed structural. If we were writing on the proverbial clean slate, I would so hold.
After all, of what consequence is the right to counsel, to an impartial judge, to a public trial, to a correct reasonable doubt jury instruction,1 or to a criminal trial at all, if a defendant can be convicted based on evidence not presented at trial. The right to have a conviction based only on the evidence presented at trial, like other rights whose deprivation constitutes structural error, is one of those protections without which “a criminal trial cannot reliably serve its function as a vehicle for determination of guilt or innocence, and no criminal punishment may be regarded as fundamentally fair.” Rose v. Clark, 478 U.S. 570, 577-78, 106 S.Ct. 3101, 3106, 92 L.Ed.2d 460 (1986) (citation omitted).
Permitting a verdict to be based on evidence other than that presented at trial, like other structural errors, has repercussions that are “necessarily unquantifiable and indeterminate, unquestionably qualifying it] as ‘structural error.’” Sullivan, 508 U.S. at 282, 113 S.Ct. at 2083. This is particularly true in light of the rules of evidence and the restrictions they quite legitimately place on any inquiry into jury deliberations. See Tanner v. United States, 483 U.S. 107, 117-125, 107 S.Ct. 2739, 2745-50, 97 L.Ed.2d 90 (1987). Because they prohibit any inquiry into the effect that additional, unopposable, and possibly inadmissible evidence might have on a jury, a juror’s exposure to such evidence is necessarily unquantifiable and indeterminate. See United States v. Bagley, 473 U.S. 667, 693, 105 S.Ct. 3375, 3389, 87 L.Ed.2d 481 (1985) (“The private whys and wherefores of jury deliberations pose an impenetrable barrier to our ability to know which piece of information might make, or might have made, a difference”) (Marshall, J., dissenting).
“In the constitutional sense, trial by jury in a criminal case necessarily implies at the very least that the evidence developed against a defendant shall come from the witness stand in a public courtroom where there is full judicial protection of a defendant’s right of confrontation, of cross-examination, and of counsel.” Turner, 379 U.S. at 472-73, 85 S.Ct. at 550.2 Consequently, just as when a jury is given an improper reasonable doubt instruction, when a verdict is not based solely on evidence presented at trial, “there has been no jury verdict within the meaning of the Sixth Amendment” and so “no object ... upon which harmless error scrutiny can operate.” Sullivan, 508 U.S. at 280, 113 S.Ct. at 2082. The most an appellate court can conclude in the case at hand is that a jury would surely have found the defendant guilty based on the evidence presented at trial. A court cannot conclude anything about the jury’s actual finding of guilt, because that finding was based to some unknown extent on evidence that was never presented at trial. Cf. id. at 279-80, 113 S.Ct. at 2081-82.
The majority rightly recognizes that an error should not be deemed structural unless its presence renders “unfair” every criminal conviction in which the error occurs. See Maj. Op. at 1138. Structural error, however, is not merely a “shorthand” form of harmless error analysis, rather it involves the broader question of the fundamental fairness of the trial process. Accordingly, an error that so infects a trial as to render the process funda*1152mentally unfair is structural, regardless of whether, looking narrowly at its “actual impact” on the verdict, the error might sometimes be deemed harmless.
For example, the Supreme Court has held that the denial of the right to a public trial is structural error, Waller, 467 U.S. at 49 n. 9, 104 S.Ct. at 2217 n. 9, because it renders the entire process unfair, not because such an error can never be “harmless” in terms of its impact on the jury’s verdict —the harm is to the trial process, not necessarily to the specific verdict reached. Similarly, because a juror’s unsupervised and unauthorized site visit, in a case in which the physical characteristics of that site are critical to the case, deprives the defendant of the right to be convicted solely on the basis of evidence presented at trial, it too renders the trial process fundamentally unfair. Such an error would seem to me therefore to be structural error, whether or not one can imagine factual scenarios in which the information obtained by the juror has no impact on the jury’s deliberations or verdict.
Nor would such a holding require reversal in every case in which a juror engaged in an unauthorized site visit. The majority convincingly explains the difficulties of a holding that would lead to this result. However, the majority has created and then defeated a straw man. Sherman specifically disavows any request for such a broad ruling. Instead, he maintains that unauthorized site visits amount to constitutional error only in cases in which “significant issues were raised at trial concerning the physical aspects of the areas visited by the juror.” Brief of Appellant at 21 (quoting Commonwealth v. Price, 463 Pa. 200, 344 A.2d 493, 494 (1975)).
An analysis that results in finding constitutional error in some circumstances but not in others is unusual, but not unprecedented. Indeed, as Sherman points out, the Supreme Court has adopted a similar approach in examining court orders forbidding criminal defendants from consulting with counsel. An order preventing a defendant from consulting with counsel during an overnight recess interferes with his Sixth Amendment right to counsel, and reversal is required; the error is not examined for harmlessness and no proof of prejudice is necessary. See Geders v. United States, 425 U.S. 80, 96 S.Ct. 1330, 47 L.Ed.2d 592 (1976). In contrast, an order preventing a defendant from consulting with counsel briefly during his testimony is regarded as so de minimis as not to be constitutional error at all. Perry v. Leeke, 488 U.S. 272, 280-84, 109 S.Ct. 594, 599-602, 102 L.Ed.2d 624 (1989). Between these two extremes, there is “a line of constitutional dimension.” Id. at 278-80, 109 S.Ct. at 599-600.
In the same way, unauthorized site visits in cases in which the physical aspects of the site are not at issue should be regarded as de minimis — not constitutional error at all. This approach has much to recommend it. It protects the fundamental right involved here, yet avoids trivialization of the structural error inquiry.
Because the Supreme Court has never dealt with a case involving juror’s unauthorized site visit — let alone such a visit in a case, like this, where the physical features of the site are of critical importance, no direct precedent prohibits the above approach. However, in those cases involving what I regard as the most similar constitutional errors — unauthorized private contacts with jurors during trial — the Supreme Court seems to have applied a harmless error analysis, holding that such contacts require reversal of a conviction only if the government fails to “establish, after notice to and hearing of the defendant that such contact was harmless to 'the defendant.” Remmer v. United States, 347 U.S. 227, 229, 74 S.Ct. 450, 451, 98 L.Ed. 654 (1954) (emphasis added).3 See also Mat-*1153tox v. United States, 146 U.S. 140, 150, 13 S.Ct. 50, 53, 36 L.Ed. 917 (1897) (“Private communications, possibly prejudicial, between jurors and third persons ... are absolutely forbidden, and invalidate the verdict, at least unless their harmlessness is made to appear") (emphasis added).
Both Remmer and Mattox were decided well before 1991, when the Supreme Court first began analyzing constitutional error in terms of structural error, which is not subject to harmless error analysis, and trial error, which is. See Arizona v. Fulminante, 499 U.S. 279, 111 S.Ct. 1246, 113 L.Ed.2d 302 (1991). Indeed, Remmer and Mattox predate many of the Court’s landmark decisions regarding the constitutional protections due to criminal defendants. E.g., Edwards v. Arizona, 451 U.S. 477, 101 S.Ct. 1880, 68 L.Ed.2d 378 (1981); Miranda v. Arizona, 384 U.S. 436, 86 S.Ct. 1602, 16 L.Ed.2d 694 (1966); Escobedo v. Illinois, 378 U.S. 478, 84 S.Ct. 1758, 12 L.Ed.2d 977 (1964). Consequently, if Remmer or Mattox were decided today, the Court might well regard the errors in them as structural.
Alternatively, the Court might conclude that even if harmless error analysis applies to unauthorized contacts with jurors, a juror’s unsupervised and unauthorized visit to examine for himself the disputed physical aspects of a site presents a more fundamental problem meriting treatment as structural error. A juror’s private fact-finding mission, which results in new facts being presented to the jury without the benefit of cross-examination by (or even the knowledge of) the defendant, certainly presents more serious Confrontation Clause problems than juror contacts having nothing to do with the facts of the case. Thus, when the Supreme Court is presented with an appropriate case, it may well conclude, as my dissenting colleagues do, that the error involved here is indeed structural.
However, in view of Remmer and Mattox, the Supreme Court’s reluctance to classify errors as structural, and the decisions of this court extending Remmer to cases involving a jury’s exposure to unadmitted evidence, see United States v. Barnes, 747 F.2d 246, 251 (4th Cir.1984); see also Hinkle v. City of Clarksburg, 81 F.3d 416, 427 (4th Cir.1996) (following Barnes without citing Remmer), I believe that I am bound to treat the error here as trial error, subject to harmless error analysis. I therefore turn to that analysis.
II.
On collateral review, we may set aside a conviction only if convinced that the asserted error “ ‘had substantial and injurious effect or influence in determining the jury’s verdict.’ ” Brecht v. Abrahamson, 507 U.S. 619, 623, 113 S.Ct. 1710, 1714, 123 L.Ed.2d 353 (1993) (quoting Kotteakos v. United States, 328 U.S. 750, 776, 66 S.Ct. 1239, 1253, 90 L.Ed. 1557 (1946)).4 If we find the issue so close as to leave us “in virtual equipoise as to the harmlessness of the error,” we must resolve the question in favor of the petitioner and grant habeas relief. O’Neal v. McAninch, — U.S.-,-, 115 S.Ct. 992, 994, 130 L.Ed.2d 947 (1995).
In the instant case, the state trial court held a hearing on Sherman’s motion for a new trial, and juror Miller was called to testify on Sherman’s behalf. On direct examination, the juror testified that two or three days into the trial, he and his wife drove to Gibson Manor, “looking for [the] tree that was so involved in the case.” He stated that he did, in fact, find the tree as well as the house in which the murders occurred. Apparently wary of violating the secrecy of the jury’s deliberations, and consistent with the State’s objections, the court did not permit Sherman’s counsel to ask the juror why he visited the site.
*1154On cross-examination by the State, the juror testified that by the time he visited the site, he had already seen (in court) the videotape of the outside of the Sherman house, but was not sure if he had yet seen the aerial photographs of the neighborhood. When asked whether, at the time of his site visit, several photographs of the tree had been introduced, the juror responded, “Yes, the tree had been in question, and that’s one of the reasons I went there.” Finally, the State asked the juror whether other photographs of the scene had been introduced by the time he made his visit, to which he responded, “I’m not sure, sir. I’m not sure what sequence — the reason why I went there was so I could see the tree that was so much in question.” The State made no further inquiry into the facts and circumstances surrounding the juror’s site visit.
All told, the record regarding the asserted error provides the following information:
(1) Two or three days into the trial, the juror drove to the neighborhood in which the murders occurred, looking for the tree in which the murder weapon was found.
(2) The juror saw the Sherman house and the tree.
(3) This visit occurred after a videotape of the outside of the Sherman house was .played for the jury, and after the jury saw several photographs of the tree.
(4) The juror’s stated purpose for this site visit was to “see the tree that was so much in question.”
To this day, no details of the juror’s site visit are known. We do not know, nor did the state trial court know, for example, the length of the juror’s site visit; how close he came to the tree; whether he got out of his car and walked to the tree or simply drove past; whether he performed “experiments” based on the testimony about the tree or merely looked at it to ascertain its size and location. Despite the sparsity of the record, the state trial court determined that Sherman suffered no prejudice as a result of the juror’s site visit.5 I cannot agree.
Based on the extremely limited information before us regarding the juror’s site visit, it is simply impossible to ascertain whether and to what extent Sherman was prejudiced by the visit. The juror’s testimony is consistent with a visit in which he simply drove through Gibson Manor and looked at the house and the tree as he drove by. Such a visit, if properly subject to harmless error analysis, might well be found to be harmless.6
However, the juror’s testimony would be equally consistent with a scenario in which, dissatisfied with the evidence presented at trial, he set out to conduct his own investigation, including taking measurements of the tree and its distance from the house and attempting to hide items in the tree in the manner in which the murder weapon was found. Such a visit would certainly be prejudicial. Contrary to the majority’s suggestion, Maj. Op. at 1142, it would not be merely “cumulative” of the other evidence at trial, as the proper interpretation of that evidence was disputed. For example, perhaps confused by the parties’ conflicting interpretations of Officer Hopkins’ testimony, the juror sought to resolve for himself whether someone of Sherman’s height and build could have wedged the gun in the tree in the manner in which it was found. Alternatively, perhaps the juror wanted to test whether it was possible to hide something in the tree without getting sap and pine needles on one’s clothing, or wanted to see whether it made sense for Sherman to have stopped at the tree en route to his grandparents’ house. Each of these scenarios would result in the discovery of new, rather than cumulative, information.
The State of Maryland seems to concede, as it must, that the record in this case with respect to the juror’s site visit leaves much to *1155be desired. The State argues, however, that “Sherman had the burden to show that an error warranting new trial relief had occurred,” Brief of Appellees at 28, and thus should bear the consequences of the insufficient record. This argument is foreclosed by Remmer, which mandates that any private communication with a juror about the matter pending before the jury is “presumptively prejudicial,” and the government bears the heavy burden of rebutting that presumption. 347 U.S. at 229, 74 S.Ct. at 451. Moreover, we have recognized as recently as a few months ago, in a ease that the majority neither overrules nor acknowledges, that this presumption governs when jurors consider evidence not admitted at trial. See Hinkle, 81 F.3d at 427. See also United States v. Brooks, 957 F.2d 1138, 1142 (4th Cir.), cert. denied, 505 U.S. 1228, 112 S.Ct. 3051, 120 L.Ed.2d 917 (1992); United States v. Greene, 834 F.2d 86, 88 (4th Cir.1987); Barnes, 747 F.2d at 250-51. Accordingly, while Sherman bore the initial burden of proving that the site visit occurred,7 once that fact was established, the burden shifted to the State to “demonstrate] the absence of prejudice.” Stockton, 852 F.2d at 743.
Concededly, due to the significant restrictions the rules of evidence place on questioning of jurors, it would have been difficult, if not impossible, for the State to prove by direct evidence that the juror’s site visit did not prejudice Sherman. We have suggested that “the state may rebut the presumption of prejudice through whatever circumstantial evidence is available, including juror testimony on the facts and circumstances surrounding the extraneous communication [or receipt of unadmitted evidence].” Id. at 744. Ultimately, however, because “[t]he right to an impartial jury belongs to the defendant,” the risk of being unable to prove the impact of an improper jury contact or a jury’s exposure to unadmitted evidence is properly borne by the State. Id. at 743-44. In this case, the State simply failed to meet its burden.
The only circumstantial evidence the State presented to meet its heavy burden was the juror’s testimony that at the time he made the visit, the jury had already seen a videotape and some photographs of the area in question. While the State argues (and a majority of this court finds) that the site visit was merely cumulative of the other evidence presented at trial regarding the tree, the. juror’s testimony in response to the State’s questions indicates otherwise. When asked whether he had seen the photographs of the tree prior to his visit, the juror responded that that was why he went to the site— because he wanted to see the tree “in question.” This implies a causal relationship between the photographs and the visit; the juror was not satisfied with the photographs and visited the site to obtain additional information about the tree. As my dissenting colleagues note, the district court, which nonetheless found the site visit to be harmless error, remarked that the state’s close-up photograph of the tree “does not show me anything on its face.”
The majority makes much of the “powerful array of evidence presented at trial” that supports the jury’s guilty verdict. This misconceives the appropriate harmless error inquiry, which focuses not on the sufficiency of the evidence absent the error, but rather on the impact of the error on the jury’s verdict. See Sullivan, 508 U.S. at 279, 113 S.Ct. at 2081 (“Harmless error review looks ... to the basis on which the jury actually rested its verdict .... not [to] whether, in a trial that occurred without the error, a guilty verdict would surely have been rendered.”) (internal citation and quotation omitted). Accordingly, the majority’s characterization of Sherman’s argument as merely a “round*1156about” attack on the sufficiency of the evidence against him is off the mark. Sherman does not challenge the sufficiency of the evidence presented at trial, he challenges the fairness of the trial and “the basis on which the jury actually rested its verdict.” Id.
In summary, the record in this case reveals that a juror in Sherman’s capital murder trial made an unsupervised and unauthorized visit to a site whose physical characteristics were disputed and critical to the State’s case. Unfortunately, the record does not reveal much else about that visit. At the post-trial hearing, the burden was on the State to rebut the presumption of prejudice raised by the juror’s site visit, and the State must therefore bear the consequences of the sparsity of the record. Given thé lack of detail as to the juror’s site visit and the wide range of conduct that his testimony might describe, the majority’s conclusion that the site visit was harmless is tantamount to a conclusion that a juror’s unauthorized site visit can never be prejudicial error. I cannot agree with this conclusion. While some of the possibilities encompassed by the juror’s ambiguous testimony might be harmless, many of them would certainly be prejudicial.
Without any evidence as to the details of the juror’s site visit, I find myself “in virtual equipoise as to the harmlessness of the error.” O’Neal, — U.S. at-, 115 S.Ct. at 992. My “grave doubt as to the harmlessness of [the] error” requires me to resolve the issue in Sherman’s favor. Id. at-, 115 S.Ct. at 999. The “obviousness” of Sherman’s guilt to a reviewing court, or to the public for that matter, is irrelevant absent a proper jury verdict. I would, therefore, grant the writ of habeas corpus, so that Sherman could receive a new trial in which the jury returns a proper verdict, based solely on the evidence presented at trial. The Constitution requires nothing less.
. Significantly, as noted above, the deprivation of a defendant's right to a public trial or to counsel is considered structural error. The error in this case deprived Sherman of the remaining three components the Court recognized in Turner as essential to the fundamental right to trial by jury. Surely this error should also be considered structural.
. The majority characterizes the instant case as "virtually indistinguishable” from both Remmer and Smith v. Phillips, 455 U.S. 209, 102 S.Ct. 940, 71 L.Ed.2d 78 (1982). I agree that the error involved in this case is very similar to the error in Remmer. However, as this court recognized in Stockton v. Virginia, 852 F.2d 740, 744 (4th Cir.1988), cert. denied, 489 U.S. 1071, 109 S.Ct. 1354, 103 L.Ed.2d 822 (1989), Phillips involves clearly distinguishable concerns. As we explained in Stockton, when, as in Phillips, "some external manifestation of a juror's predisposition subsequently calls the juror’s impartiality into question, the defendant is afforded the opportunity to establish the juror’s actual bias.” 852 F.2d at 744 (emphasis added). On the other *1153hand, where, as in the case at hand and Remmer, "the danger is not one of juror impairment or predisposition, but rather the effect of an extraneous communication upon the deliberative process of the jury, the defendant's right to an impartial jury requires that the government bear the burden of establishing the nonprejudicial character of the contact.” Id. (emphasis added).
. Like the majority, I believe that our decision in Smith v. Dixon, 14 F.3d 956 (4th Cir.) (en banc), cert. denied, - U.S. -, 115 S.Ct. 129, 130 L.Ed.2d 72 (1994), established that the Brecht standard applies on collateral review, even when the state court has failed to find an error harmless beyond a reasonable doubt.
. In my view, the Antiterrorism and Effective Death Penalty Act of 1996, Pub.L. 104-132, 110 Stat. 1214 (1996), does not apply retroactively to cases like the one at hand. See Landgraf v. USI Film Prods., 511 U.S. 244,-, 114 S.Ct. 1483, 1505, 128 L.Ed.2d 229 (1994).
. As noted above, this does not preclude characterization of that error as structural. The harmless error inquiry focuses on the impact of an error on the verdict rendered. Structural error is concerned with the integrity and fairness of the trial process, not the impact of a particular error on the actual verdict.
. Remmer requires that the improper jury contact be "about the matter pending before the jury.” 347 U.S. at 229, 74 S.Ct. at 451. Stockton likewise requires that the defendant establish that the contact “was of such a character as to reasonably draw into question the integrity of the verdict.” 852 F.2d at 743. Our cases applying the Remmer presumption where the jury has been exposed to evidence not admitted at trial suggest that, in such cases, the defendant need only establish that the jury has, in fact, been exposed to the unadmitted evidence to trigger the presumption. See, e.g., Hinkle, 81 F.3d at 427. In any event, in this case, by establishing that a juror had engaged in an unauthorized site visit and that the physical characteristics of that site were of critical importance, Sherman satisfied the additional requirements described in Remmer and Stockton.