Opinion by Judge KLEINFELD; Partial Concurrence and Partial Dissent by Judge BREWSTER.
KLEINFELD, Circuit Judge:We construe the grouping provisions of the sentencing guidelines, U.S.S.G. §§ 3D1.1, 3D1.4, to determine whether the proper base offense level was selected and whether there was impermissible double counting.
Facts
Calozza sold insurance for the Sons of Norway to its members. He fabricated a letter purporting to be from the Sons of Norway to staff. The letter offered staff members, but not their clients, an opportunity for a tax-advantaged high-yield secure investment. Then Calozza showed the fake letter to his insurance clients, told them it did not seem fair to deprive them of this opportunity, and offered to help them get into it. His scheme was that his clients would borrow against their insurance and give him their money in exchange for his personal high-interest promissory note. He would supposedly invest the money in the staff opportunity and pass the returns through to his clients, the Sons of Norway members whose money he had taken. Because the fake letter made it look as though the Sons of Norway sought to limit the nonexistent investment to staff, Calozza advised his clients to keep his pass-through scheme and their participation secret.
What Calozza was really doing was taking his clients’ money, and paying off the earlier investors with the later investors’ money. It was a Ponzi scheme. Calozza stole about $8.8 million this way, and paid back about $2.3 million to the more fortunate “investors.” He used the profits to pay off his gambling debts and to build his $1.6 million, 6700 square foot house with full gymnasium, swimming pool and tennis court.
After Calozza got caught, he was very cooperative in the criminal process and a related civil action by his victims. He pleaded guilty to eleven counts of mail, wire and *689bank fraud under 18 U.S.C. §§ 1341, 1343, 1344, transporting stolen money under 18 U.S.C. § 2314, and money laundering to conceal the scheme under 18 U.S.C. §§ 1956, 1957. He was sentenced November 27,1995.
The district court imposed a vulnerable victim adjustment under U.S.S.G. § 3A1.1, because most of the victims were retired and elderly. It also imposed an abuse of position of trust enhancement under U.S.S.G. § 3B1.3, because Calozza had known many of his victims for years or decades and had become close to their families as a representative of their fraternal order.
Here is how the district court computed the sentence:
Money Laundering
[[Image here]]
Guideline range — 121 to 151 months.
Sentence imposed. — 121 months.
Analysis
Calozza claims that there was double counting of the vulnerable victim and abuse of position of trust adjustments. We disagree with his argument in one respect, and agree in another. We conclude that the district court did not err in applying the adjustments to both groups when it selected which group was to be the base for the sentence computation. The guidelines enhance the most serious group’s level with an adjustment for equally or less serious groups, and the more serious the other groups, the longer the sentence. Counting the same adjustments in the enhancing group and the base group does, as Calozza argues, mistakenly punish him twice for the same conduct.
Calozza also argues that the district court mistakenly concluded that it could not grant him a downward departure, an argument we reject. We need not decide whether grouping the fraud and money laundering counts separately, cf. United States v. Lopez, 104 F.3d 1149 (9th Cir.1997); United States v. Edmonds, 103 F.3d 822, 826 (9th Cir.1996), or imposing adjustments for vulnerable victim and abuse of position of trust, were appropriate, because those decisions are not challenged in this appeal.
Calozza’s sentencing was pursuant to the Guidelines Manual incorporating ' amendments effective November 1,1995.
A. Selecting the highest offense level.
The combined offense level calculation starts with a base, “the offense level applicable to the Group with the highest offense level.” U.S.S.G. § 3D1.4. That base is increased by units from other groups with the same or fewer points. Id. This is analogous to starting with the most serious crime, and then looking to the equally or less serious crimes to enhance the sentence.
Calozza argues that the district court should have selected the base by applying the vulnerable victim and abuse of position of trust adjustments only to the fraud group, not the money laundering group. That would have made the fraud group the one with the highest offense level, so it would be the base. Calozza’s proposed sentence calculation would have worked like this:
Fraud
[[Image here]]
Guideline range — 87—108 months.
The principles in the grouping guidelines are that (1) counts involving substantially the same harm are grouped together, to prevent prosecutors from enhancing sentences by multiplying charges for substantially the same harm; (2) the group with the highest offense level, that is, the worst crimes, furnishes a base; (3) additional crimes add units to the base, so that punish*690ment will be greater for those who commit more crimes.
The rules in this Part seek to provide incremental punishment for significant additional criminal conduct. The most serious offense is used as the starting point. The other counts determine how much to increase the offense level.
In order to limit the significance of the formal charging decision and to prevent multiple punishment for substantially identical offense conduct, this Part provides rules for grouping offenses together. Convictions on multiple counts do not result in a sentence enhancement unless they represent additional conduct that is not otherwise accounted for by the guidelines. In essence, counts that are grouped together are treated as constituting a single offense for purposes of the guidelines.
U.S.S.G. Ch. 3, Pt. D, intro, comment. The Sentencing Commission wrote its grouping provisions “with an eye toward eliminating unfair treatment that might result from count manipulation.” U.S.S.G. Ch. 1, Pt. A, § 4(a).
The rule for choosing which group furnishes the base is to use “the offense level applicable to the Group with the highest offense level.” U.S.S.G. § 3D1.4. If adjustments such as vulnerable victim and abuse of position of trust apply to all groups, it does not matter whether they are added to all the alternatives, or not added to any of them, in making this comparison. If x is greater than y, then x plus 4 is greater than y plus 4. But if the adjustments could properly be added only to one and not the other, then it could matter a great deal. It may be that x is greater than y, but less than y plus 4. For the judge to have erred in a way that affected Calozza’s sentence, Calozza would have to show that the adjustments were only appropriate for fraud, not for money laundering.
In that contention, he is incorrect. The adjustments could properly be applied to both groups to determine which had the highest offense level. Defrauding victims of more than $5 million yields a base offense level of 20, raised to 26 for “more than minimal planning” and “affected a financial institution.” U.S.S.G. §§ 2Fl.l(b)(l)(0), (b)(2)(A), and (b)(6)(B). Money laundering more than $2 million (that this is the correct amount for the money laundering computation has not put at issue by the parties) yields a base offense level of 29. U.S.S.G. § 2S1.1. If the four level adjustment for vulnerable victim and abuse of position of trust can properly be added to both, yielding 30 and 33, money laundering remains the group with the highest level.
Calozza argues that fraud rather than money laundering should have been used as the base, because the vulnerable victim and abuse of position of trust adjustments should have been considered only with respect to fraud. The reason is that the elderly and infirm Sons of Norway members who were the vulnerable victims, and his abuse of trust relative to them, do not relate to money laundering, a crime in which the victim is the government. This argument would give the fraud group an offense level of 30, but leave money laundering at 29.
Our precedents require us to reject Calozza’s argument. In United States v. Duran, 15 F.3d 131, 134 (9th Cir.1994), a deputy sheriff was accused of stealing money seized from drug dealers and structuring financial transactions, but convicted only of structuring. The question on appeal was whether the structuring sentence could be enhanced for abuse of a position of trust. The crooked deputy sheriff had abused his position of trust only with respect to theft, of which he had not been convicted; his law enforcement position had no bearing on his ability to structure financial transactions. We held that the enhancement applied nevertheless, because the district court could properly consider “relevant conduct other than that involved in the offense of conviction,” as defined in U.S.S.G. § lB1.3(a). Id. at 133.
In United States v. Haggard, 41 F.3d 1320 (9th Cir.1994), the criminal had falsely claimed to know the whereabouts of a kidnapped child, and led the FBI on a wild goose chase. He was sentenced for obstructing justice, a crime against the government, but his sentence was enhanced by a vulnera*691ble victim adjustment, treating the child’s mother rather than the government as the victim. We held that even though the federal government, not the mother, was the victim of the crime Haggard was convicted of, the district court “properly may look beyond the four corners of the charge” and may consider “all harm” caused by relevant conduct under U.S.S.G. § lB1.3(a)(3).
Under Duran and Haggard, the test of whether such adjustments as vulnerable victim and abuse of position of trust can be applied is whether they are “relevant conduct” as defined by the guidelines, not whether they apply to the victim or the criminars position with respect to the crime to which the adjustments are being applied. The district court properly applied the adjustments to Calozza’s money laundering group because his abuse of a position of trust with respect to his aged and infirm clients was “relevant conduct” under U.S.S.G. § 1B1.3 with respect to Calozza’s money laundering. Relevant conduct is “all acts ... that occurred during the commission of the offense of conviction, in preparation for that offense, or in the course of attempting to avoid detection or responsibility for that offense.” U.S.S.G. § lB1.3(a).
Calozza argues that the guidelines limit relevant conduct for grouped offenses to those relevant to the crime of conviction, which would be money laundering:
... the base offense level ... and ... adjustments ... shall be determined ... (2) solely with respect to offenses of a character for which section 3D1.2(d) would require grouping of multiple counts, all acts and omissions ... that were part of the same course of conduct or common scheme or plan as the offense of conviction.
U.S.S.G. § lB1.3(a). This provision enhances a sentence if a drug dealer sells drugs on multiple occasions as part of a common scheme, but is convicted of only one of the sales. Id. at applic. note 3. It protects a defendant against enhancement for drug sales not part of the same course of conduct or common scheme or plan. It does not protect against enhancement of a group for acts which were part of the same course of conduct as the offense of conviction, but were not part of that offense. Calozza’s abuse of his position of trust harming vulnerable Sons of Norway members was part of the same course of conduct as his money laundering, so the limitation does not protect him against the adjustments.
Calozza’s argument implies an absurd result. Suppose a cocaine dealer, trying to avoid arrest, shoots at a police officer. Calozza’s argument would imply that because his drug crimes are groupable, his sentence cannot be enhanced for shooting at the officer. A straightforward reading of the quoted guideline section implies the contrary, and so does common sense. This does not turn the “solely ... offense of conviction” language into surplusage. It would protect the cocaine dealer from enhancement for a marijuana sale which was not part of the course of conduct leading to his conviction.
B. Double counting.
Calozza also argues that the district court’s computation amounted to double counting of the vulnerable victim and abuse of position of trust adjustments. Impermissible double counting of an enhancement occurs if a guideline provision is used to increase punishment on account of a kind of harm already fully accounted for, though not when the same course of conduct results in two different types of harm or wrongs at two different times. United States v. Alexander, 48 F.3d 1477, 1492-93 (9th Cir.1995). Thus where multiple victims were already accounted for in a grouped fraud offense, the vulnerable victim adjustment could be applied only once for all, not once for each victim (two levels total, not two levels for each victim). United States v. Caterino, 957 F.2d 681 (9th Cir.1992).
The Fifth Circuit has held that impermissible double counting occurred where an official victim adjustment was applied both to assault and to grouped marijuana counts. United States v. Kleinebreil, 966 F.2d 945, 954-55 (5th Cir.1992). We are persuaded that Calozza is correct in his double counting argument, and our decisions in Alexander and Caterino are consistent with Kleinebreil.
*692The grouping scheme protects against prosecutorial sentence manipulation, by treating together multiple charges involving substantially the same harm. Together with the relevant conduct provisions, the grouping scheme protects the public from sentences inadequate to reflect different kinds of harm by adding levels for counts not involving substantially the same harm based on a system of “units.” U.S.S.G. § 3D1.2, 3D1.4. One unit is assigned to the most serious group’s total for a group not involving substantially the same harm (fraud as opposed to money laundering in this case) if the additional group is equally serious or one to four levels less serious. U.S.S.G. § 3D1.4(a). Only one-half unit is assigned for groups five to eight levels less serious. U.S.S.G. § 3D1.4(b). The units translate into additional offense levels. U.S.S.G. § 3D1.4. Once the vulnerable victim and abuse of position of trust enhancements were added to the money laundering group, the offense level was 33. The fraud group was 26 without these adjustments, 30 with them. Thus, the fraud group was three levels less serious with the adjustments, but seven less serious without them. If adding the adjustments to the fraud group was a mistake, then Calozza’s sentence computation should have been 108 to 135 months instead of 121-151.
Calozza’s abuse of his position of trust and the vulnerability of his victims was “a kind of harm ... already fully accounted for,” Alexander, 48 F.3d at 1492, in the money laundering group. It seems a bit odd to add adjustments for these offense characteristics only to the money laundering group, where they have no direct bearing, and not the fraud group, to which they apply more directly. But we have explained above why they were properly treated as relevant conduct for the money laundering group under our precedents. The point we are considering now is whether the same factors which enhanced the money laundering counts should also enhance the fraud counts. They should not, because the wrong and its victims were the same. Calozza abused his position of trust, harming vulnerable victims, in his fraudulent conduct, not his money laundering, but because these aggravating factors were relevant conduct, they enhanced the money laundering sentence.
When the abuse of trust and vulnerable victim adjustments were added to both groups for purposes of selecting which yielded the “highest offense level” and should be the base, under U.S.S.G. § 3D1.4, the only effect was to reject the fraud group for that purpose and to select the money laundering group. They did not enhance the fraud group’s effect on the sentence, because that group’s computation was not used as the base. But now that units are being added for the fraud group, under U.S.S.G. § 3D1.4(a) and (b), the effect is to add prison time twice for exactly the same abuse of trust and vulnerable victims.1
*693Arguably, the fraud and money laundering offenses should have been grouped together. See United States v. Rose, 20 F.3d 367, 371-72 (9th Cir.1994); United States v. Mullens, 65 F.3d 1560, 1564-65 (11th Cir.1995); United States v. Leonard, 61 F.3d 1181, 1186-87 (5th Cir.1995). Had the fraud and money laundering counts been grouped together, the double counting issues raised in this appeal would have disappeared. Because both sides contended before the district court that they should not be grouped, we express no opinion as to whether the crimes should have been grouped.
The guideline ranges for the correct and the mistaken sentence computations overlapped — 108 to 135 months and 121 to 151. While the district judge could still have given Calozza the same 121-month sentence, we cannot tell whether he would have. The judge’s remarks at sentencing suggest the possibility that he might have exercised his discretion to sentence Calozza at the bottom of the guideline range, whatever it turned out to be. We do not know, therefore, whether the computation error was harmless — it might have added 13 months to Calozza’s sentence. To enable the district judge to exercise his discretion after correcting the unit calculation, we remand for resentencing.
C. Downward Departure.
Calozza argues that the district court abused its discretion in not departing downward for “extraordinary acceptance of responsibility.” The court found that the degree of cooperation was “very profound and valuable” and had “aided the prosecution greatly.”
The district court’s discretionary refusal to depart downward from the Guidelines is not reviewable on appeal. United States v. Bauer, 84 F.3d 1549, 1562 (9th Cir.1996). Calozza argues that the district judge in this case mistakenly concluded that the law prohibited departure for extraordinary acceptance of responsibility where restitution was not made, and suggests that this amounts to discrimination against the indigent, since Calozza was unlikely ever to have $8.8 million with which to repay his victims. We can review for abuse of discretion, under Koon v. United States, — U.S. —, 116 S.Ct. 2035, 135 L.Ed.2d 392 (1996), where the district court “indicates that it believes that it lacks the authority to depart.” United States v. Eyler, 67 F.3d 1386, 1390 n. 5 (9th Cir.1995).
We have studied the sentencing transcript, and conclude that the judge made a discretionary decision not to depart, not a legal decision that he could not depart. The judge said “[t]he guidelines permit departures from the guideline range in unusual cases for particular reasons, and one reason that can apply is extraordinary acceptance of responsibility.” The judge then decided that such a departure should not be made “in this case.” He thought using the bottom end of the 121-151 range was a sufficient benefit for Calozza’s conduct after he was caught.
Defense counsel has argued, very capably, that there should be a downward departure in this case based on exceptional acceptance of responsibility. The guidelines permit downward departures from the guideline range in unusual cases for particular reasons, and one reason that can apply is extraordinary acceptance of responsibility.
I do not believe that adjustment should be made in this case. I do not believe there should be a downward departure, in other words, in this case.
Mr. Calozza has done a full job of accepting responsibility and has been given the three-point credit for that. He has not been in a position to make restitution himself, and I do not think that the require-*694merits of that downward departure are met in this instance.
The judge was not required to treat restitution as though it had been made, just because Calozza had stolen so much that he could not repay it even using his best efforts.
VACATED and REMANDED.
. We have carefully considered the points raised by our dissenting colleague, but cannot agree. The dissent says our opinion "overrules Haggard and Duran by deleting relevant conduct enhancements.” That accusation is not well founded.
The point in the dissent which has given us pause is that we have failed to follow the direction in application instruction (d) of U.S.S.G. § 1B1.1, that "[i]f there are multiple counts of conviction, repeat steps (a) through (c) for each count.” Step (c) says to apply adjustments such as the vulnerable victim and abuse of position of trust adjustments at issue in this case. The dissent says that our holding prohibits the district judge from doing what IB 1.1(d) requires.
The reason we reject our colleague’s position is that application of U.S.S.G. § lBl.l(d) as he construes it is inconsistent with circuit precedent. United States v. Alexander, 48 F.3d 1477, 1492 (9th Cir.1995) says that constitutionally impermissible double counting occurs only when a part of the guidelines "is applied to increase a defendant’s punishment on account of a kind of harm that has already been fully accounted for by application of another part of the Guidelines.” 48 F.3d at 1492. That is what happened here. Calozza abused one position of trust, to take advantage of one group of vulnerable victims one way. The enhancements for those aggravating factors were applied to his money laundering counts, even though the enhancements applied to his fraud rather than his money laundering conduct. That was proper under Haggard and Duran. Once the enhancements were applied to the money laundering counts, Calozza’s punishment on account of the kind of harm he did by abusing a position of trust to cheat vulnerable victims out of their money was fully accounted for. The dissent draws the erroneous inference that if so, then fraud and money laundering cannot both take account of a higher amount of money than the base level. The inference is incorrect because in that example, the amount affects the *693crime twice, first when the person steals the money, and again when he hides it. By contrast, the vulnerability of the victims and abuse of position of trust only affected the crime once, when Calozza defrauded his victims. The Eleventh Circuit authority the dissent cites, United States v. Wimbush, 103 F.3d 968 (11th Cir.1997), and United States v. Wyckoff, 918 F.2d 925 (11th Cir.1990), are inapposite. Both hold that, in a felon in possession of a firearm case, it is permissible to count the same felony both as a predicate for the crime and also as part of the criminal history, because these uses "embody distinctly separate notions relating to sentencing.” Wyckoff, 918 F.2d at 927. The enhancements in the case at bar do not.