Irving Cohen appeals the district court’s dismissal of Cohen’s malicious prosecution complaint on the basis of res judicata. We reverse.
I.
Eight clients of the law firm Lupo & Stemmler (collectively, Lupo & Stemmler) unsuccessfully sued Cohen and more than thirty other defendants on unmeritorious allegations of securities fraud in Bastien v. R. Rowland & Co., 631 F.Supp. 1554 (E.D.Mo.1986) (Bastien I), aff'd, 815 F.2d 713 (8th Cir.1987), cert. denied, 484 U.S. 854, 108 S.Ct. 160, 98 L.Ed.2d 115 (1987). Lupo & Stemmler’s complaint against Cohen arose out of tax shelter investments in motion pictures that proved unprofitable. After four years of discovery, no probative evidence supported Lupo & Stemmler’s allegations. The district court granted summary judgment in favor of Cohen. Cohen then moved for sanctions against Lupo & Stemmier under Federal Rule of Civil Procedure 11.
In its Rule 11 decision, the court found that the Bastien I litigation had been conducted “in a manner that escalated costs unnecessarily and vexatiously.” Bastien v. R. Rowland & Co., 116 F.R.D. 619, 621 (E.D.Mo.1987) (Bastien II), aff'd sub nom. Lupo v. R. Rowland & Co., 857 F.2d 482 (8th Cir.1988), cert. denied, 490 U.S. 1081, 109 S.Ct. 2101, 104 L.Ed.2d 662 (1989). Thus, the court awarded $100,000 under Rule 11 as a sanction against Lupo & Stemmler's bad faith conduct, $50,000 of which was assessed against the attorneys and $50,000 against the eight plaintiffs.
Following the decision in Bastien II, Cohen filed this suit based upon the common law tort of malicious prosecution to recoup the extensive fees and costs of nearly one million dollars incurred in defending the Bastien litigation. The district court dismissed the complaint on the ground that imposition of Rule 11 sanctions against Lupo & Stemmier in Bastien II barred Cohen’s malicious prosecution claim on res judicata principles.
*365II.
Res judicata precludes relitigation of the same claim in a subsequent proceeding. For the doctrine to apply, the prior judgment must be rendered by a court of competent jurisdiction, it must be a final judgment on the merits, and both cases must have involved the same cause of action and the same parties. Headley v. Bacon, 828 F.2d 1272, 1274 (8th Cir.1987). Res judica-ta bars claims which were actually litigated or could properly have been raised and determined in the prior proceeding. Poe v. John Deere Co., 695 F.2d 1103, 1105 (8th Cir.1982).
We conclude that Bastien II does not have res judicata effect in this case because Cohen’s malicious prosecution cause of action does not raise the same claim that Bastien II resolved. Cohen’s malicious prosecution claim was not actually litigated and could not properly been raised and determined in Bastien II. Moreover, Rule 11 and the tort of malicious prosecution differ in their nature, the elements of the claims, and the potential remedies.
The district court did not decide whether the Bastien I complaint was filed without probable cause, whether Lupo & Stemmier acted with malice, or the amount of damages Cohen suffered as a result of Lupo & Stemmler’s misconduct. Those inquiries are irrelevant under Rule 11, but are the sum and substance of the tort of malicious prosecution. The nucleus of operative fact necessary to maintain an action for malicious prosecution includes the conclusion of the underlying action. Whether Cohen would be victorious in the Bastien litigation could not have been determined until the conclusion of that case.
TTA/taral Rule nf P.ivil PrnCPrlnrA 11 grants a court discretion to discipline parties and counsel for conducting litigation in bad faith or in a frivolous and abusive fashion. Rule 11 is a procedural tool that under the Rules Enabling Act can not “abridge, enlarge or modify any substantive right.” 28 U.S.C. § 2072. Rule 11 sanctions must be sought by motion in a pending case; there can be no independent cause of action instituted for Rule 11 sanctions. Port Drum Co. v. Umphrey, 852 F.2d 148, 151 (5th Cir.1988).
On the other hand, the common law tort of malicious prosecution is a claim in its own right under applicable state law. In Missouri, malicious prosecution requires proof that: (1) the underlying offending lawsuit was initiated or maintained without probable cause; (2) the plaintiff in the initial suit acted with malice; (3) the proceeding terminated in favor of the defendant; and (4) the defendant suffered damages as a result of the suit. Stafford v. Muster, 582 S.W.2d 670, 675 (Mo.1979). Rule 11 can not abridge the substantive state law of malicious prosecution, nor was it adopted to serve as a surrogate for an action based upon a claim of malicious prosecution resulting from frivolous, harassing, or vexatious litigation.
The district court recognized that the $100,000 award under Rule 11 represented merely “a fraction of the fees and costs incurred by [Cohen].” Lupo & Stemmier subjected Cohen to years of vexatious litigation and induced costs far exceeding that of the original investment. Cohen should have the chance to prove that Bastien I was conducted in such bad faith that it constituted malicious prosecution and, if successful in so proving, be compensated for his loss.
The district court’s order dismissing Cohen’s complaint is reversed, and the case is remanded for further proceedings consistent with this opinion.