Editor’s note: This post is part of our series for cybersecurity professionals and hobbyists, written by Harsh S., a penetration tester on ExpressVPN’s cybersecurity team.
Companies are increasingly relying on the Software-as-a-Service (SaaS) model for various administrative services, thanks to its ease of access, delivery, and maintenance. This has resulted in an increased reliance on the security of third-party SaaS vendors to ensure the strong security posture of an organization. This makes it important for organizations to collaborate with their industry partners on security matters.
ExpressVPN relies on one such third-party SaaS application, which we’ll call “RecruitmentApp” in this article, to provide recruitment-related services. Supply-chain security, however, goes much deeper than the security of our third-party vendors alone. Most of these vendors also have their own third-party vendors that they rely on. For the purpose of this article, we’ll call these fourth-party vendors. RecruitmentApp relies on the Mailgun platform to send emails on behalf of ExpressVPN. This makes Mailgun ExpressVPN’s fourth-party vendor.
In this article, we outline our investigation into a vulnerability that could have been exploited by signing up on Mailgun and then abusing the intended inheritance of TXT (SPF/DKIM) and MX records from mailgun.org to hijack ExpressVPN’s email subdomain.
Overview
Earlier this year, ExpressVPN’s Security Team received a report via our Bug Bounty Program outlining a security misconfiguration affecting Mailgun. This bug was presented to ExpressVPN as a subdomain takeover and identity-impersonation vulnerability that could be abused by malicious actors to send emails through the hijacked ExpressVPN subdomain via Mailgun.
In order to use Mailgun’s email service, RecruitmentApp registered the domain app-mail.customerdomain.com and set up the required MX and CNAME records to send emails using Mailgun. However, Mailgun also created a subdomain for extended analytics, email.app-mail.customerdomain.com, for which the DNS records were not claimed by RecruitmentApp. RecruitmentApp was not instructed to claim these records during the setup process; however, this would not have resolved the issue, as other subdomains would also likely have been susceptible to this issue.
The domains can be summarized as follows:
- Top-level organization domain: expressvpn.com
- RecruitmentApp email integration subdomain: app-mail.expressvpn.com
- Extended email analytics subdomain: email.app-mail.expressvpn.com
The subdomain for extended analytics was hence available for registration on Mailgun by other user accounts and could be used by other Mailgun users to send emails.
After remediating the vulnerability, the Mailgun team reviewed their application logs and did not identify any malicious attempts to exploit this vulnerability for ExpressVPN and other Mailgun customers. All domains that were hijacked by security researchers as proofs of concept were unregistered from the system.
The vulnerability
Background
Before we dive into the specifics of this bug, it is important to understand the three important email authentication standards, namely, SPF, DKIM, and DMARC. These standards can be used to provide authentication and integrity to the sender’s mail server by proving that the sender is indeed authorized to send emails using the specified domain.
SPF stands for “Sender Policy Framework,” and it is stored as a TXT record in the organization’s DNS records. SPF records can be used to specify the IP addresses and/or servers that are authorized to send emails for the domain. In the context of this vulnerability, we will be looking at an SPF record that appears as follows:
v=spf1 include:anotherdomain.com -all
The “v” tag describes the version of SPF used, and the “include” tag references the policy of the domain “anotherdomain.com”. This means if the domain’s (“anotherdomain.com”) policy passes the SPF check, the email will pass the SPF check, too. The “-all” tag describes the action to be performed if any of the defined tag values don’t match the actual values. It can be either defined as a hard-fail (“-all”), where all the emails are immediately rejected upon SPF failure, or a soft-fail (“~all”), where the emails will be accepted but marked.
DKIM stands for DomainKeys Identified Mail. DKIM works by digitally signing the email message sent by the organization and storing the public key in the organization’s DNS record, for verification. Thus, the digital signature in the email message can be validated by using the public key located in the organization’s DNS record.
Lastly, Domain-based Message Authentication, Reporting, and Conformance (DMARC) is also stored in the organization’s DNS record. DMARC can be configured to use both DKIM and SPF records to validate an email. It allows a sender to indicate that their messages are protected by SPF and/or DKIM standards.
To implement this, the sender specifies the actions within the DMARC record for the receiving mail server if neither of these authentication or integrity standards pass the checks. A sample DMARC record with three tag-value pairs can be configured as follows:
v=DMARC1; p=none; rua=mailto:dmarc@yourdomain.com
The “v” tag shows the version of DMARC used, the “p” tag shows the policy (action to take if the message fails DMARC), and the “rua” tag shows the email address to send DMARC reports to.
DMARC authorizes mail by comparing the “From:” header to the sending domain, specified by either SPF or DKIM. This is called alignment, and is determined by how closely the “From:” header matches either of the domains specified in the DKIM or SPF records. This alignment can be configured as strict or weak for both SPF and DKIM in the DMARC record. Thus, a given email message passes DMARC if it fulfills at least one of these checks:
- SPF authentication and SPF alignment
- DKIM authentication and DKIM alignment
A relaxed SPF alignment in an organization’s DMARC record provides sufficient spoofing protection, whereas a strict alignment can result in messages from associated subdomains to be rejected or sent to spam. Most organizations opt for relaxed SPF alignment for flexibility and convenience. In order to pass the relaxed SPF alignment, the domain in the header “From:” address of the email must match or be a subdomain of the SPF authentication domain.
Details
The proof of concept (PoC) for this vulnerability showed that an email address using the format <username>@email.app-mail.expressvpn.com could be registered and verified on Mailgun. To do this, a new domain, expressvpn.com, was created on Mailgun without the “Create DKIM Authority” option, as shown below. The DNS verification for this domain was skipped, and this domain was left unverified on Mailgun, allowing a user to create this domain for use within Mailgun without actually owning the domain or creating a DKIM authority.
Next, a subdomain “email.app-mail.expressvpn.com” was added, also without the “Create DKIM Authority” option, as shown below.
This subdomain could also be added without verifying ownership or setting a DKIM authority. This subdomain was configured with the SPF and MX records shown below.
As seen above, the SPF record for this subdomain was configured to:
v=spf1 include=mailgun.org ~all
This shows that the SPF record for the newly added domain was configured to inherit the policy from “mailgun.org”. If the email passed the policy for the domain “mailgun.org”, it was able to pass the overall SPF check.
The subdomain was also configured to use MX records of “mailgun.org” as shown above, thus allowing the subdomain to be hijacked by the Mailgun user who added the subdomain.
Upon completion of the above step, the hijacked subdomain was marked as a verified domain on Mailgun, as shown below.
The following output was obtained upon performing a DNS lookup of the hijacked subdomain, proving that the subdomain was indeed taken over.
$ host email.app-mail.expressvpn.com
email.app-mail.expressvpn.com is an alias for mailgun.org.
mailgun.org has address <SNIP>
mailgun.org has address <SNIP>
mailgun.org mail is handled by 10 mxb.mailgun.org.
mailgun.org mail is handled by 10 mxa.mailgun.org.
ExpressVPN’s Security team also identified an undocumented SMTP header that Mailgun customers could use to send emails natively and bypass modifications and rewrite mechanism of the SMTP headers. Upon inserting this header, we were able to successfully send emails on behalf of the hijacked ExpressVPN subdomain. The header has not been published here at Mailgun’s request.
As shown below, the email message was sent using the “From:” address “billing@expressvpn.com”, where the domain was “expressvpn.com” and the domain of the sending server was “email.app-mail.expressvpn.com”.
The email was sent using MX records of mailgun.org and the subdomain from which the email was sent was configured to inherit the SPF policy of mailgun.org, which resulted in the email passing the SPF check.
The email message also passes the relaxed SPF alignment configured in ExpressVPN’s DMARC criteria, as the domain in the header “From:” address (expressvpn.com) matches the subdomain of the SPF authenticated domain (email.app-mail.expressvpn.com), specified by the mailed-by email header. This allows the email to pass the DMARC criteria and land in the recipient’s inbox.
Remediation
ExpressVPN collaborated with RecruitmentApp and Mailgun to formulate remedial actions for this bug. RecruitmentApp’s support team was also able to replicate the issue and subsequently informed Mailgun of the vulnerability. Mailgun took measures to prevent the abuse of this vulnerability by performing the following:
- Newly added domains on Mailgun will not be able to inherit “mailgun.org” SPF and DKIM TXT records and hence the domain will remain unverified and cannot be used to send emails.
- For DKIM inheritance, Mailgun now requires a verified domain in order for the domain to act as a DKIM authority. This prevents subdomains from inheriting the DKIM authority of an unverified parent domain.
As part of its investigation, the Mailgun team worked on robust measures to prevent subdomain takeover abuses on their platform.
Follow-up review
We conducted a follow-up review of the vulnerability before formally closing the investigation of the reported bug. Post remediation, we noted that the registration of new domains on Mailgun required DKIM verification that could no longer be skipped.
As shown below, an error was presented when we attempted to register existing domains such as “expressvpn.com” and “app-mail.expressvpn.com”, which prevented DKIM inheritance from an unverified parent domain, as seen previously.
When new subdomains are added to Mailgun, verification of the subdomain is required by accessing the DNS management console of the parent domain to add the required DNS records for the new subdomain. In addition, unverified subdomains are no longer allowed to send emails on Mailgun and have to be verified before they can be used to send emails.
We were able to send an email using the unverified subdomain to the email account used to register Mailgun. In this case, “rijesix513@kembung.com”, a disposable email account. However, when we also attempted to send emails to other email clients, we received an error stating that the domain was unverified and requires DNS configuration, as shown below.
Based on our follow-up review of this vulnerability, we concluded that we are only able to send an email from an unverified subdomain on Mailgun to the email address used to register the Mailgun account. We could also no longer use unverified subdomains on Mailgun to send emails to other email accounts.
Acknowledgements
ExpressVPN would like to thank the (unnamed) recruitment service and Mailgun for working with us on remediating this bug. We would also like to thank caon, the researcher who first reported this issue through our bug bounty program on BugCrowd. We were determined to get this fourth-party subdomain takeover issue fixed, so that it could not be exploited in the future to affect us or any other Mailgun customers.
Timeline
- 2021-05-13: A security researcher, caon, contacted ExpressVPN regarding the subdomain takeover vulnerability on the Mailgun platform that impacted the ExpressVPN subdomain.
- 2021-05-14: ExpressVPN’s Security Team investigated the issue and confirmed that it was indeed possible to take ownership of the ExpressVPN subdomain on the Mailgun platform.
- 2021-05-18: ExpressVPN reached out to RecruitmentApp to inform them of the misconfiguration and the resulting vulnerability and offered assistance to resolve the vulnerability as this issue might have affected almost all their customers.
- 2021-05-28: RecruitmentApp confirmed that they were able to replicate the issue and reached out to Mailgun.
- 2021-06-14: ExpressVPN was notified by RecruitmentApp that Mailgun had implemented a short-term fix that stopped users from sending emails using unverified subdomains on their platform, thus mitigating the risk from the misconfiguration. Mailgun also confirmed that they were working on a permanent solution to prevent such misconfigurations in the future.
- 2021-08-24: ExpressVPN confirmed that it is no longer possible to send emails using unverified subdomains. We also confirmed that unverified subdomains are no longer able to inherit “mailgun.org” SPF/DMARC and MX records to send emails.
- 2021-09-02: Mailgun informed ExpressVPN that they conducted a review of all activity surrounding this vulnerability and have found no evidence of it being exploited except by security researchers via a bug bounty program.
