New surge in AWS credential compromises tied to Grafana SSRF attacks

Over the past several weeks, Lacework has observed a significant uptick in AWS instance credential compromises that were traced to Grafana server-side request forgery (SSRF) attacks. This is not believed to be the result of a new vulnerability but rather increased exploitation of a three-year-old SSRF vulnerability: CVE 2020-13379, with the majority of compromises attributed to the same source.

This blog describes this tactic, indicators to look out for in your environment, and how Lacework composite alerts will help you identify potential compromises. 

Why Grafana?

Grafana is an analytics and visualization platform commonly used for monitoring and observability. As a popular open source platform, it is used by a variety of products and services across different industries such as in-cloud monitoring, application performance monitoring, CI/CD, and IoT. It also integrates with other widely used tools such as Jenkins and Gitlab while it also works with monitoring systems like Prometheus, InfluxDB, and Elasticsearch. 

In the case of CVE 2020-13379, there is a relatively large affected range that spans multiple versions, specifically 3.0.1 –  6.7.4 and 7.0.0 – 7.0.2. An analysis of telemetry collected by Lacework revealed that approximately a quarter of environments we monitor are running some version of Grafana or a Grafana plugin at any given time. Among the installed versions, approximately 14% fell within the affected range for SSRF exploitation. This sampling could be representative of a much larger attack surface across all cloud infrastructure. 

What is SSRF?

Server-side request forgery (SSRF) is a vulnerability that, when exploited by an attacker, is used to instruct the affected system to make HTTP requests on their behalf. This can expose unintended or restricted resources which only the vulnerable system should have access to, inadvertently allowing attackers access to these resources. In the context of AWS, an SSRF attack can potentially lead to the leakage of AWS instance metadata credentials (T1552).

AWS instance metadata provides valuable information about an EC2 instance, including temporary security credentials associated with the instance’s Identity and Access Management  (IAM) role. 

These credentials are used to access AWS services and resources. Generally an SSRF attack is carried out as follows:

1. The attacker identifies a vulnerable application that accepts user-supplied URLs or allows requests to arbitrary resources.
2. The attacker crafts a malicious request, including a URL pointing to the AWS instance metadata service endpoint. The endpoint typically resides at “http://169.254.169.254/latest/meta-data”.
3. The vulnerable application, being tricked by the attacker, sends the crafted request to the AWS metadata service endpoint.
4. The AWS metadata service, assuming the request originates from the EC2 instance itself, responds with sensitive information, including credentials.
5. The vulnerable application, unaware of the response content, forwards it to the attacker.

What to look out for

A successful Grafana SSRF exploit may leave evidence in the form of AWS CloudTrail logs, web logs, or host and network data such as a network connection initiated by a Grafana application. An initial host-based indication may be a connection to the host metadata IP address, 169.254.169.254, originating from a Grafana process. This would be considered a weaker signal, however, and would require supporting evidence. A stronger indicator may involve connection to a domain that resolves to instance metadata. This method is an attempt to bypass URL based detections. One domain that was recently observed in the widespread attacks is redirect.studio. The attacker in control of this domain was found to only adjust the DNS when performing attacks. When not in use, the DNS was parked. 

Figure 1: Lacework anomaly event from Grafana SSRF redirect

 

A similar indication may be a connection from Grafana to Cloudflare DNS 1.1.1.1. This is the redirect used by ProjectDiscovery’s Nuclei scanner. The Nuclei scanner is a popular open source scanner leveraged for both testing purposes and malicious reconnaissance. Nuclei scanner offers templates that can easily be used for mass scanning. For Nuclei’s CVE 2020-13379 template, a successful SSRF exploit initiates a redirect to 1.1.1.1 and the returned webpage is inspected for the keywords “Cloudflare” and “DNS” to confirm successful exploitation.

Figure 2: Nuclei template for CVE 2020-13379

 

Other SSRF redirect domains to look for include those leveraged as part ProjectDiscovery’s interactsh OAST/OOB library and BurpSuite. For more information on these tools, refer to our previous blog.

With regard to evidence in AWS CloudTrail, the strongest indicator of leaked instance credentials by way of SSRF is when the keys are actually leveraged from outside of the cloud environment. These may be easily identified by way of the ARN for the identity. For example, any role assumed by an instance will be appended with the AWS instance ID as such:

    "arn": "arn:aws:sts::121749836629:assumed-role/EC2-CodeDeploy/i-0697a4ab85e5da13f"

Keys obtained from the illicit instance’s metadata request can then be used to interact with the cloud environment. (Note: Typically the attacker will first make a call to sts.GetCallerIdentity to gather information about the identity making the request.)

Figure 3: Automated investigation with composite alerts

 

In the Lacework composite alert shown above, we have the Possible Hijacking of Instance Credential signal which alerted the customer to the suspected key compromise. Additional signals, such as anomalous login and discovery activity, strengthen the alert to warrant a high severity. It’s important to note that any potential instance of credential hijacking accompanied by activity that is anomalous for the role should be considered highly suspicious. This is especially true for any infrastructure discovery or S3 reconnaissance, as this may be a leading indicator of a data breach.

It’s also necessary to audit the role used by the attacker as it’s ultimately the identity’s permissions that will decide the impact of an attack. Another signal in the sample alert is for Entitlement Management, which highlights this risk for the compromised identity as per insights from CIEM (Cloud Infrastructure Entitlements Management). 

In the rapidly evolving threat landscape, it’s easy to overlook dangers posed by older vulnerabilities. This can become particularly alarming when they possess a wide attack surface and offer a straightforward path for exploitation. Detecting SSRF attacks is of paramount importance because they not only pose a direct threat to the targeted server, but also enable access to sensitive cloud resources. Fortunately, reducing the attack surface for this SSRF attack simply involves upgrading to the latest version of Grafana or a non-affected version. Enforcing Instance Metadata Service Version 2 (IMDSv2) would also prevent unauthorized credential access via SSRF attacks in general. 

The following are related network IOCs and signals to allow for detection of this activity and similar SSRF exploits. Note: several IPs are known VPNs.

• 84.32.188.67 
• 31.171.154.133
• 169.150.197.116
• 169.150.197.68
• redirect.studio

Behavioral IOCs include:

• Connection from Grafana process to
  • instance metadata 
  • Cloudflare DNS 1.1.1.1
  • Domain resolving to instance metadata
• Usage of instance credentials from outside cloud environment
  • Anomalous behavior