24 Feb 2017 | Peter Stöckli

The Cloudflare leak

Introduction

On the 23rd of February Tavis Ormandy of Google’s Project Zero made following security vulnerability accessible to the public: Cloudflare Reverse Proxies are Dumping Uninitialized Memory. The vulnerability affects many Cloudflare customers and especially their users. A vulnerable software component in Cloudflare’s reverse proxies led to the disclosure of Personally identifiable information (PII) of users around the world. Since Cloudflare reverse proxies are shared between customers, user information could emerge in a totally different place on the Internet.

The report describes how the security researchers at Google experienced the “cloudbleed” situation:

We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.

The report contains redacted user information from the ride-sharing unicorn Uber, health tracking company FitBit and dating site OkCupid.

How Cloudflare works

Let’s have a quick look at how Cloudflare works. Typically Cloudflare’s customers use their services for DDoS (Distributed Denial of Service) protection. Often the customers use DNS services provided by Cloudflare and/or their traffic is redirected via Cloudflare’s reverse proxies before the traffic is sent to the customers web server. From a user’s point of view: the user’s traffic to the reverse proxy is encrypted, where it’s decrypted and analyzed by Cloudflare’s algorithms.

+---------+             +------------+               +--------------+
|  User   |  via HTTPS  | Cloudflare |  via HTTP(S)  |   Customer   |
|(Browser)+------------>+ (Reverse   +-------------->| (Web server) |
|(Mobile) |             |   Proxy)   |               |              |
+---------+             +------------+               +--------------+

Cloudflare’s Response

Cloudflare has published a detailed report, where Cloudflare’s talented security guys describe the technical part of the vulnerability: Incident report on memory leak caused by Cloudflare parser bug.

They write that the earliest leaking could have started on the 22th September of 2016.

They also write:

The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines. We also undertook other search expeditions looking for potentially leaked information on sites like Pastebin and did not find anything.

However users on Twitter reported that they still found cached web pages using Google or Bing.

What’s important?

One important point is that not necessarily a Cloudflare customer’s site was leaking information about their users, but a totally different site of another Cloudflare customer could have been leaking that user information.

Another important point is that the listed search engines are not the only ones collecting and storing information from websites in the Internet. Think of caches, web crawlers, archive sites, solutions that store the content of websites for legal reasons, the list goes on…

Even our newly developed web application security scanner called SecBot, that continuously scans web applications for vulnerabilities stores the HTTP responses of the requests. Since we’re still in the development phase, SecBot hasn’t yet tested a site hosted behind a Cloudflare Reverse Proxy. If that would have been the case the database of SecBot could contain sensitive data of Cloudflare customers. And so could many other crawlers in the world.

And now?