Drop-by-Drop: Bleeding through libvips

During a recent engagement we encountered a quite common web application feature: profile image uploads. One of the tools we used for the tests was the UploadScanner Burp Suite extension, that reported no vulnerabilities. However, we noticed that the profile picture of our test user showed seemingly random pixels. This reminded us to the Yahoobleed bugs published by Chris Evans so we decided to investigate further.

When we downloaded the suspicious profile image it looked like this (compressed and converted to grayscale):

What we see seems partially “structured noise” (with large blank areas and repeating patterns), some large parts of homogeneous fields of color, and even some readable characters! To figure out what’s going on, we looked up the URL of the final profile image in Logger++. It turned out that the profile image was generated after UploadScanner uploaded an image called idat_zlib_invalid_window.png. This turned out to be part of the test image set of the fingerping utility that UploadScanner relies on. This image set contains a number of corrupted PNG files, and fortunately it also includes their valid originals – here’s how control.png from the fingerping set looks:

What we saw on the first, noisy image was parts of this control picture.

Estimating impact

At this point we had no idea why and how the “noisy” images were generated: we didn’t know the software stack, we didn’t know if the noise was the result of a memory leak or just the thumbnailer generating incomprehensible data out of the corrupted input. Being optimists we created a script that uploaded a control image as the profile picture for one test user and the corrupted test image for another. After a couple of hundred tries we saw parts of the control image appearing on the thumbnails generated from the corrupted PNG. This confirmed two important suspicions:

The implementation does leak uninitialized memory, what we see is not the result of the compression/conversion algorithm running wild
The uninitialized memory is read from a process address space shared between different user sessions, allowing cross-session data leaks

Tracking the source

At this point we still didn’t know what software causes this behavior. We ran the fingerping utility, but it gave no convincing results. Luckily an unrelated interface of the application leaked the name of one of the libraries used by the application, a well-known package for Node.js. A quick internet search for Node.js image processing libraries showed that one of the most popular candidates is Sharp, that is built-on libvips, a low-level, high performance library implemented in C.

We downloaded the 8.4.3 release of libvips, compiled it, and ran against the fingerping sample set. The results matched perfectly with the ones obtained from our target, although uninitialized memory leaks appeared like mostly black images with a few colored pixels on them, since the address space of the thumbnailer didn’t contain too much previously used data. When compiling libvips with ASAN, patterns of marked memory regions became clearly visible:

Accuracy

Our target resized uploaded images with libvips that resulted in individual pixels “blurred” with the surronding ones, making the output decoding inaccurate. An obvious solution to this was to upload input files with identical dimensions to the application. To create such an image that also causes memory leak we needed to track down the issue with our test file, idat_zlib_invalid_window.png (several other test files triggered the same issue, we just stuck with our first finding). The filename indicates an “invalid window” size related to “zlib”. Looking up the RFC it can be quickly found that PNG uses deflate compression, while zlib’s RFC states the following related to the deflate Compression Method (CM=8):

For CM = 8, CINFO is the base-2 logarithm of the LZ77 window size, minus eight (CINFO=7 indicates a 32K window size). Values of CINFO above 7 are not allowed in this version of the specification.

The CMF byte, consisting of the CM and CINFO fields start the deflate stream right after the IDAT chunk header in the PNG file. Loading the test image to the Kaitai Web IDE clearly shows that both CM and CINFO are set to 8, that is an invalid value in case of CINFO:

We created a test image with the appropriate size and edited the CINFO field in a hex editor to corrupt the window size. We also modified the source of the vipsthumbnail utility to allocate and initialize with known values a number of memory regions, then free them all:

void* allocs[10240];
int asizes[4] = {6088,95000,775,6088}; // Sizes determined after some hours of debugging...
for(int i = 0; i<10240; i++){
    allocs[i] = g_malloc(asizes[i % 4]);
    memset(allocs[i], 0xf1 + (i % 4), asizes[i % 4]);
}

for(int i=0;i<10240;i++){
    if ( 1|| i % 4 == 0 || i % 4 == 2 ){
        g_free(allocs[i]);
    }
}

When running our local thumbnailer against the newly created test image we saw nice white(ish) patterns appearing, with the expected #f1f1f1 – #f4f4f4 values, proving the possibility of bit-correct memory extraction:

Fixing things

After confirming the issue we alerted our client, and immediately notified the maintainers of libvips and Sharp. Both of them responded within hours, the fix landed in libvips the same day. Fortunately the code needed just a minor change, since bitmap buffer allocations went through the same two wrapper functions around g_malloc():

 @@ -173,7 +173,7 @@ vips_malloc( VipsObject *object, size_t size )
 {
	void *buf;

-	buf = g_malloc( size );
+	buf = g_malloc0( size );

        if( object ) {
		g_signal_connect( object, "postclose", 
 @@ -317,7 +317,7 @@ vips_tracked_malloc( size_t size )
	 */
	size += 16;

-       if( !(buf = g_try_malloc( size )) ) {
+       if( !(buf = g_try_malloc0( size )) ) {
 #ifdef DEBUG
		g_assert_not_reached();
 #endif /*DEBUG*/

However this is not the end of the story. Since we are taking about an open-source library, in order to get the actual applications fixed, all downstream packages and distributions must adopt the patch. As you can see in the timeline below, this took several days and the work of close to a dozen people! While distros were generally quick to adopt the patch in their most recent distributions, getting the fixes to stable/LTS releases took an extended period of time – at the time of writing Debian stable for example is still unfixed. On the bright side, our original target, Sharp even enabled a defense-in-depth measure in libvips that halts processing if corrupt data is encountered.

To help with the coordination we allocated a CVE number for this issue at MITRE. Unfortunately, this seemingly trivial part didn’t quite go as planned: MITRE’s CVE form clearly states that “[CVE’s] will not be published in the CVE List until you have submitted a URL pointing to public information about the vulnerability”. We wanted to request a CVE so package and distro maintainers can use it before details go public, so we gave minimal information on the CVE form, omitting any URL’s. Still, MITRE looked up the corresponding commit, added the URL to the ticket, and published the CVE with a non-sense description, that was later automatically fetched by numerous other trackers:

But all in all, we were assigned CVE-2019-6976 for this issue…

Offensive contributions

To improve the accuracy and efficiency of future black-box tests we contributed the following to offensive open-source tools:

We added the fingerprint of libvips to the fingerping library
We added an additional test to the UploadScanner extension that detects noise on the images retrieved via ReDownloader by trying to compress the raw bitmap data. Low compression rate indicates high entropy data that needs to be investigated. According to our internal tests memory leaks result in ~10x worse compression ratios than normal images. The current implementation is proof-of-concept level, community feedback is much appreciated!

As for white-box approaches, detecting similar bugs in libraries looks like a difficult task: although libvips has infrastructure for fuzzing, uninitialized memory issues generally don’t result in illegal memory access, and are thus not caught by most fuzzers. As we saw above, ASAN didn’t notice the problem either, although it made it more apparent for human inspection. After some experimentation Valgrind was able to detect the problem – the differences of the output for fixed and unfixed versions can be seen on the screenshot below (outputs were sanitized for clarity):

Conclusions

In this post we gave a detailed description of identifying and assessing a previously unknown vulnerability. Taking into the account the difficulties of detection, we believe that numerous similar issues are still lurking in applications big and small.

Image processing libraries are prone to bugs related to unsafe memory handling due to the complexity of their tasks. While in case of large-scale image processing tasks choosing a low-level, high-performance implementation may reasonable, most (web) applications don’t need to perform that much processing, so choosing less performant, but memory-safe implementations to handle untrusted image data seems generally reasonable. We have to note a couple of things though:

First of all, it’s currently not easy to tell if a particular library has any components implemented as native code. We wish more libraries declare this explicitly in their documentation.
Managed runtimes and built-in libraries can also suffer from vulnerabilities. Nonetheless, they still can be considered much safer than any new C/C++ code written from scratch.

From disclosure point of view, the handling of this vulnerability was ideal, with fast and professional responses from each affected party. This example also shows how much work it takes to deliver even the smallest fixes in a coordinated fashion – something people in security often forget. We’d like to thank John Cupitt and and Lovell Fuller for their outstanding response, the provided technical insight and helping with the coordination of the fixes! We also thank everyone involved in delivering the fixes to users in time!

Timeline

2019-01-18: Vulnerability reported to libvips and Sharp
2019-01-18: libvips 8.7.4 released, fixing the issue
2019-01-18: Debian, RedHat and net-vips notified
2019-01-18: Sharp changes to fail fast
2019-01-18: libvips version bumped in Homebrew 2019-01-18: Debian updated
2019-01-18: NetVips updated
2019-01-18: Fedora and Remi’s RPM updated
2019-01-18: MacPorts updated
2019-01-19: libvips version bumped in Alpine Linux aports
2019-01-26: MITRE assigns CVE-2019-6976 and publishes inaccurate vulnerability information
2019-03-31: Patch for Debian stable uploaded
2019-04-04: Patch for Debian stable accepted
2019-04-18: Blog post published in blog.silentsignal.eu
2019-04-18: Requested CVE update from MITRE
2019-04-24: MITRE updates CVE information