Quick and dirty Android binary XML edits

Author: dnet

Last week I had an Android application that I wanted to test in the Android emulator (the official one included in the SDK). I had the application installed from Play Store on a physical device, and as I’ve done many times, I just grabbed it using Drozer and issued the usual ADB command to install it on the emulator. (The sizes and package names have been altered to protect the innocent.)

$ adb install hu.silentsignal.blogpost.apk
1337 KB/s (27313378 bytes in 22.233s)
        pkg: /data/local/tmp/hu.silentsignal.blogpost.apk
Failure [INSTALL_FAILED_CONTAINER_ERROR]

A quick search on the web revealed that the application most probably had the installLocation parameter set to preferExternal in the manifest file. Latter is an XML file called AndroidManifest.xml that contains important metadata about Android applications and is transformed into a binary representation upon compilation to reduce the size and processing power required on resource-constrained devics. Running android-apktool converted it back to text format, and revealed that it was indeed the cause.

<?xml version="1.0" encoding="utf-8"?>
<manifest android:versionCode="133744042" android:versionName="4.13.37"
  android:installLocation="preferExternal" package="hu.silentsignal.blogpost"
  xmlns:android="http://schemas.android.com/apk/res/android">

Most results of the web search agreed that the emulator (although capable of emulating SD cards) is incompatible with this setting, some suggested increasing the memory of the emulated device, others said the same about the SD card, but unfortunately none of these worked for us. The majority of accepted answers solved the problem by changing the preferExternal parameter to auto, which is the default.

Changing an Android application and repackaging it is also a breeze usually, apktool supports this natively, I just have to sign the resulting APK with a key of my own. However, this application used some features that apktool (and other tools invoked in the process, including AAPT) didn’t like. I’ve met this situation before, and there are usually two solutions.

  1. Removing such features in a way that the application can still be tested is a cumbersome series of iterations, and leads to almost certain insanity.

  2. Updating apktool and some dependencies so that they accept such features is rather painless, as it usually Just Works™.

However in this case, even beta versions couldn’t handle the task, I’ve even written some wrapper scripts around the dependencies to tweak with parameters like minimum and targeted API levels, but got nowhere. Then I realized, binary XML files still have to store the attribute names somewhere, so I fired up a hex editor and hoped that the Android runtime won’t complain about unknown attributes and will use the default value. It seemed that the format uses UTF-16 to store strings, and at offset 0x112, I changed the i to o, resulting in installLocatoon.

00f0  3a080000 82080000  0f006900 6e007300  |:.........i.n.s.|
0100  74006100 6c006c00  4c006f00 63006100  |t.a.l.l.L.o.c.a.|
0110  74006f00 6f006e00  00000b00 76006500  |t.o.o.n.....v.e.|
0120  72007300 69006f00  6e004300 6f006400  |r.s.i.o.n.C.o.d.|
0130  65000000 0b007600  65007200 73006900  |e.....v.e.r.s.i.|

Then, I simply updated the manifest in the APK file using a ZIP tool since APK files are just ZIP archives with specific naming conventions, just like JAR, DOCX and ODT files. Since the digital signature of the APK is now corrupted, I had to resign it with jarsigner, but installation still failed.

$ 7z -tzip a hu.silentsignal.blogpost.apk AndroidManifest.xml

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=hu_HU.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

Scanning

Updating archive hu.silentsignal.blogpost.apk

Compressing  AndroidManifest.xml      

Everything is Ok
$ jarsigner -sigalg SHA1withRSA -digestalg SHA1 \
    -keystore s2.jks hu.silentsignal.blogpost.apk s2
Enter Passphrase for keystore: 
$ adb install hu.silentsignal.blogpost.apk
1337 KB/s (27313378 bytes in 22.233s)
        pkg: /data/local/tmp/hu.silentsignal.blogpost.apk
Failure [INSTALL_PARSE_FAILED_NO_CERTIFICATES]

How can there be no certificates? Let’s list the contents with a ZIP tool, and look for the META-INF, an old friend from the Java world (JAR files) that contains a list of files with name and (in case of Android, SHA-1) hash (MANIFEST.MF), a public key (*.RSA in case of RSA), and a signature of the former with the latter (*.SF). The name of the public key and signature files are the same as their alias in the keystore converted to uppercase; see the last parameter of jarsigner in the above commands (s2).

   Date      Time    Attr     Size   Compressed  Name
------------------- ----- -------- ------------  --------------------
2014-04-03 14:44:42 .....   362941       110461  META-INF/MANIFEST.MF
2014-04-03 14:44:42 .....   369105       113933  META-INF/S2.SF
2014-04-03 14:44:42 .....     2046         1865  META-INF/S2.RSA
2014-02-27 13:37:16 .....      928          637  META-INF/CERT.RSA
2014-02-27 13:37:16 .....   361020       113942  META-INF/CERT.SF

See? It’s there! Also, another certificate (CERT.*), since without apktool (which rebuilds APK archives from scratch) everything except AndroidManifest.xml is included from the original file. Let’s delete the META-INF directory, and try again.

$ 7z -tzip d hu.silentsignal.blogpost.apk META-INF

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=hu_HU.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

Updating archive hu.silentsignal.blogpost.apk

      
Everything is Ok
$ jarsigner -sigalg SHA1withRSA -digestalg SHA1 \
    -keystore s2.jks hu.silentsignal.blogpost.apk s2
Enter Passphrase for keystore: 
$ adb install hu.silentsignal.blogpost.apk                            
1337 KB/s (27313378 bytes in 22.233s)
        pkg: /data/local/tmp/hu.silentsignal.blogpost.apk
Success

With this last modification, it worked, and I was able to explore the application within the emulator that makes it much more easier to set a global proxy and manipulate certificates than a physical device, and I haven’t even mentioned faking sensors and GSM information, or taking snapshots. The lesson here was that

  • if a difficult format takes more than 5 minutes to recreate, it’s worth considering manual editing in case of simple modifications,
  • most XML readers (including the Android runtime) tend to ignore unknown attributes, and
  • the META-INF directory should be removed before signing an APK file, otherwise the runtime refuses it.

Thanks to Etamme for the featured image Androids, licensed under CC-BY-3.0


Sanitizing input with regex considered harmful

Author: dnet

Sanitizing input (as in trying to remove a subset of user input so that the remaining parts become “safe”) is hard to get right in itself. However, many developers doom their protection in the first place by choosing the wrong tool to get it done, in this case, regular expressions (regex for short). While they’re powerful for quite a few purposes, as the old saying goes,

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

During a recent pentest, we found an application that did this by stripping HTML tags from a string by replacing the regular expression <.*?> with an empty string. (Apparently, they haven’t read the best reaction to processing HTML with regexes.) For those wondering about the question mark after the star, it disables the default greedy behavior of the engine, so the expression matches a less-than sign, as few characters as possible of any kind, and a greater-than sign. At first sight, one might think that’s the definition of an HTML tag, and for a minute we also believed it was the case.

In regexes, the dot matches any character. However, the definition of any excludes newlines (ASCII 0x0a, \n) by default in most implementations, while the HTML standard allows for such characters inside tags, which gives us a specific class of tags that are valid in browsers but are not stripped by the above algorithm. Below are some examples of platforms used to implement web applications and their behavior regarding this “challenge”. Some libraries have similar solutions, but only one thing was common in these five languages; by default, the above expression fails the test. For the sake of brevity and readability, examples were produced in interactive shells (REPLs); in case of Java and .NET, Jython and IronPython were used, respectively.

Java

>>> from java.util.regex import Pattern
>>> p = Pattern.compile('<.*?>')
>>> p.matcher('<foobar>').replaceAll('')
u''
>>> p.matcher('<foo\nbar>').replaceAll('')
u'<foo\nbar>'

The official documentation states that dot matches “any character (may or may not match line terminators)”. The link points to a section that says “The regular expression . matches any character except a line terminator unless the DOTALL flag is specified.” [emphasis added] Adding the flag solves the problem, as it can be seen below.

>>> p = Pattern.compile('<.*?>', Pattern.DOTALL)
>>> p.matcher('<foobar>').replaceAll('')
u''
>>> p.matcher('<foo\nbar>').replaceAll('')
u''

Python

>>> import re
>>> re.sub('<.*?>', '', '<foobar>')
''
>>> re.sub('<.*?>', '', '<foo\nbar>')
'<foo\nbar>'

Python follows a similar path, even the flag is called the same: “In the default mode, this matches any character except a newline. If the DOTALL flag has been specified, this matches any character including a newline” [emphasis added].

>>> re.sub('<.*?>', '', '<foobar>', flags=re.DOTALL)
''
>>> re.sub('<.*?>', '', '<foo\nbar>', flags=re.DOTALL)
''

PHP

php > var_dump(preg_replace("/<.*?>/", "", "<foobar>"));
string(0) ""
php > var_dump(preg_replace("/<.*?>/", "", "<foo\nbar>"));
string(9) "<foo
bar>"

Although PHP has an interactive mode (php -a), return values are silently discarded, and var_dump doesn’t escape newlines. However, it clearly illustrates that it behaves just like the others, but PHP doesn’t mention this behavior in the official manual for preg_replace (even though a user comment points it out, it lacks the solution). The PCRE modifiers page has the answer, the s modifier should be used, and it even shows the longer name for it (PCRE_DOTALL), although there’s no way to use it, in contrast with Python’s solution (re.S is equivalent to re.DOTALL).

php > var_dump(preg_replace("/<.*?>/s", "", "<foobar>"));
string(0) ""
php > var_dump(preg_replace("/<.*?>/s", "", "<foo\nbar>"));
string(0) ""

.NET

>>> from System.Text.RegularExpressions import Regex
>>> Regex.Replace('<foobar>', '<.*?>', '')
''
>>> Regex.Replace('<foo\nbar>', '<.*?>', '')
'<foo\nbar>'

Of course, Microsoft surprises noone by having its own solution for the problem. In their documentation on regexes, they also mention that dot “matches any single character except \n”, but you have to figure it out yourself; there’s no link to the Singleline member of RegexOptions.

>>> from System.Text.RegularExpressions import RegexOptions
>>> r = Regex('<.*?>', RegexOptions.Singleline)
>>> r.Replace('<foobar>', '')
''
>>> r.Replace('<foo\nbar>', '')
''

Ruby

irb(main):001:0> "<foobar>".sub!(/<.*?>/, "")
=> ""
irb(main):002:0> "<foo\nbar>".sub!(/<.*?>/, "")
=> nil

Ruby performs as usual, having easy-to-write/hard-to-read shorthands, however, its solution is almost as dumbfounding as the above. Like PHP, it expects modifiers as lowercase characters after the trailing slash (/), but it interprets s as a signal to interpret the regex as SJIS encoding (I never knew it even existed), and wants you to use m (called MULTILINE by the official documentation, adding to the confusion), which is used for other purposes in other regular expression engines.

irb(main):005:0> "<foobar>".sub!(/<.*?>/m, "")
=> ""
irb(main):004:0> "<foo\nbar>".sub!(/<.*?>/m, "")
=> ""

Javascript

> "<foobar>".replace(/<.*?>/, "")
''
> "<foo\nbar>".replace(/<.*?>/, "")
'<foo\nbar>'
> "<foo\nbar>".replace(/<.*?>/m, "")
'<foo\nbar>'

JavaScript has three modifiers (igm), none of them useful for making dot match literally any character. The only solution is to do it explicitly, the best one of these seems to be matching the union of whitespace and non-whitespace characters.

> "<foo\nbar>".replace(/<[\s\S]*?>/, "")
''
> "<foobar>".replace(/<[\s\S]*?>/, "")
''

Conclusion

The above solutions address a single problem only (stripping HTML tags having line breaks), processing untrusted input is much more than this. If you build a web application that must display such content, use a proper library for this purpose, preferably a templating language that performs escaping by default. For other purposes, use a DOM and don’t forget to test for corner cases, including both valid and broken HTML.


Testing websites using ASP.NET Forms Authentication with Burp Suite

Author: dnet

Testing a website is usually considered just another day at work, Burp Suite is usually the tool of our choice for automating some of the scans that apply in this field. Assessing the authenticated part of the site is also common, and since Burp can be used as an HTTP proxy, it can capture our session tokens (usually HTTP cookies) and perform scans just like we’d do as humans. This token is usually remain unchanged over the time of the session, and the session itself is kept alive by the scanner activity itself.

However, a few weeks ago, we encountered an ASP.NET site that offered the regular ASP.NET_SessionId cookie and after successful login we got another one called .ASPXFORMSAUTH indicating that the application uses Forms Authentication. We started issuing requests including these two cookies, but within 5 minutes, the server started responding like we’re not logged in. Investigation followed, and we could reproduce that even by sending a request every 10 seconds, after 2 minutes pass from login, the requests with the two original cookies are denied. The operators confirmed that the session timeout is 2 minutes (which is resonable given the normal use-cases of the application), and looking at the responses revealed that around halfway between login and the 2 minutes deadline, a new .ASPXFORMSAUTH cookie is set by the server.

Apparently, Burp Suite ignored such Set-Cookie headers at the time both in its Scanner and Intruder modules, so I wrote a simple plugin that would hook HTTP requests within Burp and behave like a browser for this specific cookie. If such a cookie is received it gets stored as a protected field of the class, and subsequent requests will be modified to include the latest value. Since Burp uses multiple threads, this value also needs locking in order to avoid race conditions. Burp also offers some help with parsing and mangling HTTP headers, so besides hooking HTTP traffic and initialization, the plugin starts by storing a reference to an instance of the IExtensionHelpers interface.

public class BurpExtender implements IBurpExtender, IHttpListener
{
  protected IExtensionHelpers helpers;
  protected String cookie_value = null;
  public static final String COOKIE_NAME = ".ASPXFORMSAUTH";
 
  @Override
  public void registerExtenderCallbacks(IBurpExtenderCallbacks callbacks)
  {
    helpers = callbacks.getHelpers();
    callbacks.setExtensionName("AspxFormAuth");
    callbacks.registerHttpListener(this);
  }

By registering a HTTP listener, the processHttpMessage method will be called by Burp every time an HTTP request is issued, regardless of its source, including the browser, the Scanner or Intruder modules. The messageIsRequest parameter can be used to define different behavior for requests and responses; in case of former, any Cookie: headers with the matching name in the request will be updated to the latest value (if there’s one, hence the null check).

@Override
public void processHttpMessage(int toolFlag, boolean messageIsRequest, IHttpRequestResponse messageInfo)
{
  if (messageIsRequest)
  {
    synchronized (this) {
      if (cookie_value == null) return;
    }
    byte[] request = messageInfo.getRequest();
    IRequestInfo ri = helpers.analyzeRequest(request);
    List parameters = ri.getParameters();
    for (IParameter parameter : parameters) {
      if (parameter.getType() == IParameter.PARAM_COOKIE &amp;&amp;
          parameter.getName().equals(COOKIE_NAME)) {
        synchronized (this) {
          messageInfo.setRequest(helpers.updateParameter(request,
                helpers.buildParameter(COOKIE_NAME, cookie_value,
                  IParameter.PARAM_COOKIE)));
        }
      }
    }
  }

The only remaining task is feeding the cookie_value member with tokens by parsing responses. Again, the Burp helpers are used to analyze the headers and if a cookie with a matching name is found, its value is stored, taking special care to let only a single thread access the value at a time.

else
{
  IResponseInfo ri = helpers.analyzeResponse(messageInfo.getResponse());
  List cookies = ri.getCookies();
  for (ICookie cookie : cookies) {
    if (cookie.getName().equals(COOKIE_NAME)) {
      synchronized (this) {
        cookie_value = cookie.getValue();
      }
    }
  }
}

Using this technique, we’d been able to perform the assessment, the full source code can be found in a branch of our GitHub repository. If you don’t want to compile it for yourself, a binary JAR is also available on the release page. We’re also proud that since we released the Burp OAuth plugin this one is based on, former has been built upon and improved by SecureIdeas as part of their Co2 plugin.