Regex: Regularly Exploitable

Thursday, June 11, 2015

Here's a quick demonstration of why Regular Expressions (regex) can be bad for implementing character whitelisting.

I was reading through an application security assessment report recently and noticed a recommendation for preventing Operating System Command Injection (OSCI) that implemented character whitelisting on a given file name through the following regex.

/^[\/a-zA-Z0-9\-\s_]+\.rpt$/m

At first glance, the regex seems legit, right? It attempts to match any combination of letters, numbers, dashes, underscores, slashes, and whitespace, ending with the ".rpt" extension. Already knowing that there was a flaw here (we'll get to that in a moment), I put together the following proof-of-concept to demonstrate the security (or in-security) of the filter.

<?php
$file_name = $_GET["path"];
if(!preg_match("/^[\/a-zA-Z0-9\-\s_]+\.rpt$/m", $file_name)) {
    echo "regex failed";
} else {
    echo exec("/usr/bin/file -i -b " . $file_name);
}
?>

I tried all the typical attack payloads, and sure enough, the regex prevented injection into the shell command. The key here, and why one must always use caution when implementing regex filters, is understanding what the \s character class represents. Most resources are vague and say that it includes "any whitespace character", but what does that include? In most regex implementations, whitespace includes [ \t\r\n\f], i.e. spaces, tabs, line breaks, and form feeds. See the problem yet?

Many testers don't think about the impact of line breaks when dealing with injections, but when we're dealing with shell commands, line breaks become very important. Consider the following attack payload.

/path/to/file%0Aid%0A.rpt

%0a is a URL encoded line feed/break (whitespace), so according to the regex, this payload is safe. However, what happens when you put this into a shell? Below is the output from copying and pasting the decoded version of the above payload into a terminal prompt.

# /usr/bin/file -i -b /path/to/file
ERROR: cannot open `/path/to/file' (No such file or directory)
# id
uid=0(root) gid=0(root) groups=0(root)
# .rpt
bash: .rpt: command not found
# 

Do you see what happened? Each line break started a new command and we can see that the shell executed our arbitrary id command. Here's what it looks like through a web interface.

So let's fix this. Show of hands for how many people think the below regex solves the injection issue? (I replaced the \s with a space .)

/^[\/a-zA-Z0-9\- _]+\.rpt$/m

If we use the same payloads as before, including the one that resulted in a successful injection, we can see that the issue has been resolved.

Or has it? Consider the following attack payload.

/path/to/file.rpt%0aid

What just happened?! Let's look at the new regex again.

/^[\/a-zA-Z0-9\- _]+\.rpt$/m

See that m at the end of the regex pattern? It means something. At the end of the regex pattern declaration in PHP (available in other frameworks as well, but may be declared differently) there is a spot for modifiers. Regex modifiers change how the regex engine applies the pattern to the string. Discussing the different regex modifiers is outside the scope of this article, but what we want to focus on here is that the filter pattern is using the multiline modifier (m is the flag for multiline). The multiline modifier basically changes the way the beginning (^) and end ($) of line characters behave. When the multiline modifier is absent, the ^ and $ characters act as the beginning and end of the string, as opposed to the line. This is an important distinction, because in the payload, we are able to leverage the multiline modifier's effect on the $ character and a line break to create a match. We can then add anything we want to the end of the string to execute arbitrary commands within the shell.

There are a couple of takeaways here.

First, be mindful of how you build whitelists. Be as explicit as possible. The higher the level of whitelist, the better. For instance, in the above example, the optimal solution would be to build a whitelist of complete file names that are allowed, and ignore regex all together. If the file names are not known and we need to whitelist at the character level, then we would need to build a better regex that accounts for what is included in all allowed character classes within the context of the filter, e.g. %0a and its significance to shell commands and the multiline modifier.

Second, Burp was not able to find this vulnerability when the scanner speed was set to Normal (default). It wasn't until I set the scanner speed to Thorough and hard coded the ".rpt" extension into the payload that Burp was able to find it. There is no replacement for thorough manual testing by someone that knows what they're doing.

A shout out to John Poulin, who taught me a thing or two about exploiting regex that ultimately lead to this article. Thanks John.

Recon-ng Update (v4.6.0)

Thursday, May 21, 2015

Recon-ng v4.6.0 is the largest single framework commit to date. Most of the changes are behind the scenes and won't effect the average user's experience. Below is a summary of the important changes, including the one change that will affect everyone: dependencies.

Dependencies

Starting with Recon-ng v4.6.0, the framework no longer includes dependencies within the code base. Rather, the framework requires users to install dependencies before the dependent modules can be used. I know, I know. I've been quite public about how much I dislike requiring the installation of 3rd party libraries for my tools to function. However, I recently came to the realization that managing the integration of other software packages within my own was a waste of time and effort. Package managers exist for a reason. So I elected to go against my own personal preference and enforce 3rd party dependency installation. For Kali users, this is seamless, as dependencies are installed alongside the framework. For those not using Kali, you'll want to use the Python Package Index (PyPI or PIP) to install the dependencies listed in the REQUIREMENTS file. Follow the guidance on the Recon-ng Wiki Usage Guide for installation instructions. I recommend using virtual environments (virtualenv) to install the required packages and prevent clutter in your local Python instance.

All current dependencies are module specific, so the core framework is still completely functional without the dependencies. The framework conducts a module dependency check at runtime and disables the modules that fail to load due to missing dependencies. The framework will provide a warning for each disabled module.

Module Changes

As always, the update includes lots of module changes. I added threading to several modules, made lots of bug fixes, removed a few defunct modules (jigsaw, breachalarm), and merged a few new modules (unmangle, fullcontact). In addition, all modules were updated to the new module template discussed below.

Developer Changes

The biggest changes in Recon-ng v4.6.0 are for module developers and include new structures for both the framework's core package and the module template.

The files that make up the framework have been rearranged into a Python package. Therefore, importing API elements is a bit different now. Browse the package structure to get an idea of how to access API elements. As the framework moves forward, internal functionality will be moved into modules that can be imported from logical locations within the package. Several utilities have already been moved. Also, in order to properly handle missing module dependencies, some internal functionality has been abstracted out into mixins. The mixins and utilities are located at "recon/mixins" and "recon/utils" respectively.

The framework now loads all of the modules into memory when the framework launches as opposed to when the user invokes the module. This was originally how the framework behaved, but in a lapse of judgment, the developer perspective was favored over the user's, which led to the poor design choice and current loading system. Loading the modules when the framework launches is slightly slower, but affords the opportunity to capture load errors in modules, disable them, and notify the user of issues before they begin using the framework. With this comes the reintroduction of the "reload" command. The reload command is available in the global context and reloads all of the modules without the developer having to restart the framework.

Recon-ng Home Page

Method Interchange: The Forgotten Vulnerability

Tuesday, May 12, 2015

(Originally posted @nvisium. Updated on Thursday, May 14, 2015.)

When you think of the most prolific scorers in NBA history, you think of names like Kareem Abdul-Jabbar, Bill Russell, and Karl Malone. Most casual basketball fans aren't familiar with names like Oscar Robertson, Bob Cousy, or John Stockton, but without these men, the previously mentioned scorers would have been much less effective as an offensive threat. And thus is the life of the Method Interchange vulnerability.

I've mentioned Method Interchange in several application security circles recently and have been greeted with strange looks and questions of "Method what?". Once I explained Method Interchange, the reaction changed from "huh?" to, "Oh my, I need to check for that from now on!" Quite simply, Method Interchange is the ability to send parameters via the URI query string or request payload and have them processed by the application regardless of which was originally intended. While seemingly benign, as we'll see later in the article, Method Interchange drastically increases the exploitability of other, more capable, attacks.

GET and POST Methods

The HTTP protocol specifies two traditional methods for passing parameters from a user-agent to a server. Parameters can be passed as name-value pairs within the URI query string, or as name-value pairs within the payload of the request. When parameters are passed in the URI query string, the request is typically sent using the GET method. When parameters are passed within the request payload, the request is typically sent using the POST method.

  • GET
GET /resource.ext?name1=value1&name2=value2 HTTP/1.1
Host: www.example.com
  • POST
POST /resource.ext HTTP/1.1
Host: www.example.com

name1=value1&name2=value2

There are other methods for sending parameters to a server (alternate data formats such as XML and JSON, embedded in the structure of the URI as in RESTful services, other parts of an HTTP request such as custom headers), but here we are referring to the traditional methods provided by the HTTP protocol: GET and POST.

Method Interchange Manifested

Server frameworks will either store parameters in global arrays as they parse the request, or have method calls for pulling parameter data from the request object when needed. For example, PHP stores GET and POST parameters in the $_GET and $_POST global arrays as it parses each request. The $_GET array stores all of the parameters that were sent as part of the URI query string. The $_POST array stores all of the parameters that were sent in the request payload. When a developer expects data from the request payload, he uses the $_POST array to fetch it. When he expects data from the URI query string, he uses the $_GET array. What many security testers forget, or are unaware of, is that PHP, like many frameworks, also has the $_REQUEST array, which holds all parameters and values regardless of whether they were sent in the URI query string or as part of the request payload. It is the use of the $_REQUEST array and similar implementations that allows for Method Interchange.

Below is an abbreviated list of methods and objects from several popular development frameworks that could allow for Method Interchange. With some of these frameworks, this is only one of multiple ways to introduce Method Interchange. And in some cases, vulnerability requires other conditions to exist.

Framework Method/Object Description
PHP $_REQUEST Array that contains GET, POST, and Cookie values.
Rails match Routing method that allows for routing requests to multiple verbs e.g. GET, POST, etc.
Django HttpRequest.REQUEST Dictionary that contains both GET and POST parameter values.
Spring MVC HttpServletRequest.getParameter() Controller method that provides access to both GET and POST parameter values if the @RequestMapping annotation method paremeter is not set.
.NET MVC public ActionResult <action>(<type> <name>) Configuration that binds parameters from all sources e.g. GET, POST, route, etc., to Action method parameters.

Discovery

Discovering Method Interchange is incredibly simple. First, identify a request and move the payload from the URI query string to the request payload or vice versa. Keep in mind that POST requests require a Content-Type header to function properly, so add/remove the necessary headers as needed. While this can be done manually, Burp Suite makes this incredibly easy through the "Change request method" context menu option (thanks Mike).

Then, fire off the request and analyze the response. If the response mirrors that of the original request, then the application is vulnerable to Method Interchange.

Exploitation: The Great Assist

So where does the assist analogy fit in to all of this? And so what if parameters can be sent in the URI query string or request payload interchangeably? ... Have you ever tried to conduct a Cross-Site Request Forgery (CSRF) attack that leveraged parameters sent in the request payload? How about a Cross-Site Scripting (XSS) attack? Or how about exploiting a Session Fixation vulnerability where the a session variable was sent in the request payload? If Method Interchange exists, then the CSRF and XSS attacks become much simpler, and Session Fixation vulnerabilities may become exploitable.

Exploiting XSS in a POST parameter requires a GET-to-POST script on a 3rd party server, resulting in a suspicious link with which to exploit a user. Exploiting CSRF in a POST parameter requires embedding complex JavaScript into a 3rd party application, also resulting in a suspicious link. Method Interchange allows for a simple URL containing a XSS/CSRF payload to appear as a link to a trusted resource. Exploiting Session Fixation may not be possible when a session is sent via a POST parameter. Method Interchange allows an attacker to fixate a session via a simple URL that appears as a link to a trusted resource.

Remediation

Preventing Method Interchange is as simple as discovering it. Use methods, objects, etc. that explicitly provide values from the URI query string (GET) and request payload (POST). All popular frameworks are capable of preventing Method Interchange. Below are a few resources that address acquiring parameters from requests for an explicitely defined method.

Parameter binding is the recommended approach for accessing parameter values of an explicitely defined method. However, .NET MVC is only vulnerable to Method Interchange when using parameter binding. Binding parameters in .NET MVC without a AcceptVerbsAttribute decorator causes the framework to search all parameter sources (query string, request payload, route data, etc.) for a matching parameter name. .NET MVC is otherwise immune.

Conclusion

So as you see, Method Interchange alone is rather harmless. But when combined with other vulnerabilities, Method Interchange provides the needed assist to drastically elevate the risk exposure of the vulnerability.

Update #1

I've had a few folks reach out with regards to the similarities between HTTP Verb Tampering and Method Interchange. Quite simply, Method Interchange is a subset of Verb Tampering. The difference is that Method Interchange is a result of how an application processes parameters, specifically related to the GET and POST methods, where Verb Tampering is a result of how the server and application apply access controls around all HTTP methods (verbs). Below is a great whitepaper describing Verb Tampering. It also addresses the concept of Method Interchange.