Welcome to Session 503, Creating Secure iPhone and Mac OS X Applications. I'm Geoff Keating. I'll be initially talking about security hardening features, new features that we've implemented in Snow Leopard and all features that we've made better from Leopard and before. So, in this part of the presentation, I'll be talking about new hardening features we've enabled by default in Snow Leopard. I'll be talking about what they mean for you as developers, and I'll be telling you about some features that we didn't turn on, features that you probably might want to consider turning on, maybe something you actually should turn on.

Later on, my colleague, Drew will be talking about what happens after you've turned them on and you're still trying to find remaining security issues in your program. You'll be talking about determining the exploitability of crashes that you've found through testing both automatically and by hand. So, I was talking about features that we've turned on by default. What does that mean, on by default? Well, even though they're on by default, you still need to do some-- there are still some situations under which they can't be on.

In particular, for most of the features I'm talking about, you need to be targeting, 10.5, or higher, and building 64-bit, that will get you the best combination of hardening features. It doesn't matter what language you use, C, C++, Objective-C, and Objective-C++, all of these that's fully supported by the-- for these hardening features.

So, something you're probably thinking about already is, wait, stuff on by default, new, won't that cause trouble in my program? Well, so far as we can tell, these hardening features have very small performance impact. When we tried to measure we found that they were too small to reliably measure.

So not faster but not slower either and all of the features together gave only a very small code size impact as we've turned them on. Also, I guess I should put in a disclaimer, these are just hardening features. They will help. They will not save you if you managed to introduce a truly bizarre security vulnerability, but they will help. So let's start with a simple example of a security vulnerability, and then I'll tell you about some of the hardening features we've introduced which will greatly improve the ability of developers to-- greatly reduce the ability of the attackers to exploit this, well you know, program.

So here's a simple example. This example has a bug. You can see that it's taking some untrusted data from the network and then it's doing some processing, very simple processing, it's upper-casing it and printing it out. The problem is that the data that's taken from the network appears to have an arbitrary size link, it's taking a size T, but the buffer that you can see here is a fixed size. Thus, when it copies the data from the network to the buffer, the buffer might not be large enough to hold all of the data so we would have a buffer overflow.

In this case, the buffer's on the stack so it's the stack overflow. So what happens when the stack overflows? Well, the local variables are stored on the stack. Below, the other information like the saved registers, the return address for this function, the arguments to the function, and so on.

So, if a string is passed to this routine that's too long, it will start by filling up the buffer, then it will keep going. It'll overwrite other local variables, it'll overwrite saved registers, it'll overwrite the return address. An attacker can insert their own data into the return address and point it at some place to do what the attacker wants, which probably is not what you want.

So, in-- since Leopard and now on by default in the compiler in Snow Leopard, we used-- we've added an additional form of protection. We called it stack canaries and you can see we have the canary there on screen. The way this works is that the canary is a word which is randomly chosen as your program starts up so the attacker can't predict it.

So when the attacker attempts to use a buffer overflow, like this, the attacker can't reliably overwrite the canary with the correct value. The canary will-- and the routine when it returned, will then be able to check it whether this canary is the correct value and if and unfortunately it's not, the-- your program will be stopped so that the attacker can't continue to exploit it, in particular the return address doesn't get used.

So what does this look like when this has happened to you and your program? Possibly you've got a test case which has actually triggered a bug of this nature. Well, you'll see a crash report. Your program will call abort and hopefully you'll be watching for crash reports in your test week. You'll notice that the crash report does not appear to be very distinctive other than noticing that it calls abort.

The reason for this is that, actually it turns out the routine that stops your program calls abort as the very last thing it does. So you can see here on screen we have some symbol plus 0 in a crash report. Invariably what this means is that your crash occurred at the last instruction of a routine and that really means it's the first instruction of the next routine.

So to diagnose such a case, what you want to do is find the shared library that was-- that contains the symbol involved and look to see what symbol was immediately before it, which you can do with a command like this one on the screen. You can see that in this case we have release_file_streams_for_task which doesn't appear to have anything to do with anything but immediately before it is stack_chk_fail and that immediately tells you what went wrong, the stack check failed. You can see which routine contained the buffer that had the overflow simply by looking one down the stack.

You'll see in this case it's print off of the routine from the previous slide. The compiler doesn't automatically test for every-- it doesn't automatically add stack protection to every single routine. If it did that, we would no longer be able to say that it had very low overhead, but this does mean that occasionally you might discover you have a routine with an overflow that doesn't contain the-- You might occasionally find that you have a routine with an overflow but it didn't get a stack canary added to it.

If you want to-- if you have a piece of code that might be particularly dangerous, you might want to add more stack overflow checking and you can do that with this flag, this stack protector or flag. That will ensure that every single routine in your program will get a stack canary, whether it appears to need one or not.

The disadvantages, this has an increased performance impact, but the advantage is, you get much greater security. So that was stack projector. But, so where does the attacker go from there? Well, so the attackers have overwritten the stack and I said that the return address was going to be redirected somewhere to do something bad but where exactly? I mean, chances are there isn't already code in your program to do whatever it is the attacker wants to do.

Usually the attacker wants to execute their own code. Well, in many of these cases, the attacker has an easy way to get code into your program already. In this case, it's especially simple, the attacker is already overwriting a large area of the stack so the attacker can simply redirect the return address to the stack and place their executable code there and, or pass the return address if there wasn't room.

So, we have added the feature to Leopard and later that prevents this under certain circumstances and it's very simple. On the stack, code is not executable, likewise on the heap and so on. But this feature is only available in its full form if you compile your programs 64-bit. So on Mac OS X, the stack, the static data and the heap, so that's everywhere that you would normally have writeable data, are all protected on 64-bit PowerPC and 64-bit Intel.

On 32-bit Intel, the only protection is on the stack so if the attacker can force the code into the heap, then they can still execute that code and on 32-bit Intel, there's no way for you to explicitly protect the heap. So we recommend that you make sure you build a 64-bit version dof your program. Not only do you get these benefits, you get a whole pile of other really nice benefits like it runs faster, it can use more memory, but also it's more secure.

And if you have a situation where you actually wish to generate code, so maybe you're writing adjusting time compiler, maybe you're writing something that does conveniently-- that does graphics processing, generating code for that. So you can use the mprotect system call to make data executable when necessary. On iPhone, the situation is even better. On iPhone, no data can ever be made executable.

All code must also be signed so that the attacker would have to download sign code and even then they still can't make it executable without calling into the system. And in addition, on iPhone, contractually, no interpreters are allowed except for the system JavaScript. So the attacker can't even get around it by using an interpreter you might already have in your program to interpret some form of byte code, for example.

So all these together make it very hard to exploit a system. So-- but we still want to eliminate all these buffer overflows, and another common case, if you didn't write a routine yourself that contained the loop, is to misuse a library function. Take this example here. We have some untrusted data, we're copying it into a fixed sized buffer.

This does just the same thing as the previous routine. So what we'll want to do-- so you shouldn't write code like this if possible. A better routine to use is the strlcpy function that takes the size of the buffer you're copying it into and there are similar equivalents for every routine in the C standard library.

For example instead of sprintf, you should use snprintf. A disadvantage of this particular case here is that strlcpy will truncate the buffer. So, you have in particular-- so if the string is very long, strlcpy will just cut it off at the end. The problem with this is that there might be important data.

For example, if you are logging this information, the information vital to tracking down where this attacker is coming from might be past the end. So a better alternative is to, you still use strlcpy but check its return value. A strlcpy returns the size that it wished to copy. So, if it asks to copy-- if it wanted to copy more data than is available in the buffer, it will return such a value. You can check for this and handle the error and that's our recommended approach towards string functions.

But what if you have a large program already and the program-- you don't really have time to check everything in the program, all of these with the string functions, and even if you did check everything, you definitely wouldn't have time to fix it. Well, in Snow Leopard, we've turned on a feature by default that helps with this. The compiler will attempt to determine the size of an array, so in the previous slide we had an array called buffer.

If it can work out what size the array is, it will replace the standard library function with a new version with _chk at the end of the name. It will pass to that routine the size of the buffer and the routine will determine-- will ensure that it never writes past the end of the buffer. If it appears it needs to, the routine would stop your program rather than allow it to be possibly exploited by an attacker and you'll get a crash report.

Again, it will call abort. In this case it will call a chk_fail from the routine named with _chk at the end. So it's a very distinctive crash report pack and you'll see which routine in your program called the offending string function. You can see here, strcpy_chk was the routine that-- was the routine that would have overflowed.

So that helps with stack and heap issues but it's still possible to write your own, but we don't have canaries in the heap of the same kind that we have on the stack. So we've managed to do something similar. For-- here's an example of an attacker overwriting a buffer in the heap.

You can see there's a buffer and then some critical data after that. The attacker, because of heap execute protection, can't simply insert their own code into the heap, but if you have critical data in the heap that controls access controls, permissions whether the data is trusted or untrusted, then the attacker can still attempt to overwrite that, make their data which is coming from the network or untrusted place appear to be trusted.

So the attacker has a buffer, overwrites past that, past other stuff on the heap and overwrites the beginning of your critical data. To do this, the attacker might have to overwrite several blocks on the heap because it's often difficult for an attacker to precisely control the heap layout. Some of these will be free blocks. All, on Mac OS X, all free blocks contain a small 4-bit checksum that prevents-- that attempts to detect heap corruption.

So, and in addition, the attacker is going to want the program to continue running. He has overwritten your critical data but he still can't actually exploit your program until you've managed-- until he is-- until the critical data is actually used and that might be quite a bit later in the program.

So the attacker wants to make sure that memory allocation continues to work. In previous versions of the operating system, this was somewhat easy. The attacker could simply ensure that the checksum was preserved. But in Mac OS X Snow Leopard, we have improved the heap checksum. It's now slightly randomized.

So now, the attacker has to manage to guess the correct value of the checksum and if the attacker fails, again, your program is terminated. This is an especially visible-- helpful in a web browser. It's highly unlikely a user is going to sit there and wait for an attacker to attempt to guess the right value restarting the web browser every time it crashes. Eventually the user will realize that this side is doing something wrong, hopefully, and will go and will stop trying to access it.

This works on 64-bit Snow Leopdard by default. On 32-bit Snow Leopard, you-- the heap checksum is still randomized but your program is not stopped so the attacker can still possibly exploit it. You can turn on the program termination on a memory error within environment variable. OK. So another feature we've added in Snow Leopard is administrator versus root.

The first account created by default, when you install Snow Leopard for the first time or a Leopard, is an administrator account, that is the account that can control the system. And on many systems, only one account is ever created this and will use the systems. The problem with this is that it means, well, every account is an administrator.

The design goal for administrator accounts and the reason we have this administrator-root separation is that what we would like to have happen is that administrators have no extra privileges except that they can use their password to become root. This adds significant protection against malware because malware doesn't know the user's password and a prompt in the middle coming from nowhere that you need to enter your administrator password to perform this action is a sign that something is possibly wrong.

Unfortunately, because of backwards compatibility, historically many system directories are writable by administrators, the root directory and so on. On Snow Leopard, we've tightened this up a bit In particular, directories inside system applications, so for example the plug-in directory in Safari is no longer writable by administrators. We also managed to change the ScriptingAdditions directory inside /Library. In future versions of the operating system, we do intend to tighten this up more. In the long run we would really like the previous-- the goal stated on the previous slide that administrator has no extra privileges.

Unfortunately, we couldn't do all of this at the Snow Leopard and the principal reason was installers. A number of installers-- what an installer should really do if it has to install stuff inside system directories is authenticate as root using authorization services. Unfortunately, some legacy installers didn't do this. They relied on being run by an administrator, which is also something that you don't really want to do, and just copied files directly into this-- into the root directories.

So it's particularly important that installers be written to properly authenticate this root and that will have the advantage that it means non-root, non-administrator users can install your program by finding an administrator, dragging them over to the machine and having them type their password. OK. So, now let's talk about setuid programs. So set-user-ID programs are a UNIX concept back from the dawn of time or the '60sstyl`Q in which a program can be given additional privileges simply by being executed.

This sets up a trust boundary between the user and the setuid program. But there's a problem with setuid programs that makes them difficult to write. The user has a user which is not as trusted as the setuid tool, has a lot of access, and if that is an attacker, the attacker can get to the setuid tool in a number of ways.

For example, through usual I/O, right, standard input standard output files, through the command-line arguments passed to the tool, through environment variables, through the working directory, through file descriptors, through the file mode mask, through interval timers, they can set them up and have them go off after the program is running, through the signal masks signals can be blocked, through Mach ports and so on. That's a lot. And it turns out, it's really hard to write a program that defends against every single one of these attack factors.

So, what to do? Well, the solution that we found that works pretty well is, don't use set-user-ID. In Snow Leopard, a number of system applications are no longer using setuid. In particular, some frameworks that cannot safely be used as a setuid program, like AppKit, will refuse to start up a program setuid.

If you have a legacy program that is marked setuid and uses AppKit, AppKit will prompt-- will drop all privileges on start and then prompt for an administrator password. If the user successfully authenticates, the program will then continue to run as the setuid user. So this preserves backwards compatibility in an inconvenient kind of way but ensures security. Your executables generally should not be setuid, especially if they're using AppKitt or any high-level system library.

Running a setuid program is incredibly hard and there are better alternatives. For example, one alternative that we found works really well is to split your application up into the application which can be untrusted, use the GUI, and a daemon, which can be written using only a constrained set of interfaces and carefully audited. So, but how to solve the problem? You don't really want to make the daemon set-user-ID that just repeats the same problem. So, the way, the technique that we found that works really well is to use launchd.

Launchd is a system service that, when your application opens a socket or tries to use Mach service, can start your daemon in a controlled environment with environment variables, file descriptors, and all the other things that are listed on the previous limited to just-- limited to a standard configuration. Then your application can communicate with the daemon through the socket or Mach service. So there's only one vector into your daemon, stream I/O. If you're interested in learning more about launchd, there's a talk tomorrow.

And it turns out, it's perfectly possible to do just about everything this way. In fact, on iPhone, there are no setuid programs. Everything that needs privileges is started up through launchd or uses authorization services or another mechanism. OK. So, now let's talk about another smaller hardening feature that we've done in Snow Leopard that was prompted by a particular high-profile attack.

So, there's a network protocol called DNS, the Domain Name System, whose purpose is to look up names-- look up IP addresses given names. The actual details of the protocol aren't terribly important, other than to know that it contains data which sometimes is trusted or probably shouldn't be but sometimes is.

The key thing about the attack is the way the Domain Name System protocol works. It operates using a UDP, the datagram protocol. The application creates a query, sends it off to the server, the server adds its answer into the query and sends it back, so not very unusual for any kind of datagram protocol. The problem is that when the application needs to verify that this was the correct response, all it can really check is that it was sent to and from the right place. It has the right host and port, source and destination.

It can check that it was the answer to the question that's sent, and it also checks that there is a small query ID, a 16-bit query ID and it can check that that query ID reflects to the right question. The problem is, an attacker who wishes to substitute their own answer can simply create an answer while the remote server is creating it, send it back, hopefully faster than the remote server, and if it gets there first and contains all the right values, then the application will take it as the correct answer. The attacker knows many of the values.

I mean it knows what-- the attacker knows what host is being attacked and what the correct DNS server is and what question was asked because the answer to that question is what they're trying to substitute. So the only remaining parts of the packet that the attacker might have difficulty creating are the query ID which as I said is 16 bits so the attacker might have to send about 60,000 packets to be sure of getting the right query ID, but that's not hard, 60,000 packets isn't a lot of data, and the source port.

This attack really only works, really only works sufficiently if the attacker can predict the source port. Historically, the source port was simply created sequentially. So if the previous source port was 10,000, the next one would be 10,001. To fix this attack in Mac OS X 10.5.7 and Security Update 2008-09, we simply randomized the source port. That doesn't completely prevent it but it means the attacker has to send about a billion packets and that is a substantial amount of data.

Eventually, of course, the solution is to use a cryptographic authentication like DNSSEC and that will completely prevent the attack because the attacker cannot, hopefully, write the cryptographic protection. So, that fixed DNS. We we're concerned that there would be other protocols that used a very similar query response framework sending out packets, something like that.

So on Snow Leopard, UDP ports, the UDP port number is now randomized for all protocols. There is a potential gotcha here, which is that previously ports were allocated sequentially, now they're random and so you might get a port reused faster than you would previously. So if you do have a protocol like this, you might want to watch for that particular issue.

So, earlier I talked about some features that were on by default. One feature we've had on by default since Leopard is system library randomization. The idea here is that you have a-- is that even if the attacker can't inject code directly into your program, knowing the layout of the program, the attacker might be able to chain together a sequence of operations that nonetheless does what he wants.

So, code layout randomization protects against this to some degree. Code layout randomization is not on by default, not even in Snow Leopard, not even 64-bit but you can-- But system library randomization is on by default and you can arrange for your executable to be protected using code layout randomization.

Some system executables are also adopting code layout randomization in Snow Leopard and you can turn it on in Xcode by setting these three settings. You need to specify -fpie for both linking and compiling and you need to make sure that your code is not explicitly generated-position dependent. OK. So, all of these as I said help and-- but they're not perfect protection.

So suppose that you still have a library in your application, an old library, maybe it came from a third party, maybe you've had it around for a long time. It's performance-critical possibly, you can't really rewrite it, it's got-- it has the possibility of having a security vulnerability and worse than that, it also processes untrusted data that comes from the network perhaps. So, if an attacker can attack this library, the attacker can obviously control its behavior but since your application is in the same address space, the attacker can't control your application.

And because your application is running as the user, the attacker can also expand his control to take over the entire user account. So to prevent this we have since Leopard a sandboxing feature. And by-- And the sandbox feature is by the way on by default on iPhone. You as a developer can turn this on in your program using the sandbox_init API.

There are five sandboxes available but four of those are less recommended. The one that you really want to be using is the pure computation only sandbox which prevents all access to system functionality except for reading and writing to files that are already open. This is ideal for, for example, a file format converter. You might have a legacy file format and the code to interpret it that was written a long time ago by people who, you know, are no longer willing to maintain it and don't have the time if they did.

And their format might still be accepted from, for example, the network. So you can put your converter into a sandbox. A separate process that doesn't-- that is limited to pure computation only, your application can communicate with that through a pipe or a limited communication method and you can carefully audit what your application does-- what your application will accept from that pipe.

That way if an attacker attempts to control the behavior of the library because it has a security vulnerability, well they can do that but they can't escape the sandbox to control your application or take over the user's account. In the future we may be able to add custom sandboxes but they are not yet available in Snow Leopard. Anything that you might see in Leopard is-- in Leopard or Snow Leopard, along that line, is not a documented API yet.

One example where we've used this functionality is the H.264 Decoder library that takes in a movie from the internet possibly or some other untrusted source. It's got to be very high performance. The code is, and as a result the code is now very complicated and we've fixed a lot of security vulnerabilities in it.

Hopefully, we've got them all but there might still be one or more. And, but we now have it inside a sandbox so all an attacker can do who can supply a movie is control what the movie looks like, which they could have done before. And so, now I'll be handing you over to Drew to talk about crash reports.

Thanks Jeff.

Hi I'm Drew Yao and I'm going to be talking about crash exploitability triage. What that means is being able to quickly and accurately determine whether or not a given crash represents a security issue, or I mean exploitable security issue or just a crash. This is important because we need to be able to determine whether or not it's exploitable because we need to prioritize the exploitable ones to be fixed much quicker. So I'm going to be talking about doing it using both automated tools. There's a tool named CrashWrangler which we've developed that I'll be describing. And also using manual means either crash logs, looking at crash logs, or by looking at the processor-- the processor and the debugger.

Before I talk about the how of how the crash triage works. I want to talk about the why, at least the original motivation for developing CrashWrangler and that's fuzzing. Fuzzing is a technique in which we use automation to feed malformed input to the program and try to get it to crash. Fuzzing is a primary tool for hackers because it's a black box technique. It doesn't require any source code.

It doesn't require any debugging symbols and it's fairly easy to create and to run fuzzers. Attackers are actively fuzzing many commercial products. We know that many of the reports that we get that are external that are security issues were found by fuzzing. By using fuzzing, we can find the same or similar bugs and fix them, often even before the product is released.

So to give you a more concrete example of how fuzzing might work, at least an example, is let's say we want to find bugs in QuickTime. What we can do is download a movie file and for each of the files, we apply some noise to some field in the file by which I mean we'll write some random bytes over an area in the file leaving most of the file valid. So there's just a little bit that's malformed.

And it's bad enough that it might break stuff but good enough that it can get by many validation checks. So for each of these files, what we can do is send it to QuickTime by playing it in QuickTime Player and then it will either crash like this one or it won't crash.

Some crash, some don't. Once we have these three crashes, we have some challenges. One is we need to figure out which of these crashes are duplicates, that is we only care about the number of unique bugs that we have found. It's quite possible that many of the test cases that we generate actually all trigger the same bug. So by figuring out which ones of these crashes are duplicates, we know how many unique bugs we have. We also need to know which ones are exploitable because we need to prioritize them.

The exploitable ones need to be fixed much sooner. When we just had those three bugs it was not that big of deal, but if we have these many, many, many cases or many crashes like if we have say a thousand crashes, we might generate using a hundred thousand test cases then it becomes much more difficult to handle those challenges that I mentioned earlier, determining duplicates and determining exploitability. So some kind of automation was necessary. So this is the problem.

CrashWrangler is the solution. CrashWrangler is a tool which runs on Mac OS X 10.5 and 10.6. It doesn't currently run on iPhone but I will be describing some manual techniques that you can use later in the talk and these do apply to iPhone. CrashWrangler automatically handles those challenges I mentioned earlier.

It determines exploitability and whether or not it's a duplicate. It uses heuristics which is a fancy word for guessing, so it's not 100 percent accurate. It may think something is exploitable when it's not or vice versa. But overall I found it to be about 99 percent accurate. And it will be made generally available at some point in the near future.

That is to anyone with an ADC account. It should be on the attendee site for the session now or at least sometime in the very near future. How it works is it's a Mach exception handler which forks and execs a process and if the process-- the child process crashes it will examine the state of the process at the time of the crash and output a log file which contains some diagnosis of the exploitability of the crash. Once we have a few of these log files, then there's a script which can analyze logs to determine which ones are duplicates.

As I mentioned earlier, it's useful for fuzzing but it's also useful for simply screening like a one-off crash. Let's say you get a report of a crash and it is reproducible, what you can do is just use the tool to automatically quickly determine whether or not it's exploitable. The benefit of this is that it doesn't require any lengthy debugging. It doesn't require the person who wrote the code to actually figure this out. So it can be done by a screener or a QA person.

So now I'd like to do a demo. OK. The first thing I would like to show you is just-- I'm going to configure it using an environment variable called CW current case and we'll just call this foo and I'm running an exception handler named Exception Handler. And I'm passing it the argument of a program name that's called DIV 0 and DIV 0 is just a very simple sample program that causes a crash by a divided by 0 exception.

So when I run that, it prints out the-- that trace of the crashing thread and it prints out some diagnosis information. One thing we note here is that the name of the crash log is foo.crashlog.txt which came from here. And we see here that it's marked is exploitable=no. I'll do the same thing but I'll call it foo2. And just the name of the crash log is different. And then I'll do one more.

This one is considered to be exploitable because of the fact that it aborted in free which indicates maybe some heap corruption was detected and now that I have these, I can run a script which detects whether or not they're duplicates. So, this one stands by itself. These two, foo and foo2 are marked as duplicates of each other.

When we have a tool like this it's no fun to just run it on a sample test program.

So I also have a real test case. It's running a movie file in QuickTime Player, oops, which causes a crash. It's not considered exploitable. And each of these crash logs that come out is basically it has this header at the top which includes the is exploitable field and then following that is just simply a normal crash reporter crash log.

So, how that script that I just ran works is it has some magic in it but basically we'll be doing something like this.

Setting the log name to be a certain-- the name of the file running exception handler and passing the name of the program and passing an argument to the program.

Or actually you can see that it's just running the program normally.

As soon as it crashes-- ah, I have to remember to play it. As soon as it crashes it prints out the same-- the stuff, the information. So it's pretty simple to run and it can be used for fuzzing very simply or just for one-off crashes.

OK. Back to the slides. Once we have this tool, we also-- it's still useful to know how to do the same things manually. Both because manually there's different things that you can do that an automated tool can't do and also because, for example, you can use manual techniques to examine an iPhone crash.

When I talk about exploitability up here, I'm only talking about arbitrary code execution. I'm not talking about any other bugs like denial of service or information disclosure. The reason is that arbitrary code execution is one of the easiest ones to detect simply and without access to source code. But what we're looking for is just a very limited number of things.

And I also should mention that if the crash is triggered by trusted input, it's not considered exploitable. That is, let's say for example we have a buffer overflow that can only be triggered by the root user. Even though it can be used for arbitrary code execution, it's not considered to be an exploitable security issue because the root user can already do whatever he wants anyways.

So that's just something to keep in mind. When we're talking about exploitable crashes, we're pretty much always talking about memory corruption. There are other security bugs like some shell injection, for example, that don't involve memory corruption, but when we're looking at crashes, it's almost always memory corruption, which I'm going to define as writing where you aren't supposed to be able to write.

This includes be-- or writing past the end of the buffer with a buffer overflow or writing to an arbitrary pointer, maybe you control it, maybe you don't, or including writing to freed memory. Because if the memory is freed, then maybe another thread will allocate it and will be using it for something and you'll be modifying the behavior of the program in unexpected ways. Memory corruption often allows the attacker to alter the control flow of the program.

If the attacker can overwrite a function pointer or the return address on the stack, the attacker can get arbitrary code execution and even just by overwriting a variable in memory, for example, if we have a server that has a variable in memory that determines whether or not a given connection is authenticated, then if the attacker can overwrite that variable in memory, he can cause the program to think that his session is authenticated and thus gain privileges that way.

It's important to note that any memory corruption is considered to be exploitable, not just controlled buffer overflows where you control exactly the amount that gets written and the values that get written. So, for example, also very large buffer overflows where you don't control the number of bytes that get written or byte swapping after the end of a buffer where you don't control the values that are getting written or bzeroing, writing no bytes, same thing, or using after free.

Even though maybe you don't control who else is writing to this buffer that has been freed, it's still maybe possible in some instances and we found that attackers are very resourceful in exploiting any kind of memory corruption and a lot of times there were bugs that people thought were not exploitable and they turn out to be.

Once we understand that we're looking for memory corruption, we need to know how to determine if a crash was caused by memory corruption. We're going to divide the crashes or the types of crashes into three classes. The first one is crashes where it's very likely that they were caused by memory corruption.

The first is stack corruption. In the case that the return address has been overwritten, we're looking for executing an invalid address. So first we're looking for the question marks on the left indicating that the module that the-- of the instruction that crashed was not known. It's not any known library or executable name. On the right, we see that there's a zero plus some number which indicates that symbolication could not figure out what function the crash was in.

And we also see here that the access address at the top, that is the crash that caused the-- or the address that caused the crash, is the same as the executing address here at the bottom. They're both 41414141. And thus we see that we're executing an invalid address.

Specific to stack corruptions, we also are looking for a stack trace in which there's only one entry. And the reason is that if the stack has been corrupted, it's going to be impossible to get a proper back trace. If we overwrite a function pointer then it's possible to jump to some code that's invalid and this will result in an illegal instruction exception.

Geoff mentioned the -fstack-protector flag earlier. In the case where you have a very, very large stack buffer overflow, what it looks like is this. There is an abort-- there's a stack_chk_fail and a crash in strlen. And what is happening there is that because stack_chk_fail calls this log and eventually the strlen is being used on some corrupted memory causes the crash. But if there's a relatively small overflow then what happens is you'll see an abort in release_file_streams_for_task in Snow Leopard anyways. As Geoff mentioned, there's a bug in the symbolication and so it happens that release_file_streams_for_task is immediately after stack_chk_fail.

On Leopard, they function immediately after its append_int, so that's another one to look out for. Geoff also talked about the _FORTIFY_SOURCE where the standard C Library functions will be replaced by one with the _chk suffix and it will abort if a buffer overflow would have happened. So what we're looking for here is an abort with some standard C Library and then our standard C Library function and then _chk.

There's also a MallocCorruptionAbort which is new in Snow Leopard. It's on by default in 64-bit and it can be turned on on 32-bit with an environment variable, MallocCorruptionAbort. And it aborts if heap corruption is detected and that would indicate that the heap has been smashed by some kind of buffer overflow or something.

What we're looking for here is an abort in szone_error and we also see looking at the application-specific information that it indicates the some freed object may have been overwritten. In general, any crash in malloc/free or the Objective-C runtime indicates that some kind of heap corruption has happened. If we assume that these functions don't have any crash bugs in themselves, the only possibility is that heap was corrupted. Specifically, with objc_msgSend one common cause of crashes is, well, heap corruption but also if the object was released and then sent a message and some new stuff happened to be there.

This is considered exploitable if the input-- if it's triggered by untrusted input. And the reason is that having-- a method called an Objective-C is basically through a few layers of indirection, it's a call through a function pointer. So if the attacker can control the object that message get sent to, the attacker can get arbitrary code execution. A write to an invalid address would indicate that if the attacker could control the address that gets written to, number corruption would be possible. It's not possible to tell if the crash was on the write by looking at the crash log.

What we can do is in a debugger you can look at the disassembling of the program in GDB. You can do x/i $pc, which means examine one instruction at the program counter.

And you kind of have to know assembly language to know whether it's not-- it's a read or a write, but a simple mnemonic for Intel is we're looking for the parenthesis around the right operand, the operand on the right. So if the right operand has parentheses, then it's a write access, W-R-I-T-E, write. Executing in a valid address would also indicate that some kind of function point or something has been overwritten. It's very similar to what I mentioned earlier with the stack corruption.

We're looking for symbolication failing and accessing the same address that's executing. But unlike the stack corruption we have a valid stack trace. Calling invalid address is very similar but you can't, again, you can't tell that this was happening just by looking at the crash log. So you need to do a disassembly. And what we're looking for here is a call instruction where we're calling an invalid register or a register pointing to invalid memory.

Another very common indicator of heap corruption or memory corruption is when you get a different crash for the same test case, a different crash each time you run. And this is because each time you run the test case, it's overwriting something different because of different timing or whatever. So each time you run it, something else has been overwritten and it doesn't crash immediately.

It crashes some point later when that thing that was overwritten gets used. To help find out where the real bug is, it's very useful to run with libgmalloc a.k.a. Guard Malloc which I'll be talking about more later. There's another class of crashes where they're very unlikely to be caused by memory corruption. One would be divide by zero.

Very simple, you're looking for the arithmetic exception. Another would be recursion. When every time you call a function it pushes a little bit more data onto the stack which grows down in memory, eventually, if it grows too large it will hit a guard page and crash. Since there is a guard page protecting the rest of the address space, it's not considered to be a memory corruption crash because it's guarded with all those crash, at least in most cases. So what we're looking for here is a very long stack trace with repetition.

That is, we see here foo is repeated many times. One thing to look out for with the recursion stack overflow is that if you do a disassembly of the crashing instruction, it's common that you'll see a crash on a call instruction and you might be tempted to think that it's calling an invalid address and it's an exploitable issue. But what you'll notice in this case is that we see here it's calling foo which is a valid function. So what you need to understand about the call instruction on Intel is that it has two parts.

The first part is it pushes the return address onto the stack and the second part is it calls or it jumps to the function that it's trying to call. If it's crashing on the first half, they're pushing onto the stack then it's not an exploitable issue, it's just recursion.

So the way that you tell again is at the top, the non-exploitable one is calling a valid function. At the bottom you see the-- it's calling with a register, that's EAX here. In general, aborts are not exploitable. Aside from the ones that I've mentioned earlier, the -fstack-protector, the _FORTIFY_SOURCE, and MallocCorruptionAbort, so yeah, just any other one would generally not be exploitable.

There's also another class where they are considered to be improbable but more investigation may be warranted. One would be a null dereference, we're looking for an access near zero or at zero. One thing you have to look out for is that you-- if there's accessing an offset of a null pointer you need to make sure that the offset can't reach outside the bounds of the null page. That is on 32-bit, there is a page defined at the zero address that has a protection such that you can't read or write to it. On 64-bit, there's a whole 4-gigabyte area near the null address where you can't read or write to it.

If we have a null pointer here called buff and we're accessing the offset of it that is attacker controlled, it may be possible to write to any address, zero plus some crazy number. So this one could be exploitable. The second one is-- would not be considered exploitable because it's accessing the offset zero of the null pointer.

And the third one, accessing the member of a struct usually would not be exploitable because a struct would not be big enough to reach outside of the null page or the null area on 64-bit, but it may be possible with a sufficiently large struct. A read out of bounds is again usually not a memory corruption issue. For Intel we're looking at the left operand having parentheses around it. They are potentially exploitable though, for example if we had some kind of memory corruption that overrode a pointer and then we read from the pointer, well the crash was caused by memory corruption.

So a good way to quickly check if it might be memory corruption is to run it again with libgmalloc which again I'll talk about later. There is also a potential information disclosure bug with the reading out of bounds which is if you're reading some memory that you shouldn't have access to and then you send that buffer to the attacker, then the attacker has some information he shouldn't have access to.

This is generally difficult to diagnose very quickly, so enough said about that. I should also note that overread, overreading followed by overwriting is possible and that it should be considered exploitable. For example in this code sample here, we have a buffer that's allocated a certain size and we're byte swapping past the end of it.

What a byte swap is, is it's reading from a certain address in memory, flipping that data in the register and then writing it back out to the same address. So because there is writing happening, this would be considered memory corruption and this is something to look out for. There is an option for libgmalloc called MALLOC_ALLOW_READS which I'll be talking about later which helps deal with this issue.

When looking at CrashWrangler versus Manual Assessment, there are-- CrashWrangler can pretty much do everything that Manual Assessment can do in terms of determining if the bug is a memory corruption bug or not. What it can't do is to determine the trust level of the input which is just something that the user should be able to tell and it also can't diagnose other kinds of security issues other than memory corruption. There are also some memory tools which are useful in doing crash triage and these are listed in the malloc main page.

One is MallocScribble which writes-- when you allocate a new buffer on the heap, it will fill the buffer up with OXAA bytes and when you free a buffer, it will overwrite the whole buffer with OX55 bytes. So if you get a crash accessing OXAAAA or OX55555, then you kind of understand the cause to some degree.

There is also MallocGuardEdges which puts a guard page after every large buffer and so if you write past the end of the large buffer, it will crash immediately and you'll know what's going on there. And MallocCorruptionAbort which I mentioned earlier which crashes if a heap corruption is detected. Now all of these are fairly lightweight, so when doing testing, it's possible to just leave them on all the time.

libgmalloc is a much more heavyweight tool which you could check out the main page for, it's a.k.a. Guard Malloc and what it does is it puts a guard page after each allocation and it also unmaps each allocation when you free it. So when you access either after at the end of the buffer or if you access freed memory, it will crash immediately. This is as opposed to the cases where memory corruption happens but it didn't crash until much later.

The downside is that it's extremely slow, approximately 20 times slower than running normally so you wouldn't want it all the time on all the time and you also probably wouldn't ever want it on all the time when doing testing but you can use it when there's a question, when you're unsure about the exploitability of an issue and this will usually be in the cases that I described last which were the probably not exploitable but maybe.

I also should mention the MALLOC_ALLOW_READS environment variable. What that is is it sets the page, the guard page, after each allocation to be readable but not writable. So in the case of the byte swapping for example, if you read past the end of the buffer, it won't crash but then as soon as you write pass the end of the buffer it will crash and since you see the crash on the right, you'll know that it is a memory corruption issue that's potentially exploitable.

To summarize the whole process for working through the exploitability, if it's if the crash was caused by stack corruption, illegal instruction, or crash on execute or write, we'll call it exploitable. If it was caused by divide by zero or recursion, we'll call it not exploitable, probably. If it was caused by a crash on read, null dereference, or abort, the first thing we'll do is look in the stack trace and try to see if there are any of those magic functions in the back trace of the crashed thread, for example malloc/free or stack_chk_fail, and these would indicate that it is potentially exploitable.

And I should note that with regards to the hardening features like stack_chk_fail or _FORTIFY_SOURCE, even though the overflow was detected in this case, it should still be considered exploitable because there may be some way to work around the hardening feature or something like that.

It's still a very dangerous thing to leave in your code so it still would be high priority to fix. If the crash did not have any of those functions in the stack trace, then we can run it again with libgmalloc. If it crashes with the same crash that is crash on read, null dereference or abort, then we'll just call not exploitable.

But if it crashes with the different crash, then we'll run through the whole process again. So overall, this is not that complicated of a procedure but I found it to be pretty accurate, about 99 percent accurate. So in summary, what we're looking when we're looking for exploitability is memory corruption and all memory corruption bugs should be considered exploitable.

It's very difficult to prove that a given memory corruption bug is not exploitable. For any given program, just the fact that we're writing to stuff that we're not expected to be writing to, to really ensure that this can never be exploited for anything is quite difficult. Memory corruption can be detected by inspecting a crash log, by inspecting the process in the debugger or by using CrashWrangler.

CrashWrangler is a new tool that will be released soon generally for general availability and it's useful for fuzzing and useful for triaging one-off crashes. It's-- when you're doing this crash triage stuff, it's useful for to use the malloc tools, libgmalloc, and the hardening built flags like -fstack-protector and __FORTIFY_SOURCE. CrashWrangler is now or should be soon available at the page for the session and as always, product security can be contacted at [email protected].