Sunday, January 14, 2018

Forensics Quickie: Methodology for Identifying Linux ext4 Timestamp Values in debugfs `stat` Command

FORENSICS QUICKIES! These posts will consist of small tidbits of useful information that can be explained very succinctly.

I saw this tweet from @JPoForenso recently.

Archived tweet here.

I didn't know what this was either, so I began testing. And as always, remember that it's one thing to know that an artifact exists; it's another to know how to find, understand, and make use of it. This is more of a methodology post related to problem solving, but having the logic behind the approach is typically pretty useful.

So, from start to finish, let's delve into how you'd go about answering the question Jonathon posted.

You want to determine what a specific part of the debugfs `stat` command refers to (but this process can be applied to any other Linux command!).

The Solution
My first thought on how to approach this question was to see if we could get lucky by looking at readily available source code. Since verifying program behavior from source code has proven to work several times in the past, it doesn't hurt to look in this instance. So, knowing that the Linux kernel is open source, I first Googled something fairly simple, just to see what I'd get:

Googling "debugfs stat source code"

The first result is what seems to be the stat.c source file, related to the standard `stat` Linux command. It's not exactly what I want, but it's close enough for now. As we open the link, we can see a bunch of Linux kernel versions on the left sidebar. I didn't know what kernel version I was running on my Ubuntu virtual machine, so I ran a `uname -or` via command line to find out.

Running `uname -or` to get the operating system and Linux kernel version

Looks like I'm running kernel version 4.10.0-42-generic. Since I'm going to be running my tests on this Ubuntu VM (I already had a clean snapshot), I'd like to make sure I'm reading the stat.c source code for the closest kernel version I can get. And since we have the luxury of picking the version of the source code using this first Google search result, I'm going to go ahead and select version 4.10 from the left sidebar on the page (direct link here).

Looking at the source code, I can already see that it has a lot of the same strings that we see in Jonathon's screenshots (and just by using the `stat` command in the past) -- namely, I can see references to atime (access time), mtime (modify time), and ctime (change time).

stat.c source code showing references to relevant timestamp variable names

As I Ctrl+F for "ctime" within this source file, I noticed something at line 389: the added "nsec."

stat.c source code showing references to "sec" and "nsec". Possibly referring to nanoseconds

As I looked back at Jonathon's screenshots showing the output of both the debugfs's `stat` command and the standard `stat` command, I noticed that the standard command was showing the timestamps with nanosecond granularity. The theory at this point was that the characters after the hex bytes and the colon were probably the encoded nanoseconds value. To confirm this, I ran some tests.

I needed to further ensure that my setup was the same as Jonathon's; I needed to make sure my Ubuntu VM was using an ext4 file system. First, I ran `sudo fdisk -l` to list the disks and partitions present on my VM.

`fdisk -l` output showing /dev/sda1 as the boot partition and that it is of type 0x83.

We see that /dev/sda1 is our boot partition, that it is the largest partition, and that it is of type 0x83. Brian Carrier's "File System Forensic Analysis" book tells us in Table 5.3 (Chapter 5 > DOS Partitions) that a partition ID of 0x83 is, in fact, a Linux partition. But as this web page suggests:

"Various filesystem types like xiafs, ext2, ext3, reiserfs, etc. all use ID 83. Some systems mistakenly assume that 83 must mean ext2."

Therefore, this is not enough information and we will need to learn something new. With some quick Googling, we can see that the `df -T` command will get us the information that we need to confirm we're running an ext4 file system.

`df -T` output showing that /dev/sda1 is running an ext4 file system

Now that we know our setup more or less matches the original, let's run a quick and dirty test to get a feel for what Jonathon was doing. First, I'm going to do a quick `ls -la` in my home directory to see what existing file I can use to run a standard `stat` against. I found a file called ".xsession-errors". We'll use that. I then run the standard `stat`.

Standard `stat` command output of a file showing nanosecond granularity

Running a standard `stat` (stat /home/b/.xsession-errors) gives us the following that we will jot down for later:

Access: 2018-01-14 17:59:51.272474556 -0800
Modify: 2018-01-14 17:59:52.632462056 -0800
Change: 2018-01-14 17:59:52.632462056 -0800

Now, let's run the debugfs `stat` command on the same file and see what we get.

Before running the debugfs `stat` command

Note that, to use the debugfs `stat` command, you need to first specify the file system you want to use with the -w option. Since we already ran an `fdisk -l` before, we already know that we need to specify /dev/sda1. So we run `sudo debugfs -w /dev/sda1`, we will then get a debugfs prompt where we will run our `stat` command: `stat /home/b/.xsession-errors`. Also note that you can run the `stat` command against an inode or the full path of a file (as I did here) within the specified file system.

Output of the debugfs `stat` command, showing encoded timestamps

Let's go ahead and jot down the relevant lines of this output:

ctime: 0x5a5c0b18:96ca6ba0 -- Sun Jan 14 17:59:52 2018
atime: 0x5a5c0b17:40f686f0 -- Sun Jan 14 17:59:51 2018
mtime: 0x5a5c0b18:96ca6ba0 -- Sun Jan 14 17:59:52 2018
crtime: 0x5a5c0b17:40f686f0 -- Sun Jan 14 17:59:51 2018

(An off-topic tidbit here is that we see a "crtime," which is the "born/creation time" of the file being queried; ext file systems prior to ext4 did not support this).

Jonathon already identified the first section of hex bytes (before the colon) to be the Unix epoch representation of each timestamp. Let's confirm using TimeLord.

Using TimeLord to decode the first section of a debugfs `stat` command timestamp

As we can see, Jonathon is correct; the first section of the debugfs `stat` timestamp is the Unix time timestamp (remember, my Ubuntu VM is set to Pacific time (-8), so you must apply the offset to get it to match).

But what about the second portion? That's the real question! Knowing what we know already, we have some really good reasons to believe that the second portion of the debugfs timestamp is the nanoseconds value: we saw references to possible nanoseconds in the source code, we confirmed nanosecond granularity in the standard `stat` command, and the modify and changed times are the same using both `stat` commands.

But how do we confirm this? Well...let's try Googling it.

Googling "ext4 debugfs nanoseconds"

With our testing, we were able to search more effectively for what we needed. And this time, it really paid off -- there's already an answer for us! In fact, the answer goes one step further and even links us to one of Hal Pomeranz's "Understanding EXT4" articles that walks us through the ext4 timestamp format in-depth. I would highly recommend reading the full series, but to answer our initial question, we do not need to look further than the "Fractional Seconds" portion of Hal's post.
"The hex value of the create time "extra" is 0x148AF06C, or 344649836. But the low-order two bits are not used for counting nanoseconds. We need to throw those bits away and shift everything right by two bits- this is equivalent to dividing by 4. So our actual nanosecond value is 344649836 / 4 = 86162459."
Hal explains that the second part of the debugfs `stat` timestamps (after the colon) need to be divided by 4 after being converted to decimal. So let's try that with our example. The values for each stat command are replicated below (we'll only use the access and modify times since there are only 2 unique timestamps among the four total timestamps for the .xsession-errors file).

Standard `stat` command
Access: 2018-01-14 17:59:51.272474556 -0800
Modify: 2018-01-14 17:59:52.632462056 -0800

debugfs `stat` command
atime: 0x5a5c0b17:40f686f0 -- Sun Jan 14 17:59:51 2018
mtime: 0x5a5c0b18:96ca6ba0 -- Sun Jan 14 17:59:52 2018

0x40f686f0 (hex) --> 1089898224 (decimal) / 4 = 272474556
0x96ca6ba0 (hex) --> 2529848224 (decimal) / 4 = 632462056

We convert hex to decimal, then divide by 4, and we get the nanoseconds value!

With that, we've answered the question and learned a thing or two. Again, this was more of an exercise in problem solving and verification, but knowing how to go about solving a problem is something that you can apply to any question you may have.