13Cubed Linux memory forensics
13Cubed have provided a memory sample from an Ubuntu host for participants to practice their Linux memory analysis skills.
Richard at 13Cubed recently released a special Linux memory forensics challenge on YouTube where entrants could compete to win a coveted challenge coin. It was suggested that those who participate provide a blog-based walk through of the analysis, so here's my contribution.
Scenario: Walter O'Reilley was hired by a company called 12 Square Industries and his job was to develop a new website. On his first day, he was assigned an Ubuntu 22.04 developers workstation. After completing the new employee orientation, he started work around 1920hrs (UTC).
Walter began searching for articles on how to develop a website using AI. During his research, he decided to download some YouTube videos locally to his system. Through a combination of malicious ads and social engineering, he was tricked into downloading a binary and running it with super user privileges.
We need to determine what happened on this computer system. Richard has also provided the following hints;
HINT 1: To get started, run the Volatility 3 banners plugin to determine the correct kernel version, and subsequently install the correct symbols and create the ISF. HINT 2: The kernel version in use on this Ubuntu 22.04 machine was 6.5.0-41. It is recommended that Ubuntu 22.04 be used for the analysis.
Download memory sample: https://13cubed.s3.amazonaws.com/downloads/linux_challenge.zip
If you're stuck (or you don't want to generate the profile yourself, I've provided it here)
Files
We're provided with the following files;
| File name | Description |
|---|---|
memory.vmem | Memory snapshot of the running virtual machine |
memory.vmsn | Metadata related to the running state of the virtual machine |
The most important thing to identify at this stage is that as we're dealing with a Linux host, the symbol tables aren't easily accessible like Windows kernel profiles are. We can't just run Volatility against it and pull the kernel profile, we'll have to build it.
I've previously written a guide on how to build a custom profile for a Linux host. To summarise;
Download Ubuntu 22.04 ISO, install it, and boot.
Configure additional repository for debug symbols.
Identify the required kernel profile (Richard mentions 6.5.0-41 in the video, but we'll verify that)
Install debug symbols
Boot virtual machine with new kernel
Run dwarf2json and build kernel symbol table
Transfer symbol table to analysis machine and run Volatility3
This is how you identify possible kernels (and to validate Richard's comments above regarding version)
After we've built the kernel profile (using the guide and steps above), move the resultant .json file to your volatility3 folder; /path/to/volatility3/volatility3/symbols/
You can execute a few linux plugins to identify whether the profile is functioning correctly.
| Plugin | Output |
|---|---|
linux.bash | Bash/console history |
linux.envars | Environment variables used by a shell or its child processes |
Challenge Questions
Let's start looking at some of the questions
Question 1: What is the hostname of this device?
This question can be addressed in a couple of ways. First would be to inspect the memory.vmsn file which was provided. This provides information/metadata about the virtual machine itself.
We can see the the Ubuntu ISO was mounted as sata0, displayName is 'LiWS01' and the hostlog reflects the same LiWS01 value. It's worth noting that just because this is the VM's name in the Hypervisor (looks like an ESXi host), it doesn't definitively mean that the guest OS has the same name. The ESXi guest name could be LiWS01, and the guest virtual machine's name may be something entirely different.
You could validate this against another source, such as the memory of the VM itself. If you run a simple strings search against the vmem file, you'll see a stack of variables which reflect LiWS01 and indicate that it's the hostname. This is teetering on the edge of confirmation bias.
Let's run linux.envars and see what it produces. These plugins can sometimes take a long time to run, so it's worth redirecting output into a text file and reviewing it while you run another plugin.
Answer: LiWS01
Question 2: What is the username of the primary user on this device?
The above environment variables provide some additional insight to this answer. Richard said in the briefing that the user's name was Walter O'Reilly, and there's a $HOME variable set as /home/woreilly.
Answer: woreilly
Question 3: What is the IP address assigned to this device?
Guest variables for ESXi guest virtual machines are passed to the hypervisor for networking purposes, including the installation of guest tools (for remote management, shared folder, etc).
Search the vmsn file for 'ip' or 'address'
To support this, if you run bulk_extractor against the memory dump (which we did in question 9) you'll have a resultant PCAP file which you can open using wireshark. Search for that MAC address (00:0C:29:D9:F3:91) and you'll find the corresponding IP address, and you'll be able to match it against traffic to the malicious IP we later identified.
Answer: 192.168.8.11
Question 4: What is the name of the malicious file downloaded by the victim?
Reviewing the bash history above, we see the user downloaded youtube_downloader.elf, made it executable (chmod 0755) and then executed it using sudo. However, at this stage we're not sure where it came from. We just know the filename.
Answer: youtube_downloader.elf
Question 5: What is the PID associated with the execution of the file downloaded by the victim?
Using the plugin 'linux.pslist', we can see recently executed processes.
I've snipped everything above the first line (gnome-calendar) because it's important to understand parent and child processes. We can see in this instance that the PID (process ID) 4910, is the child process of 4909 (sudo). We know from the bash history that the user executed youtube_downloader using sudo, and the grandparent is 4348, so it was executed by the user using the console.
Answer: 4910
Question 6: What is the PID spawned by that process?
Using the same text file (because we piped the output into a text file.. remember..) we want to perform the reverse of what we just did. We want to find the child process of the PID 4910. Instead of looking for PID=4910, we want PPID=4910.
Answer: 6279
Question 7: What is the full name and path of the malicious process used for persistence?
So, youtube_downloader > bash > cron > sh > nettstat
What's cron? Cron is a command line utility which is used to schedule the execution of jobs on a Linux/Unix host. Given this question relates to persistence and we've observed something relating to cron, this would be something to focus on.
We can use the lsof plugin
We can see the PID for nettstat is 6422 - so let's review the output of lsof to see where it was executed.
This doesn't necessarily mean the file remained in that location for persistence. It was executed in /tmp, but it could have been moved elsewhere to evade detection.
Open the memory dump using a hex editor (or even notepad++) and start reviewing it looking for the string 'nettstat' in the context of 'cron' which we observed above.
Answer: /usr/sbin/nettstat
Question 8: What is the full path and filename of the file created via a popular text editor?
Back to the bash history, we can see commands relating to 'subl' (sublime text editor).
We can see the user changed directory into /var/www, made a directory called html, moved into that directory, then created a file (using sublime) called index.html
Answer: /var/www/html/index.html
Question 9: What is the IP address from which one of the malicious binaries was downloaded?
(From Richard) HINT: This is a difficult question. Don't make assumptions. If you are unable to find a plugin that can provide you with this information, consider more "foundational" approaches to enumerating data within the memory dump.
I used bulk_extractor against the memory dump and retrieved a tonne of URLs.
We know at least two suspicious/malicious binaries (youtube_downloader, and nettstat) so have a look in the resultant text files for hits against both filenames.
Question 10: A user was manually added to this device by the Threat Actor. The UID is 1001. What is the username?
(From Richard) HINT: This is a difficult question. Don't make assumptions. If you are unable to find a plugin that can provide you with this information, consider more "foundational" approaches to enumerating data within the memory dump.
This question requires a bit of familiarity with how users are stored on an Ubuntu system.
The /etc/passwd file contains information about users who may log into a system, or other system/service accounts. It's also plain text, which is great. Consider this output from an Ubuntu host;
So we know the format, and we also know that the third value is the UserID (UID). root is 0, www-data is 33, so on. The question indicates that the UID is 1001, so we can do a basic text/strings search for :1001: and see what pops.
1000 is the UID for woreilly, the first user, and 1001 is the UID for 'systemservice' (which is an interactive account because it has a bash/shell set (/bin/bash).
Answer: systemservice
Bonus - hashcat
If you've made it this far, well done. On your Ubuntu 22.04 virtual machine (the one you deployed to build the kernel profile), have a look at /etc/shadow (example below)
Here is a very good explanation of how the shadow file works, and the information it contains.
Why is this relevant? Well, go back to your memory dump (either in a text editor or hex editor) and search for ":::" or "$6$" and you'll find the following;
Download hashcat and your favourite wordlists
Try the same with the value for woreilly (you'll need to use John the Ripper though).
Last updated
