13Cubed Linux memory forensics
13Cubed have provided a memory sample from an Ubuntu host for participants to practice their Linux memory analysis skills.
Last updated
Was this helpful?
13Cubed have provided a memory sample from an Ubuntu host for participants to practice their Linux memory analysis skills.
Last updated
Was this helpful?
Was this helpful?
Richard at 13Cubed recently released a special Linux memory forensics challenge on YouTube where entrants could compete to win a coveted challenge coin. It was suggested that those who participate provide a blog-based walk through of the analysis, so here's my contribution.
Scenario: Walter O'Reilley was hired by a company called 12 Square Industries and his job was to develop a new website. On his first day, he was assigned an Ubuntu 22.04 developers workstation. After completing the new employee orientation, he started work around 1920hrs (UTC).
Walter began searching for articles on how to develop a website using AI. During his research, he decided to download some YouTube videos locally to his system. Through a combination of malicious ads and social engineering, he was tricked into downloading a binary and running it with super user privileges.
We need to determine what happened on this computer system. Richard has also provided the following hints;
HINT 1: To get started, run the Volatility 3 banners plugin to determine the correct kernel version, and subsequently install the correct symbols and create the ISF. HINT 2: The kernel version in use on this Ubuntu 22.04 machine was 6.5.0-41. It is recommended that Ubuntu 22.04 be used for the analysis.
Download memory sample: https://cdn.13cubed.com/downloads/linux_challenge.zip
If you're stuck (or you don't want to generate the profile yourself, I've provided it here)
We're provided with the following files;
memory.vmem
Memory snapshot of the running virtual machine
memory.vmsn
Metadata related to the running state of the virtual machine
The most important thing to identify at this stage is that as we're dealing with a Linux host, the symbol tables aren't easily accessible like Windows kernel profiles are. We can't just run Volatility against it and pull the kernel profile, we'll have to build it.
I've previously written a guide on how to build a custom profile for a Linux host. To summarise;
Download Ubuntu 22.04 ISO, install it, and boot.
Configure additional repository for debug symbols.
Identify the required kernel profile (Richard mentions 6.5.0-41 in the video, but we'll verify that)
Install debug symbols
Boot virtual machine with new kernel
Run dwarf2json and build kernel symbol table
Transfer symbol table to analysis machine and run Volatility3
This is how you identify possible kernels (and to validate Richard's comments above regarding version)
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem isfinfo
Volatility 3 Framework 2.10.0
Progress: 100.00 PDB scanning finished
URI Valid Number of base_types Number of types Number of symbols Number of enums Identifying information
[snip] b'Linux version 6.5.0-41-generic (buildd@lcy02-amd64-120) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 (Ubuntu 6.5.0-41.41~22.04.2-generic 6.5.13)\n\x00'After we've built the kernel profile (using the guide and steps above), move the resultant .json file to your volatility3 folder; /path/to/volatility3/volatility3/symbols/
You can execute a few linux plugins to identify whether the profile is functioning correctly.
linux.bash
Bash/console history
linux.envars
Environment variables used by a shell or its child processes
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem linux.bash
Volatility 3 Framework 2.10.0
Progress: 100.00 Stacking attempts finished
PID Process CommandTime Command
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt dist-upgrade
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt upgrade
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt update
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt install open-vm-tools-desktop
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt upgrade
4453 bash 2024-07-03 19:40:49.000000 UTC exit
4453 bash 2024-07-03 19:40:49.000000 UTC sudo apt update
4453 bash 2024-07-03 19:45:41.000000 UTC cd Downloads/
4453 bash 2024-07-03 19:45:43.000000 UTC ls
4453 bash 2024-07-03 19:45:48.000000 UTC chmod 755 youtube_downloader.elf
4453 bash 2024-07-03 19:45:53.000000 UTC ls -al
4453 bash 2024-07-03 19:45:58.000000 UTC sudo ./youtube_downloader.elf
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt dist-upgrade
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt upgrade
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt update
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt install open-vm-tools-desktop
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt upgrade
4954 bash 2024-07-03 19:47:23.000000 UTC exit
4954 bash 2024-07-03 19:47:23.000000 UTC ��U��SH�H�U{
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt autoremove -y
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt update
4954 bash 2024-07-03 19:47:30.000000 UTC cd /var/www
4954 bash 2024-07-03 19:47:32.000000 UTC cd /var
4954 bash 2024-07-03 19:47:35.000000 UTC mkdir www
4954 bash 2024-07-03 19:47:39.000000 UTC sudo mkdir www
4954 bash 2024-07-03 19:47:59.000000 UTC cd www
4954 bash 2024-07-03 19:48:03.000000 UTC sudo mkdir html
4954 bash 2024-07-03 19:48:07.000000 UTC cd html
4954 bash 2024-07-03 20:02:53.000000 UTC subl index.htmlLet's start looking at some of the questions
This question can be addressed in a couple of ways. First would be to inspect the memory.vmsn file which was provided. This provides information/metadata about the virtual machine itself.
We can see the the Ubuntu ISO was mounted as sata0, displayName is 'LiWS01' and the hostlog reflects the same LiWS01 value. It's worth noting that just because this is the VM's name in the Hypervisor (looks like an ESXi host), it doesn't definitively mean that the guest OS has the same name. The ESXi guest name could be LiWS01, and the guest virtual machine's name may be something entirely different.
You could validate this against another source, such as the memory of the VM itself. If you run a simple strings search against the vmem file, you'll see a stack of variables which reflect LiWS01 and indicate that it's the hostname. This is teetering on the edge of confirmation bias.
Let's run linux.envars and see what it produces. These plugins can sometimes take a long time to run, so it's worth redirecting output into a text file and reviewing it while you run another plugin.
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem linux.envars > linux.envars.txt
Answer: LiWS01
The above environment variables provide some additional insight to this answer. Richard said in the briefing that the user's name was Walter O'Reilly, and there's a $HOME variable set as /home/woreilly.
Answer: woreilly
Guest variables for ESXi guest virtual machines are passed to the hypervisor for networking purposes, including the installation of guest tools (for remote management, shared folder, etc).
Search the vmsn file for 'ip' or 'address'
To support this, if you run bulk_extractor against the memory dump (which we did in question 9) you'll have a resultant PCAP file which you can open using wireshark. Search for that MAC address (00:0C:29:D9:F3:91) and you'll find the corresponding IP address, and you'll be able to match it against traffic to the malicious IP we later identified.
Answer: 192.168.8.11
Reviewing the bash history above, we see the user downloaded youtube_downloader.elf, made it executable (chmod 0755) and then executed it using sudo. However, at this stage we're not sure where it came from. We just know the filename.
Answer: youtube_downloader.elf
Using the plugin 'linux.pslist', we can see recently executed processes.
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem linux.pslist> linux.pslist.txt
I've snipped everything above the first line (gnome-calendar) because it's important to understand parent and child processes. We can see in this instance that the PID (process ID) 4910, is the child process of 4909 (sudo). We know from the bash history that the user executed youtube_downloader using sudo, and the grandparent is 4348, so it was executed by the user using the console.
Answer: 4910
Using the same text file (because we piped the output into a text file.. remember..) we want to perform the reverse of what we just did. We want to find the child process of the PID 4910. Instead of looking for PID=4910, we want PPID=4910.
Answer: 6279
So, youtube_downloader > bash > cron > sh > nettstat
What's cron? Cron is a command line utility which is used to schedule the execution of jobs on a Linux/Unix host. Given this question relates to persistence and we've observed something relating to cron, this would be something to focus on.
We can use the lsof plugin
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem linux.lsof > linux.lsof.txtWe can see the PID for nettstat is 6422 - so let's review the output of lsof to see where it was executed.
This doesn't necessarily mean the file remained in that location for persistence. It was executed in /tmp, but it could have been moved elsewhere to evade detection.
Open the memory dump using a hex editor (or even notepad++) and start reviewing it looking for the string 'nettstat' in the context of 'cron' which we observed above.
Answer: /usr/sbin/nettstat
Back to the bash history, we can see commands relating to 'subl' (sublime text editor).
$ vol.py -f /mnt/d/13cubed/memory/memory.vmem linux.bash
Volatility 3 Framework 2.10.0
Progress: 100.00 Stacking attempts finished
PID Process CommandTime Command
4954 bash 2024-07-03 19:47:23.000000 UTC exit
4954 bash 2024-07-03 19:47:23.000000 UTC ��U��SH�H�U{
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt autoremove -y
4954 bash 2024-07-03 19:47:23.000000 UTC sudo apt update
4954 bash 2024-07-03 19:47:30.000000 UTC cd /var/www
4954 bash 2024-07-03 19:47:32.000000 UTC cd /var
4954 bash 2024-07-03 19:47:35.000000 UTC mkdir www
4954 bash 2024-07-03 19:47:39.000000 UTC sudo mkdir www
4954 bash 2024-07-03 19:47:59.000000 UTC cd www
4954 bash 2024-07-03 19:48:03.000000 UTC sudo mkdir html
4954 bash 2024-07-03 19:48:07.000000 UTC cd html
4954 bash 2024-07-03 20:02:53.000000 UTC subl index.htmlWe can see the user changed directory into /var/www, made a directory called html, moved into that directory, then created a file (using sublime) called index.html
Answer: /var/www/html/index.html
(From Richard) HINT: This is a difficult question. Don't make assumptions. If you are unable to find a plugin that can provide you with this information, consider more "foundational" approaches to enumerating data within the memory dump.
I used bulk_extractor against the memory dump and retrieved a tonne of URLs.
$ bulk_extractor -o be-output memory.vmemWe know at least two suspicious/malicious binaries (youtube_downloader, and nettstat) so have a look in the resultant text files for hits against both filenames.
(From Richard) HINT: This is a difficult question. Don't make assumptions. If you are unable to find a plugin that can provide you with this information, consider more "foundational" approaches to enumerating data within the memory dump.
This question requires a bit of familiarity with how users are stored on an Ubuntu system.
The /etc/passwd file contains information about users who may log into a system, or other system/service accounts. It's also plain text, which is great. Consider this output from an Ubuntu host;
$ sudo cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologinSo we know the format, and we also know that the third value is the UserID (UID). root is 0, www-data is 33, so on. The question indicates that the UID is 1001, so we can do a basic text/strings search for :1001: and see what pops.
1000 is the UID for woreilly, the first user, and 1001 is the UID for 'systemservice' (which is an interactive account because it has a bash/shell set (/bin/bash).
Answer: systemservice
If you've made it this far, well done. On your Ubuntu 22.04 virtual machine (the one you deployed to build the kernel profile), have a look at /etc/shadow (example below)
$ sudo cat /etc/shadow
sshd:*:18375:0:99999:7:::
landscape:*:18375:0:99999:7:::
pollinate:*:18375:0:99999:7:::
username:$6$hE2wnL95dHqesJhM$/J/6otQvJ8lmUrBpOx7fL5H9TGGucGkkfYVOGrYWChQtaG/hvOy23oXxuhv5o2G79Uh/ysEXNXJuZ.YnTqPiN.:19997:0:99999:7:::
snort:*:18701:0:99999:7:::Here is a very good explanation of how the shadow file works, and the information it contains.
Why is this relevant? Well, go back to your memory dump (either in a text editor or hex editor) and search for ":::" or "$6$" and you'll find the following;
Download hashcat and your favourite wordlists
hashcat.exe -m 1800 -a 0 13cubed.hash C:\wordlist1.txt C:\wordlist2.txt
Try the same with the value for woreilly (you'll need to use John the Ripper though).