Mounting UFS VMDK from NetScaler/Citrix ADC
We'll cover how to mount a VMDK, which contains multiple partitions, originating from a NetScaler VM. This is to support analysis in relation to CVE-2023-3519.
We have our VMDK representing a single disk from a compromised system. In this example, we're just using the VMDK available from the installer archive provided by Citrix. We've setup a test VM using the same VMDK, run through the initial setup (setting IP, netmask, gateway, etc) and the system is operational.
This also assumes you have a single flat VMDK, and not a standalone snapshot. If you have a snapshot, you need to consolidate it first so you can examine the resultant disk.
$ file NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk
NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk: VMware4 disk image
$ sudo mkdir /mnt/nsvpx
Execute guestfish and add (-a) our VMDK.
$ guestfish -a NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.
Type: ‘help’ for help on commands
‘man’ to read the manual
‘quit’ to quit the shell
><fs> run
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs>
We'll first attempt to list identified filesystems, but given UFS support is sometimes questionable, we're not sure whether it will be identified or not.
><fs> list-filesystems
libguestfs: error: list_filesystems: sfdisk exited with status 1: sfdisk: /dev/sda: partition 5: partition table contains only 4 partitions
This is what the same VMDK looks like in FTK Imager.
This maps to the following disk size information from the NetScaler console itself
Regarding our VMDK, we'll be interested in the following;
- /dev/md0 (the RAM disk/partition)
- /dev/da0s1a
- /dev/da0s1e
Side note: If this was a live system, you could image each using dd and transfer them to another host via SSH, which would make life a lot easier;
dd if=/dev/md0 | gzip -1 - | ssh user@remotehost dd of=/path/folder/md0.gz
dd if=/dev/da0s1a | gzip -1 - | ssh user@remotehost dd of=/path/folder/da0s1a.gz
dd if=/dev/da0s1e | gzip -1 - | ssh user@remotehost dd of=/path/folder/da0s1e.gz
><fs> list-partitions
/dev/sda1
/dev/sda5
/dev/sda6
/dev/sda7
/dev/sda8
We're mostly interested in logs, which are stored in /var (/dev/da0s1e)
To identify which partitions relate to which filesystem (listed above), we need to mount them.
><fs> mount-vfs ro,ufstype=ufs2 ufs /dev/sda1 /
><fs> df-h
Filesystem Size Used Avail Use% Mounted on
/dev/root 4.0G 714M 3.1G 19% /
/dev 600M 0 600M 0% /dev
shmfs 606M 0 606M 0% /dev/shm
tmpfs 243M 72K 243M 1% /run
/dev/sda1 1.6G 156M 1.3G 11% /sysroot
><fs> mount-vfs ro,ufstype=ufs2 ufs /dev/sda8 /
><fs> df-h
Filesystem Size Used Avail Use% Mounted on
/dev/root 4.0G 714M 3.1G 19% /
/dev 600M 0 600M 0% /dev
shmfs 606M 0 606M 0% /dev/shm
tmpfs 243M 72K 243M 1% /run
/dev/sda8 14G 1.3G 12G 10% /sysroot
Ok, so to recap;
- We've mounted our VMDK in guestfish
- We've mounted each partition as identified they contain relevant data
We can mount the partition (identified through guestfish) on the host machine
><fs> mount-local /mnt/nsvpx
><fs> mount-local-run
Open another console on your host machine, and navigate to /mnt/nsvpx
$ cd /mnt/nsvpx
$ ls /mnt/nsvpx
AAA clusterd core cron download gui krb log netscaler nsinstall nsproflog ns_sys_backup nstmp opt pubkey run tmp vpn
app_catalog configdb crash dev gcf1 install learnt_data mastools ns_gui nslog nssynclog nstemplates nstrace osr_compliance python safenet vmtools vpns
[/mnt/nsvpx/log]
$ cat httpaccess.log| head
127.0.0.1 - - [23/Aug/2023:08:30:00 +0000] [1048] "GET / HTTP/1.1" 200 18705 "-" "curl/7.85.0" "Time: 102541 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1046] "GET / HTTP/1.1" 200 18705 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 46242 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1049] "GET /admin_ui/common/js/jquery/jquery.keyfilter.min.js HTTP/1.1" 200 756 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 2792 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1046] "GET /admin_ui/common/css/ns/ui.css HTTP/1.1" 200 8642 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 8205 microsecs"
You can now use your mount point (/mnt/nsvpx) as the root directory/base to search for IOCs.
ShadowServer have released a series of indicators here;
Technical Summary of Observed Citrix CVE-2023-3519 Incidents
Mandiant have released their analysis, as well as a bash script to search for IOCs. https://github.com/mandiant/citrix-ioc-scanner-cve-2023-3519
You can use Mandiant's IOC scanner on a mounted image, like s
$ wget https://github.com/mandiant/citrix-ioc-scanner-cve-2023-3519/releases/download/v1.2/scanner-cve-2023-3519-v1.2.sh
$ chmod +x scanner-cve-2023-3519-v1.2.sh
Example:
$ bash ./scanner-CVE-2023-3519-v1.1.sh /mnt/path/to/evidence/root/
$ ./scanner-cve-2023-3519-v1.2.sh /mnt/nsvpx