Mounting UFS VMDK from NetScaler/Citrix ADC

We'll cover how to mount a VMDK, which contains multiple partitions, originating from a NetScaler VM. This is to support analysis in relation to CVE-2023-3519.

We have our VMDK representing a single disk from a compromised system. In this example, we're just using the VMDK available from the installer archive provided by Citrix. We've setup a test VM using the same VMDK, run through the initial setup (setting IP, netmask, gateway, etc) and the system is operational.

This also assumes you have a single flat VMDK, and not a standalone snapshot. If you have a snapshot, you need to consolidate it first so you can examine the resultant disk.

$ file NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk
NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk: VMware4 disk image

$ sudo mkdir /mnt/nsvpx

Execute guestfish and add (-a) our VMDK.

$ guestfish -a NSVPX-ESX-13.0-90.12_nc_64-disk1.vmdk

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> run
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs>

We'll first attempt to list identified filesystems, but given UFS support is sometimes questionable, we're not sure whether it will be identified or not.

><fs> list-filesystems
libguestfs: error: list_filesystems: sfdisk exited with status 1: sfdisk: /dev/sda: partition 5: partition table contains only 4 partitions

This is what the same VMDK looks like in FTK Imager.

This maps to the following disk size information from the NetScaler console itself

Regarding our VMDK, we'll be interested in the following;

• /dev/md0 (the RAM disk/partition)

• /dev/da0s1a

• /dev/da0s1e

Side note: If this was a live system, you could image each using dd and transfer them to another host via SSH, which would make life a lot easier;

dd if=/dev/md0 | gzip -1 - | ssh user@remotehost dd of=/path/folder/md0.gz
dd if=/dev/da0s1a | gzip -1 - | ssh user@remotehost dd of=/path/folder/da0s1a.gz
dd if=/dev/da0s1e | gzip -1 - | ssh user@remotehost dd of=/path/folder/da0s1e.gz

><fs> list-partitions
/dev/sda1
/dev/sda5
/dev/sda6
/dev/sda7
/dev/sda8

We're mostly interested in logs, which are stored in /var (/dev/da0s1e)

To identify which partitions relate to which filesystem (listed above), we need to mount them.

><fs> mount-vfs ro,ufstype=ufs2 ufs /dev/sda1 /
><fs> df-h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  714M  3.1G  19% /
/dev            600M     0  600M   0% /dev
shmfs           606M     0  606M   0% /dev/shm
tmpfs           243M   72K  243M   1% /run
/dev/sda1       1.6G  156M  1.3G  11% /sysroot

><fs> mount-vfs ro,ufstype=ufs2 ufs /dev/sda8 /
><fs> df-h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  714M  3.1G  19% /
/dev            600M     0  600M   0% /dev
shmfs           606M     0  606M   0% /dev/shm
tmpfs           243M   72K  243M   1% /run
/dev/sda8        14G  1.3G   12G  10% /sysroot

Ok, so to recap;

• We've mounted our VMDK in guestfish

• We've mounted each partition as identified they contain relevant data

We can mount the partition (identified through guestfish) on the host machine

><fs> mount-local /mnt/nsvpx
><fs> mount-local-run

Open another console on your host machine, and navigate to /mnt/nsvpx

$ cd /mnt/nsvpx
$ ls /mnt/nsvpx
AAA          clusterd  core   cron  download  gui      krb          log       netscaler  nsinstall  nsproflog  ns_sys_backup  nstmp    opt             pubkey  run      tmp      vpn
app_catalog  configdb  crash  dev   gcf1      install  learnt_data  mastools  ns_gui     nslog      nssynclog  nstemplates    nstrace  osr_compliance  python  safenet  vmtools  vpns

[/mnt/nsvpx/log]
$ cat httpaccess.log| head
127.0.0.1 - - [23/Aug/2023:08:30:00 +0000] [1048] "GET / HTTP/1.1" 200 18705 "-" "curl/7.85.0" "Time: 102541 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1046] "GET / HTTP/1.1" 200 18705 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 46242 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1049] "GET /admin_ui/common/js/jquery/jquery.keyfilter.min.js HTTP/1.1" 200 756 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 2792 microsecs"
192.168.1.10 -> 192.168.1.222 - - [23/Aug/2023:08:31:27 +0000] [1046] "GET /admin_ui/common/css/ns/ui.css HTTP/1.1" 200 8642 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0" "Time: 8205 microsecs"

You can now use your mount point (/mnt/nsvpx) as the root directory/base to search for IOCs.

ShadowServer have released a series of indicators here; Technical Summary of Observed Citrix CVE-2023-3519 Incidents

Mandiant have released their analysis, as well as a bash script to search for IOCs. https://github.com/mandiant/citrix-ioc-scanner-cve-2023-3519

You can use Mandiant's IOC scanner on a mounted image, like s

$ wget https://github.com/mandiant/citrix-ioc-scanner-cve-2023-3519/releases/download/v1.2/scanner-cve-2023-3519-v1.2.sh
$ chmod +x scanner-cve-2023-3519-v1.2.sh

Example:
$ bash ./scanner-CVE-2023-3519-v1.1.sh /mnt/path/to/evidence/root/

$ ./scanner-cve-2023-3519-v1.2.sh /mnt/nsvpx