Identification, acquisition, and examination of iSCSI LUNs and VMFS datastores
I was recently presented with a situation where I needed to access data stored directly on an iSCSI LUN, and needed to assess the capability of forensic tools to identify and acquire data stored in this location.
Accessing a device via iSCSI provides the potential ability to acquire a full physical image of the underlying block storage (single device, RAID array, etc) and potentially recover deleted material. This would otherwise be unavailable if you were to simply connect to an SMB/CIFS share and logically copy files/folders. It is also an alternative to potentially seizing an entire SAN/NAS which may not always be appropriate in the circumstances.
To repicate a basic scenario, I setup the following;
1 x ESXi host (existing in development environment) 1 x Ubuntu 20.04 server with a 10GB block device setup as an iSCSI target. 1 x Windows workstation with FTK Imager 1 x Linux ‘forensic’ workstation
In this example we'll just touch on the use of FTK Imager, however in reality you'll likely be using a more robust acquisition/analysis tool, and possibly a purposed built file recovery tool which supports VMFS6 (depending on the circumstances)
We need an iSCSI target setup with some test data to provide some level of response and real-world behaviour to what we’re trying to achieve.This does not consider security/authentication/access control list restrictions; it is simply setup for testing purposes and does not contain any sensitive data.
Quick refresher on terminology; iSCSI initiator is the client, iSCSI target is the host/server.
$ sudo apt update; sudo upgrade
$ sudo apt install tgt
$ sudo systemctl status tgt
Verify tgt is running. Edit tgt configuration to define block storage.
$ sudo nano /etc/iscsi/iscsi.conf
$ systemctl restart tgt
$ sudo tgtadm --mode target --op show
Target 1: iqn.2020-07.example.com:lun1
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1998-01.com.xxxx:hostname.domain:67650190:64 alias: iscsi_vmk
IP Address: 192.168.1.50
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Removable media: No
Prevent removal: No
Backing store type: null
Backing store path: None
Backing store flags:
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 10737 MB, Block size: 512
Removable media: No
Prevent removal: No
Backing store type: rdwr
Backing store path: /dev/sdb
Backing store flags:
Our Ubuntu server has been setup with a 10GB block device (/dev/sdb) and will provide access to that device via iSCSI, without any restrictions. Now we want to connect our ESXi host to that iSCSI LUN and configure it as a datastore.
In ESXi (v7.0 u3) go to Storage > Adapters > click Software iSCSI
Enter iSCSI LUN target name, and IP address. Save configuration. Storage > Devices > click rescan/refresh.
Storage > Datastores > New datastore
Next > Full Disk > VMFS6 > Finish
Now we have our ESXi host connected to an iSCSI LUN (10GB) which is formatted with VMFS6
To replicate a data acquisition scenario, we need to populate the ESXi datastore with data to see how it's presented to an analyst.
1 x ISO (using Hiren’s Boot CD – uploaded using web interface) 1 x photo (dog-photo.jpg – uploaded using web interface) 1 x VMDK (Ubuntu focal cloud image – copied from existing datastore)
Now that our basic environment has been setup, we can move onto identification and acquisition.
Requirements – Linux operating system with open-iscsi installed
Oftentimes these types of acquisitions involve assistance from systems administrators and storage experts. There can be access control lists, IP restrictions, firewall rules etc which prohibit direct iSCSI LUN access. For example, a Windows server connects to an iSCSI LUN, creates an NTFS file system, then creates folder shares. The original iSCSI LUN may be restricted to only be accessible from the server’s IP address, may require some level of authentication, etc.
This guide provides general insight into identification and acquisition of those LUNs where these restrictions are not involved. Brute forcing or gaining otherwise ‘unauthorised’ access is outside the scope of this article.
iSCSI operates on TCP/3260. You could begin by scanning an IP (or range) with nmap to identify whether it’s accessible from your workstation.
$ sudo nmap -p 3260 192.168.1.0/24 -sV -Pn -v
Discovered open port 3260/tcp on 192.168.1.18
PORT STATE SERVICE VERSION
3260/tcp open iscsi?
MAC Address: 00:00:00:00:00:00 (VMware)
Enumerate LUNs on target
$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.18
Log into iSCSI
$ sudo iscsiadm --mode node --targetname iqn.2020-07.example.com:lun1 --portal 192.168.1.18 --login
Logging in to [iface: default, target: iqn.2020-07.example.com:lun1, portal: 192.168.1.18,3260]
Login to [iface: default, target: iqn.2020-07.example.com:lun1, portal: 192.168.1.18,3260] successful.
$ sudo iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
Target: iqn.2020-07.example.com:lun1 (non-flash)
Current Portal: 192.168.1.18:3260,1
Persistent Portal: 192.168.1.18:3260,1
Iface Name: default
Iface Transport: tcp
Attached SCSI devices:
Host Number: 33 State: running
scsi33 Channel 00 Id 0 Lun: 0
scsi33 Channel 00 Id 0 Lun: 1
Attached scsi disk sdc State: running
New disk is shown (and matches SCSI device list above)
$ sudo fdisk -l
Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AAB4DDB3-B29E-46E1-A605-915D6A16F884
Device Start End Sectors Size Type
/dev/sdc1 2048 20969472 20967425 10G VMware VMFS
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdc 8:32 0 10G 0 disk
└─sdc1 8:33 0 10G 0 part
Proceed with acquisition using preferred tool
Acquire image using ewfacquire (if available) or dd
$ sudo ewfacquire /dev/sdc
$ dd if=/dev/sdc1 of=/mnt/point/output/file.dd
If we were to assess the ability to acquire this datastore (or recover individual files from within), we need access to the iSCSI LUN, a tool to acquire a full physical image, and an analysis/parsing/examination tool which supports VMFS5/6.
Windows 10 does not support mounting and modifying VMFS6 filesystems directly, so we can safely use Windows 10 to connect to the iSCSI target, mount the block device as a local disk, and image it using a forensic tool all without making changes to the device at a filesystem or block level. In this example FTK Imager will be shown, but you could use whatever you like for acquisition as long as it can read local physical block devices.
Start > Run > iscsicpl
Target: IP address of host
Click Quick Connect, LUN will appear
Select Volumes and Devices, click Auto Configure. Volume/mount point should populate.
FTK Imager - acquire an image
Add Evidence Item
Physical Drive > \\.\PHYSICALDRIVEX - VMware Virtual Disk SCSI
(FTK Imager can't interpret VMFS filesystems)
Right click \\.\PHYSICALDRIVEX
Export Disk Image
Image Destinations > Add
Raw (dd) > Next
Case Info (if applicable) > Next
Destination Folder & File Name
No chunks > Finish
The filesystem on the acquisition target block device should be taken into consideration when assessing the ability to acquire a forensic image. Our objective is to analyse the data we've acquired. If it's ext3/ext4, NTFS, BTFRS, there are more possibilities than if it's just VMFS5/6.
The filesystem also determines the possibility of modification of the original data from the client/initiating operating system. If the target iSCSI LUN contains an NTFS file system, it may automatically mount in Windows when the block device is initialised.
We've removed the iSCSI LUN from our previous machine, deleted the datastore and zeroed the disk using scrub (to remove the VMFS6 filesystem and all remnant data).
$ sudo scrub -p fillzero /dev/sdb
Connected to iSCSI LUN, enabled disk, formatted with NTFS and copied 2 x VMDK as test data.
By following the steps above (loading iSCSI, connecting to the target, automatically configuring devices), it did not result in the volume being automatically mounted. This was tested on this DF workstation, as well as a fresh Windows 10 install (v10.0.19044.1288)
Switching back to our Linux DF machine, we can mount the same block device (after connecting with open-iscsi). After mounting and viewing files, we'll delete cloud-img.vmdk
└─$ sudo mount -t ntfs /dev/sdc2 /mnt/sdc2
└─$ cd /mnt/sdc2
└─$ ls -lah
drwxrwxrwx 1 root root 4.0K Apr 7 00:50 .
drwxr-xr-x 4 root root 4.0K Apr 7 00:37 ..
drwxrwxrwx 1 root root 0 Apr 7 00:49 '$RECYCLE.BIN'
-rwxrwxrwx 1 root root 599M Apr 7 00:48 cloud-img.vmdk
-rwxrwxrwx 1 root root 599M Apr 7 00:50 focal-server-cloudimg-amd64.vmdk
drwxrwxrwx 1 root root 0 Apr 7 00:47 'System Volume Information'
└─$ rm -f cloud-img.vmdk
Even though cloud-img.vmdk was deleted, we can re-add the iSCSI/physical drive in FTK Imager and assess our ability to inspect the file and its size. As we see here, although it's marked as deleted, we can see from the hex stream it has a valid header (and also contained valid data - entire stream hasn't been reproduced).
Out of curiosity - what if we use scrub?
└─$ scrub focal-server-cloudimg-amd64.vmdk
scrub: using NNSA NAP-14.1-C patterns
scrub: scrubbing focal-server-cloudimg-amd64.vmdk 627740672 bytes (~598MB)
As expected (as is the functionality of the tool), the file still exists, but its contents have been replaced/scrubbed/overwritten with random data.
Accessing iSCSI LUNs provides the best opportunity to acquire the most valuable data, however, that requires some planning and forethought about how it is going to be accessed, the platform from which it is accessed, the tools which are going to be used to examine/acquire and subsequently analyse the data.
If you're looking at a SAN/NAS with iSCSI LUNs, accessing those LUNs from a (possibly live) Linux workstation may provide analysts with a reliable and easy to use platform to perform an acquisition. This may require credentials, modification of ACLs/rules on the iSCSI host, etc. These workarounds are necessary as the alternative may simply be not acquiring the data at all. Given these scenarios are mostly applicable to large enterprise environments (or sophisticated home users), then being provided with assistance will be greatly beneficial.