Identification, acquisition, and examination of iSCSI LUNs and VMFS datastores

I was recently presented with a situation where I needed to access data stored directly on an iSCSI LUN, and needed to assess the capability of forensic tools to identify and acquire data stored in this location.

Accessing a device via iSCSI provides the potential ability to acquire a full physical image of the underlying block storage (single device, RAID array, etc) and potentially recover deleted material. This would otherwise be unavailable if you were to simply connect to an SMB/CIFS share and logically copy files/folders. It is also an alternative to potentially seizing an entire SAN/NAS which may not always be appropriate in the circumstances.

To repicate a basic scenario, I setup the following;

1 x ESXi host (existing in development environment) 1 x Ubuntu 20.04 server with a 10GB block device setup as an iSCSI target. 1 x Windows workstation with FTK Imager 1 x Linux ‘forensic’ workstation

In this example we'll just touch on the use of FTK Imager, however in reality you'll likely be using a more robust acquisition/analysis tool, and possibly a purposed built file recovery tool which supports VMFS6 (depending on the circumstances)

Setup a basic iSCSI target on Ubuntu.

We need an iSCSI target setup with some test data to provide some level of response and real-world behaviour to what we’re trying to achieve.This does not consider security/authentication/access control list restrictions; it is simply setup for testing purposes and does not contain any sensitive data.

Quick refresher on terminology; iSCSI initiator is the client, iSCSI target is the host/server.

$ sudo apt update; sudo upgrade
$ sudo apt install tgt
$ sudo systemctl status tgt

Verify tgt is running. Edit tgt configuration to define block storage.

$ sudo nano /etc/iscsi/iscsi.conf
<target iqn.2020-07.example.com:lun1>
     direct-store /dev/sdb
</target>
$ systemctl restart tgt
$ sudo tgtadm --mode target --op show

 Target 1: iqn.2020-07.example.com:lun1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1998-01.com.xxxx:hostname.domain:67650190:64 alias: iscsi_vmk
            Connection: 0
                IP Address: 192.168.1.50
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 10737 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/sdb
            Backing store flags:
    Account information:
    ACL information:
        ALL  

Our Ubuntu server has been setup with a 10GB block device (/dev/sdb) and will provide access to that device via iSCSI, without any restrictions. Now we want to connect our ESXi host to that iSCSI LUN and configure it as a datastore.

In ESXi (v7.0 u3) go to Storage > Adapters > click Software iSCSI

Enter iSCSI LUN target name, and IP address. Save configuration. Storage > Devices > click rescan/refresh.

Storage > Datastores > New datastore

Next > Full Disk > VMFS6 > Finish

Now we have our ESXi host connected to an iSCSI LUN (10GB) which is formatted with VMFS6

To replicate a data acquisition scenario, we need to populate the ESXi datastore with data to see how it's presented to an analyst.

1 x ISO (using Hiren’s Boot CD – uploaded using web interface) 1 x photo (dog-photo.jpg – uploaded using web interface) 1 x VMDK (Ubuntu focal cloud image – copied from existing datastore)

Now that our basic environment has been setup, we can move onto identification and acquisition.

Accessing the iSCSI LUN from a Linux machine

Requirements – Linux operating system with open-iscsi installed

Oftentimes these types of acquisitions involve assistance from systems administrators and storage experts. There can be access control lists, IP restrictions, firewall rules etc which prohibit direct iSCSI LUN access. For example, a Windows server connects to an iSCSI LUN, creates an NTFS file system, then creates folder shares. The original iSCSI LUN may be restricted to only be accessible from the server’s IP address, may require some level of authentication, etc.

This guide provides general insight into identification and acquisition of those LUNs where these restrictions are not involved. Brute forcing or gaining otherwise ‘unauthorised’ access is outside the scope of this article.

iSCSI operates on TCP/3260. You could begin by scanning an IP (or range) with nmap to identify whether it’s accessible from your workstation.

$ sudo nmap -p 3260 192.168.1.0/24 -sV -Pn -v
Discovered open port 3260/tcp on 192.168.1.18
PORT     STATE SERVICE VERSION
3260/tcp open  iscsi?
MAC Address: 00:00:00:00:00:00 (VMware)
Enumerate LUNs on target

$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.18
192.168.1.18:3260,1 iqn.2020-07.example.com:lun1        
Log into iSCSI
$ sudo iscsiadm --mode node --targetname iqn.2020-07.example.com:lun1 --portal 192.168.1.18 --login

Logging in to [iface: default, target: iqn.2020-07.example.com:lun1, portal: 192.168.1.18,3260]
Login to [iface: default, target: iqn.2020-07.example.com:lun1, portal: 192.168.1.18,3260] successful.
$ sudo iscsiadm -m session -P3

iSCSI Transport Class version 2.0-870
version 2.1.8
Target: iqn.2020-07.example.com:lun1 (non-flash)
        Current Portal: 192.168.1.18:3260,1
        Persistent Portal: 192.168.1.18:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                
                <snip>
                 ************************
                Attached SCSI devices:
                ************************
                Host Number: 33 State: running
                scsi33 Channel 00 Id 0 Lun: 0
                scsi33 Channel 00 Id 0 Lun: 1
                        Attached scsi disk sdc          State: running               
                
New disk is shown (and matches SCSI device list above)

$ sudo fdisk -l

Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AAB4DDB3-B29E-46E1-A605-915D6A16F884

Device     Start      End  Sectors Size Type
/dev/sdc1   2048 20969472 20967425  10G VMware VMFS

$ sudo lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sdc      8:32   0   10G  0 disk
└─sdc1   8:33   0   10G  0 part

Proceed with acquisition using preferred tool

Acquire image using ewfacquire (if available) or dd
$ sudo ewfacquire /dev/sdc
OR
$ dd if=/dev/sdc1 of=/mnt/point/output/file.dd

Accessing the iSCSI LUN from a Windows machine

If we were to assess the ability to acquire this datastore (or recover individual files from within), we need access to the iSCSI LUN, a tool to acquire a full physical image, and an analysis/parsing/examination tool which supports VMFS5/6.

Windows 10 does not support mounting and modifying VMFS6 filesystems directly, so we can safely use Windows 10 to connect to the iSCSI target, mount the block device as a local disk, and image it using a forensic tool all without making changes to the device at a filesystem or block level. In this example FTK Imager will be shown, but you could use whatever you like for acquisition as long as it can read local physical block devices.

Start > Run > iscsicpl

Target: IP address of host

Click Quick Connect, LUN will appear

Select Volumes and Devices, click Auto Configure. Volume/mount point should populate.

FTK Imager - acquire an image

File
Add Evidence Item
Physical Drive > \\.\PHYSICALDRIVEX - VMware Virtual Disk SCSI

(FTK Imager can't interpret VMFS filesystems)
Right click \\.\PHYSICALDRIVEX
Export Disk Image
Image Destinations > Add
Raw (dd) > Next
Case Info (if applicable) > Next
Destination Folder & File Name
No chunks > Finish

Alternative scenarios

The filesystem on the acquisition target block device should be taken into consideration when assessing the ability to acquire a forensic image. Our objective is to analyse the data we've acquired. If it's ext3/ext4, NTFS, BTFRS, there are more possibilities than if it's just VMFS5/6.

The filesystem also determines the possibility of modification of the original data from the client/initiating operating system. If the target iSCSI LUN contains an NTFS file system, it may automatically mount in Windows when the block device is initialised.

We've removed the iSCSI LUN from our previous machine, deleted the datastore and zeroed the disk using scrub (to remove the VMFS6 filesystem and all remnant data).

$ sudo scrub -p fillzero /dev/sdb

Connected to iSCSI LUN, enabled disk, formatted with NTFS and copied 2 x VMDK as test data.

By following the steps above (loading iSCSI, connecting to the target, automatically configuring devices), it did not result in the volume being automatically mounted. This was tested on this DF workstation, as well as a fresh Windows 10 install (v10.0.19044.1288)

Switching back to our Linux DF machine, we can mount the same block device (after connecting with open-iscsi). After mounting and viewing files, we'll delete cloud-img.vmdk

┌──(user㉿IBLUE-DF1)-[~]
└─$ sudo mount -t ntfs /dev/sdc2 /mnt/sdc2
└─$ cd /mnt/sdc2
└─$ ls -lah
total 1.2G
drwxrwxrwx 1 root root 4.0K Apr  7 00:50  .
drwxr-xr-x 4 root root 4.0K Apr  7 00:37  ..
drwxrwxrwx 1 root root    0 Apr  7 00:49 '$RECYCLE.BIN'
-rwxrwxrwx 1 root root 599M Apr  7 00:48  cloud-img.vmdk
-rwxrwxrwx 1 root root 599M Apr  7 00:50  focal-server-cloudimg-amd64.vmdk
drwxrwxrwx 1 root root    0 Apr  7 00:47 'System Volume Information'

┌──(user㉿IBLUE-DF1)-[/mnt/sdc1]
└─$ rm -f cloud-img.vmdk

Even though cloud-img.vmdk was deleted, we can re-add the iSCSI/physical drive in FTK Imager and assess our ability to inspect the file and its size. As we see here, although it's marked as deleted, we can see from the hex stream it has a valid header (and also contained valid data - entire stream hasn't been reproduced).

Out of curiosity - what if we use scrub?

┌──(user㉿IBLUE-DF1)-[/mnt/sdc1]
└─$ scrub focal-server-cloudimg-amd64.vmdk
scrub: using NNSA NAP-14.1-C patterns
scrub: scrubbing focal-server-cloudimg-amd64.vmdk 627740672 bytes (~598MB)

As expected (as is the functionality of the tool), the file still exists, but its contents have been replaced/scrubbed/overwritten with random data.

Summary

Accessing iSCSI LUNs provides the best opportunity to acquire the most valuable data, however, that requires some planning and forethought about how it is going to be accessed, the platform from which it is accessed, the tools which are going to be used to examine/acquire and subsequently analyse the data.

If you're looking at a SAN/NAS with iSCSI LUNs, accessing those LUNs from a (possibly live) Linux workstation may provide analysts with a reliable and easy to use platform to perform an acquisition. This may require credentials, modification of ACLs/rules on the iSCSI host, etc. These workarounds are necessary as the alternative may simply be not acquiring the data at all. Given these scenarios are mostly applicable to large enterprise environments (or sophisticated home users), then being provided with assistance will be greatly beneficial.

Last updated