Vulnerability Archeology: Stealing Passwords with IBM i Access Client Solutions

As part of our ongoing research of the IBM i platform we monitor news and updates related to the platform. Two weeks ago IBM published a support article about a compatibility issue affecting IBM i Access Client Solutions (ACS) when running on Windows 11 24H2. The “no man’s land” between system boundaries is always a playground for hackers, and this article was fascinating because it pointed to the Local Security Authority subsystem of Windows:

IBM ACS Application Package *WINLOGON support (“Use Windows user name and password, no prompting”) is incompatible with LSA Protection.

LSA is responsible for the secure handling of the credentials that occur on the system, so it’s an obvious target for attacks - the infamous Mimikatz tool for example made its name by cleverly parsing all kinds of secrets from the memory of the LSASS.exe process, housing credentials. LSA Protection is a set of security features developed by Microsoft to protect credentials even from privileged local attackers.

Could it be that IBM is playing dirty games with LSASS just like Mimikatz?! This was the question that started our little journey into ACS.

Recon

Before diving into the sea of bytes, it’s usually worth looking around for existing research and relevant documentation that can help our work.

First of all, the support article also states this:

The Windows 11 24h2 update enables Local Security Authority (LSA) on install: https://support.microsoft.com/en-us/windows/inside-this-update-93c5c27c-f96e-43c2-a08e-5812d92f220d

The only relevant point in the linked MS article seems to be this:

Local Security Authority (LSA) protection enablement on upgrade: automatic reinforcement of security during system upgrades

It’s unclear and quite strange how some restrictions “during system upgrades” may affect the usability of a 3rd party software overall. We couldn’t find further information about this additional protection for Windows, and based on our further results it seems unlikely that this particular change affects ACS.

When searching for the *WINLOGON authentication mode we found a much more useful article from IBM:

This update disables the “Client Access Network” network provider, which was used to enable *WINLOGON authentication support. To re-enable, follow these steps:

1. Open regedit.exe
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order
3. Edit the ProviderOrder value and add Cwbnetnt to the beginning of the comma separated list of providers
4. Restart your system If you encounter problems, contact IBM Support for assistance.

Now that Registry path looks quite promising! Another search brings us to the research of Grzegorz Tworek and the Network Provider DLL attack technique (T1556.008)!

In short, Network Provider DLLs can be registered by privileged users on a system via the previous Registry key. On user login registered DLLs get automatically loaded by the freshly spawned mpnotify.exe process that passes them cleartext credentials via the NPLogonNotify() callback exported by the DLL.

So the good news is that ACS probably doesn’t play Mimikatz with our most sensitive Windows processes. The bad news is that ACS probably harvests plaintext Windows credentials, and secure handling of such data is hard.

Our high-level theory is now this:

• ACS gets plaintext Windows passwords when a user logs in via mpnotify.exe. Plaintext credentials are required because of the different cryptographic algorithms used by Windows and IBM i.
• ACS wants to use those credentials to log in to remote IBM i systems. Note that at this point, mpnotify.exe and the Network Provider loaded in it are no longer running.
• Consequently, ACS must persist the credentials so they are available for remote system connections at later times.

To verify this theory we should check the implementation!

An easy approach is to fire up ProcMon, filter for mpnotify.exe, and look for any persistent data storage operations that may save the password. Surely enough, the recorded Registry write to this value looks a bit suspicious:

HKLM\SOFTWARE\Wow6432Node\IBM\Client Access\CurrentVersion\Volatile\Communication\Time Stamps\.windows\<USER>\Function Admin Timestamp

Now if you are smarter than me, you can search the web for “Function Admin Timestamp”, and find this article from 2016 by Tenable describing that IBM System i Navigator stores Windows passwords with weak obfuscation in a Registry value readable by all local users. If you aren’t that smart, you’ll end up reversing the whole thing yourself, and maybe even write a deobfuscator :)

At this point, we could just write down our work as DUPLICATE and grab a drink, but the timeline still bugged us: there is a CVE for this problem from 2016, yet it was not earlier than 2024 that IBM decided to deprecate the *WINLOGON feature, and even in early 2025 they had to document the compatibility problem. This should mean that the feature is still alive and may still provide us with some nice leaks!

Test Environment

IBM’s product naming is…special and changes over time so let’s summarize the main actors of our little play:

• System i Navigator is the product investigated by Tenable.
• iSeries Navigator is an older product from the V5 times (IBM i is now at the V7 version). Our test lab has a 2005 version installed. We reversed the original obfuscation with this one, which helped us a lot with the new ACS products.
• Nowadays if you download IBM i Access Client Solutions you get a Java-based application ( IBM i Navigator ), that doesn’t interact with the logon process.
• There is a separate package named “ACS Windows App Pkg” (aka. the “Application Package”) that provides integration tools (think ODBC driver) for Windows developers so their software can interact with IBM i.

Following this guide, you can configure “shared login” (that uses Windows credentials on IBM i) with both Navigator and the Application Package installed.

We couldn’t get the latest version of the Application Package to register a Network Provider DLL - this is probably the result of deprecation and fixes all the problems described. I could confirm the relevance of pre-2024 versions by hunting down a 2019 Application Package (Application Package version 11.26.00, cwbnetnt.dll version 13.64.26.0) that still registered itself.

This version sets strict ACLs on the values under “Function Admin Timestamp”, which is likely the fix for Tenable’s CVE-2016-0287. Since I don’t think this adequately mitigates the problem (more on this at the end of the post), I tried to write a deobfuscator for this version too.

Reversing

The obvious starting point is CWBNETNT.DLL (see the above quote) which implements the Network Provider. Looking at the decompiled NPLogonNotify export we immediately see the dynamic resolution of a related library, which will probably do the heavy lifting. In the 2005 version, this export parsed the lpAuthentInfo structure, the 2019 one just passes the whole thing to the core DLL:

hModule = LoadLibraryW(L"cwbcore.dll");
if (hModule != (HMODULE)0x0) {
  coreFuncPtr = GetProcAddress(hModule,(LPCSTR)0x5b7);
  if (coreFuncPtr != (FARPROC)0x0) {
    (*coreFuncPtr)(lpAuthentInfoType,lpAuthentInfo,lpStationName);
    FreeLibrary(hModule);
  }
}
return 0;

By getting a stack trace from ProcMon at the Registry write event, we can see where the password data is headed.

Let’s call this the “Admin_Timestamp()” function because the name of the Registry value is directly referenced here for the RegSetValue operation. By breaking at this function with a debugger in mpnotify.exe we can see that the function is called with the plaintext password data. But how do we debug mpnotify.exe?

There are two small tricks:

• The target process runs as SYSTEM, so we used PSExec to spawn a debugger with the same privileges (psexec -i -s).
• We statically patched NPLogonNotify() to contain an infinite loop, giving us time for attaching. We took this approach because automatic injection to the process (e.g. via an AppInit DLL) wouldn’t guarantee that we could interact with the debugger since we don’t have access to the SYSTEM desktop (and we didn’t feel like setting up KD for this…). The infinite loop can be disabled once the debugger is attached. ProTip: don’t log out of the test system (as I accidentally did) after the patch, as without a debugger no subsequent logins will ever finish :P

That was easy enough, what happens to our data now? By browsing the call tree of Admin_Timestamp() this exported function (Ordinal #1470 in both versions) catches our eye with its weird XOR’ing:

char * Ordinal_1470(char *key0,char *key1,char *pw0,char *pw1,ulonglong size)
{
  char cVar1;
  char cVar2;
  uint counter;
  ulonglong i;

  if (size != 0) {
    i = 0;
    counter = 0;
    do {
      cVar2 = key1[(uint)i & 7];
      cVar1 = pw0[i];
      pw1[i] = cVar2 + cVar1;
      pw1[i] = key0[counter % 7] ^ cVar2 + cVar1;
      i = (ulonglong)(counter + 1);
      counter = counter + 1;
    } while (i < size);
  }
  return pw1;
}

The function is called twice from Admin_Timestamp() and we can confirm that the first call receives the plaintext password, and the buffer returned by the second call is stored in the Registry.

So this is not cryptographic quality, but maybe the keys are generated in some special, unbreakable way?

Well, not exactly. During the first round of Ordinal_1470 the key0 and key1 arguments are derived from the current timestamp and tick count. Now this is a problem - for legitimate users at least - because when you want to access the stored credential you won’t have the tick count anymore. So in the second round, these first keys are appended to the start of the result of round1, and the operation is keyed with the first 8 characters of the hostname and

• in the 2005 version: a seemingly random 2-byte value “endian-mirrorred” as two DWORDS (e.g: DEAD0000 0000ADDE)
• in the 2019 version: an 8-byte value derived from the Build GUID and Product ID of the machine.

Note that the 2005 version still has randomness in the key - no wonder we couldn’t make the connection work in our lab… The derivation in the 2019 version works by “XOR folding” the concatenated input strings to the target buffer - one can find the generation algorithm by looking at the functions referencing the global variable used by Admin_Timestamp():

while (currentChar != 0) {
  (&obfuscator_round2_key0)[idx] = (&obfuscator_round2_key0)[idx] ^ *wBuildGuidprodId;
  wBuildGuidprodId = wBuildGuidprodId + 2;
  idx = (ulonglong)((int)idx + 1U & 7);
  currentChar = *(short *)wBuildGuidprodId;
}

To deobfuscate, we need the blob from the registry, the hostname, the Build GUID, and the OS Product ID. With this, we can invert the second round to get the time-based keys, and invert the first round to get the plaintext password.

From these inputs, the keys can be considered public data, but as of 2019, the blob is protected with Registry ACLs. So what’s the big deal?

A Red Teamer’s Take

So far we’ve seen that IBM ACS stores Windows passwords essentially as plain text, but newer versions of the product protect this data with Registry ACLs.

Red Teams (and real-world attackers) tend to operate along established trust relationships so they don’t need expensive and potentially noisy exploits (that would allow breaching security boundaries) at each step of their path toward crown jewels. LSA protections mentioned at the beginning of the post aim to restrict access to locally stored credentials even after the attacker obtained administrative privileges. Note that local credential access is one of the most common and useful techniques attackers tend to use, no wonder Microsoft put considerable effort into hardening this area.

Storing plaintext credentials negates all these efforts, and using obfuscation and innocuous-looking Registry keys deceives primarily the security teams, not attackers (reverse engineering cheap tricks like this is our bread and butter!). It’s worth mentioning that plaintext credentials are especially useful, because they can be used on interfaces where pass-the-hash doesn’t work (e.g.: RDP, web apps). In many scenarios (e.g. when using RAT’s) attackers don’t have access to plaintext credentials.

So, if you are on a Blue Team, make sure to run a quick scan for “Function Admin Timestamp” keys and remove them as soon as possible.

There is one problem though: LSASS memory doesn’t get saved in backups, Registry hives do…

Alternative Ways of Credential Storage

As we wrote earlier protecting Windows credentials (taking into account modern, in-the-wild threats) is a hard problem, so we just quickly highlight some alternative approaches and some of their shortcomings:

• We can’t ask the user for an additional password and use symmetric encryption to protect the collected credentials, because the whole point of *WINLOGON was to spare the user the effort of typing credentials.
• Windows DPAPI provides developers with tools to tie stored secrets to the logon credentials of the current user. A Network Provider could in theory store the plaintext password with DPAPI, but these would remain accessible to its rightful owner and (in some cases) Administrators.
• Hashing the password so it remains useful for IBM i authentication (just like in case of NTLM) resolves the plaintext problem, but would likely still provide the attacker with data to efficiently crack or reuse to connect to IBM i at least.

Note that we are dealing with a single sign-on problem here: instead of reinventing the wheel (badly), using Kerberos would eliminate the problems of password handling. Unfortunately even IBM admits Kerberos is “non-trivial to configure” on their side…

What about Windows 11?

If we observe Registry accesses on Windows 11 24H2 we see something strange:

• mpnotify.exe tries to open the Registry key
• it fails because the key doesn’t exist
• then instead of trying to create the path as expected, it just exits

We tried to create the subtree by hand, and found that even if the “Function Admin Timestamp” value exists, mpnotify.exe deletes it and then exits!

We also observed some funny behavior: the Registry operations occur in the SOFTWARE\Wow6432Node tree, and if the same subtree is not present under SOFTWARE\ the process opens the ….windows\userb1 key, but starts to operate on ….windows\New Value #1 o.O This may deserve some further investigation!

The stack displayed by ProcMon is a bit messy, but we could trace back the behavior once again to Admin_Timestamp():

void credhandler_Admin_TimeStamp_1800d9440
               (wchar_t *buf,wchar_t *path,wchar_t *password0,longlong param_4,int param_5)

{
  // ... Variable declarations ...
  
  if ((path != (wchar_t *)0x0) && (password0 != (wchar_t *)0x0)) { // NULL ptr check
    pwVar1 = buf + 4;
    Ordinal_1645((longlong *)pwVar1,path);
    if (*password0 == L'\0') { // Zero-length passowrd - We hit this path! 
      local_3f8 = 7;
      local_400 = 0;
      local_410 = L'\0';
      pw_len = 0xffffffffffffffff;
      do {
        pw_len = pw_len + 1;
      } while (L"Function Admin Timestamp"[pw_len] != L'\0');
      FUN_1800088c0((ulonglong *)&local_410,L"Function Admin Timestamp",pw_len);
      Reg_Open_Delete_180075f40((longlong)buf,&local_410,0x10,4); // Registry access
      if (7 < local_3f8) {
        operator_delete((void *)CONCAT62(uStack_40e,local_410));
      }
    }
    else {
      // ... Store obfuscated password ...
    }

Debugging the entry point in cwbnetnt.dll also confirms that password information is no longer passed to the Network Provider!

This change was documented by Microsoft here in March 2024, we believe IBM should’ve referenced this document in their memo. This is an important change from Microsoft - let’s hope not many applications rely on this backdoor and their insecure artifacts get cleaned up properly! On the offensive side, a quick search for implementers of NPLogonNotify() e.g. on Winbindex may yield some additional interesting research targets :)

As usual, our code is on GitHub: silentsignal/ACS-dump

Header image from Fortepan showing the proper use of a traditional Hungarian “wine stealer”