Why can't I set MAXWS to use all the memory in my system?

[Morten|Dyalog]

Under 32-bit Windows, each and every process is given an address space of 2GB. This is not the same as the amount of memory it will occupy, but you can think of it as the maximum amount of memory it could use. The distinction between address space and memory is an important one. Address space is mapped to memory (physical memory plus page space) by Windows. Whilst address space is contiguous (0 to 2GB), the memory it is mapped to is not.

The workspace associated with any Dyalog APL application has to be allocated a contiguous address space. The size of workspace that you can obtain is therefore ultimately restricted to the size of the largest area of contiguous address space in the process.

Windows processes typically consist of a main program plus any number of DLLs (and their data) that get loaded dynamically on demand. DLLs do not necessarily get loaded neatly into a process at progressively incrementing addresses, instead they get loaded in a way that tends to fragment the address space into a series of chunks. Fragmentation of the address space will limit the maximum workspace available.

Note that a Dyalog APL mapped file also requires contiguous address space, and the size of file you can map is limited to the maximum contiguous area of address space that is currently unallocated within the process.

A dyalog.exe or dyalogrt.exe process is entirely controlled by Dyalog APL. When the process starts, it reserves an amount of contiguous address space for the workspace which is defined by the MAXWS parameter. If there is insufficient contiguous address space available in the process. APL will issue an appropriate error message and terminate. In practice, fragmentation of the address space by Windows DLLs reduces the maximum workspace available to Dyalog APL to around 1.6GB.

To get a larger contiguous address space for a larger MAXWS, you can use either
ListDLLs (see http://technet.microsoft.com/en-us/sysinternals/bb896656.aspx) or
Rebase (which is part of the Windows SDK) - see http://msdn.microsoft.com/en-us/windowsserver/bb980924.aspx.

Some hints on rebasing dlls:

Make a safe backup copy of the dll first.

You cannot rebase a dll while it is in use--for example, close dyalog.exe if you wish to rebase dyares32.dll.

Various sources seem to indicate that the "best" range to rebase DLLs is 0x60000000 - 0x68000000.

If the badly placed dll is a system or device dll, you may need to do this:

Start Windows in Safe Mode with Command Prompt Enter commands:

Code: Select all

cd \tmp
copy c:\windows\system32\xpsp2res.dll c:\tmp
rebase -b 0x67000000 xpsp2res.dll
move c:\windows\system32\xpsp2res.dll c:\
copy xpsp2res.dll c:\windows\system32


Restart Windows.