Dynamic Memory Allocation on Windows in NASM

I'm currently learning how to write programs in Assembly, specifically NASM, with the intention of writing a "high-level to Assembly" code compiler. One of the earliest challenges I encountered was allocating more memory during runtime which can be used for data storage.

In most languages, data is placed either on the heap or the stack. The stack follows a strict structure of "last in, first out", whereas the heap is very flexible regarding the order in which data is allocated and freed. Stack allocation is trivial in Assembly and there are an abundance of tutorials about it, so I'll be focusing on heap allocation.

I've chosen to use the Windows API for convenience, though I believe you could link against the C standard libraries to gain access to platform-agnostic POSIX functions like malloc.

Getting the Heap Handle

Windows automatically gives every process its own heap, which we can access using the Windows API's GetProcessHeap function. This function takes no arguments, so in NASM its name is _GetProcessHeap@0. Upon being called, it places a "handle" representing the process' heap in the EAX register. There's not much we can do with the handle alone, but all other heap functions require this handle as a parameter.

Here is a code example of obtaining the heap handle:

global _main
    extern _ExitProcess@4
    extern _GetProcessHeap@0

    section .text

_main:
    ; Place heap in eax
    call _GetProcessHeap@0

    ; Successfully exit
    push 0
    call _ExitProcess@4

When linking this, you must link kernel32.lib, which on my machine was located at:

C:\Program Files (x86)\Windows Kits\8.1\Lib\winv6.3\um\x86\kernel32.lib

I used the following commands in Command Prompt to assemble my code:

nasm -fwin32 program.asm
link /subsystem:console /entry:main program.obj path/to/kernel32.lib

Allocating Memory

Allocating memory is done by a different function, the HeapAlloc function. This takes 3 DWORD (4-byte) arguments, so in NASM is called _HeapAlloc@12. It places the starting location of the newly allocated memory in EAX, or places null (0) there instead if memory couldn't be allocated. The arguments are as follows (in left-to-right order):

hHeap: The handle of the heap to allocate on, obtained with GetProcessHeap.
dwFlags: Flags to modify how this function behaves. For more information, see the table on the HeapAlloc MSDN page.
dwBytes: The number of bytes to allocate.

We need to push these arguments to the stack in right-to-left order, in other words beginning with the number of bytes. So, to allocate 6 bytes on the heap whose handle is EAX and make all bytes zero:

push 6   ; dwBytes
push 8   ; dwFlags
push eax ; hHeap
call _HeapAlloc@12

Putting it All Together

This example program uses the two functions detailed above to dynamically allocate 6 bytes, then writes the null-terminated string "Hello" to this new memory. It displays this string in a message box using the MessageBoxA function.

global _main
    extern _ExitProcess@4
    extern _MessageBoxA@16
    extern _GetProcessHeap@0
    extern _HeapAlloc@12

    section .text

_main:
    ; Set EAX to heap handle
    call _GetProcessHeap@0

    ; Allocate 6 bytes
    push 6
    push 8
    push eax
    call _HeapAlloc@12

    ; Write string
    mov byte [eax], "H"
    mov byte [eax+1], "e"
    mov byte [eax+2], "l"
    mov byte [eax+3], "l"
    mov byte [eax+4], "o"
    mov byte [eax+5], 0

    ; Display message
    push 0
    push eax
    push eax
    push 0
    call _MessageBoxA@16

    push 0
    call _ExitProcess@4

Note: This is a republished, slightly modified version of a post on my old blog.