C calling convention

This calling convention and ABI, as implemented in LLVM-MOS, is designed to maximize performance and flexibility on the 65xx architecture, which is characterized by a small set of hardware registers and a limited hardware stack. To overcome these limitations, the convention makes extensive use of imaginary registers mapped to zero page memory, and employs a hybrid approach to register saving and argument passing, inspired by conventions such as RISC-V and informed by research on efficient calling conventions.

This calling convention is inspired by RISC-V and informed by research on efficient register usage and calling conventions. The convention uses a "smarter hybrid" approach, leveraging both caller-saved and callee-saved registers, and is optimized by LLVM’s shrink wrapping and caller save-restore placement optimizations.

Register Kinds and Their Purposes

Registers in the MOS ABI are divided into two main categories: real registers and imaginary registers.

Real registers include the accumulator (A), index registers (X, Y), processor status flags (C, N, V, Z), program counter (PC), stack pointer (S), direct page register (D), and interrupt flag (I).

Imaginary registers are zero page memory locations, divided into 8-bit rc* (register of character size) registers (rc0 - r31) and 16-bit rs* (register of short size) registers (rs0 - rs15) where each rs register is a little-endian pair of rc registers.

Caller-Saved and Callee-Saved Registers

The following registers are caller-saved: A, X, Y, C, N, V, Z, and RS1 through RS9 (which correspond to RC2 through RC19). A function may freely overwrite any of these registers; it is the caller’s responsibility to preserve their values if needed across a call.

The following registers are callee-saved: PC, S, D, I, RS0 (RC0 and RC1), and RS10 through RS15 (RC20 through RC31). A function may use these registers, but must restore their original values before returning. Callers can rely on these registers being preserved across function calls.

Argument Passing

Arguments are assigned from left to right, and the return value is assigned in the same manner as the first argument.

Numeric arguments are passed one byte at a time, first in A, then X, then in RC2 through RC15. For example, a 32-bit integer would be split into four bytes, each assigned to the next available register in this sequence.

Pointers are passed using RS1 through RS7, which are 16-bit registers (each composed of two RC registers). This allows for efficient passing of up to seven pointers in registers.

If there are more arguments than available registers, the remaining arguments are passed on the soft stack (the stack implemented in zero page memory, managed by RS0).

Aggregate Types

Aggregates (structs, arrays, etc.) of 4 bytes or smaller are split into their constituent value types and passed individually, using the same register assignment rules as for basic types. Such types are also returned by value.

Aggregates larger than 4 bytes are passed by pointer. The caller is responsible for allocating the memory for the aggregate and passing a pointer to it as an argument. The callee may freely write to this memory, and the caller must assume its contents are overwritten by the call.

Large aggregates are returned by a pointer passed as an implicit first argument, and the function then returns void.

Variable Arguments

Variable arguments (those within the ellipsis of a variadic function) are always passed on the soft stack. Named arguments before the ellipsis are passed as usual (registers first, then stack). This distinction means that variadic functions must be prototyped to ensure correct calling convention, as the handling of variable arguments differs from regular arguments.

Return Values

Return values are assigned in the same manner as the first argument: in A, then X, then RC2–RC15, as needed for the size of the return type. Large aggregates (greater than 4 bytes) are returned via a pointer passed as an implicit first argument, and the function returns void.

Stack and Frame Pointer Usage

The primary stack for local variables and most ABI-managed data is the soft stack, implemented in zero page and managed by the RS registers. The soft stack pointer is RS0, and the soft frame pointer is RS15. These are used for stack allocation, frame management, and passing overflow arguments.

The hardware stack pointer (S) is used for return addresses and some leaf calls, but is not the primary stack for local variables. Its limited size (256 bytes, page 1) makes it unsuitable for general-purpose stack management. Still, the compiler may use up to 4 bytes of hardware stack for saving/restoring temporary values.

The soft frame pointer (RS15) is used to reference function frames and local variables, enabling source-level debugging, stack unwinding, and certain kinds of C99 variable-sized arrays. Use of the frame pointer is affected by `-f[no-]emit-frame-pointer`, and the default is to not emit a frame pointer.

DWARF Register Numbering

Each RC register is assigned a DWARF register number: 0x10 + (N * 2) for rcN.

Each RS register is assigned a DWARF register number: 0x210 + N` for rsN.

The hardware registers (A, X, Y, P, SP, PC) are mapped to the first six registers in the GDB remote stub and have their own DWARF numbers. [need to verify this]

Examples

The table below illustrates how function arguments are distributed over the (imaginary) registers.

Function	A	X	rc2	rc3	rc4	rc5	rc6	rc7	Output	Comment
char f(int a)	a	a							A	Numeric args passed...
long f(long a, int b)	a	a	a	a	b	b			A, X, rc2-3	...by value. Even...
void f(int64_t a)	a	a	a	a	a	a	a	a		...very long ones.
int * f(void *a)			a	a					rc2-3	Pointers only through...
int f(int a, int b, void *c)	a	a	b	b	c	c			A, X	...imaginary register pairs.
int f(void *a, char b, int c)	b	c	a	a	c				A, X	Assign from left to right.
void f(struct div_t a)	a	a	a	a						By members if sizeof ≤ 4.
void f(struct ldiv_t a)			a	a						By pointer if sizeof > 4.
div_t f(void *a)			a	a					A, X, rc2-3	Return members.
ldiv_t f(void *a)			👻	👻	a	a				Return via write through implicit 1st arg pointer.

References

Davidson, J. W., & Whalley, D. B. (1991). Methods for Saving and Restoring Register Values across Function Calls. Software—Practice and Experience, 21(2), 149-165.

RISC-V Calling Convention (for design inspiration)