GC is a huge and hugely important subject, so I recommend you look into it in more depth; I have this book and like it reasonably well.
There’s no need to base outside-the-VM GC/mem mgmt on in-VM GC/mm, and in fact it’s nigh impossible to do automatic tracing GC correctly in a language like C because representation and behavior aren’t tied down the way most people seem to imagine. (You can do it if you treat C as an IR-DSL, but that makes integration with/use of normal programming patterns difficult.) Integrating anything involving the call stack directly (e.g., vars of auto
or—heaven forfend—register
storage class) into the VM’s memory management scheme should be avoided, letting the build-compiler* do its thing; rare exceptions should do e.g.
struct GCRoot root; GCRoot_init(&root, obj_handle); … GCRoot_deinit(&root);
for specific target objects, or maybe
VM_fence(); GC_suspend(&theVM); use things VM_fence(); GC_resume(&theVM);
Even assuming you didn’t load up your allocator with UB, a normal build-compiler’s free to reorder anything nonvolatile as long as things appear to work correctly after-the-fact per its model of reality, so you need very careful fencing and sequencing (for all VM/GC-visible behaviors and data, not just pointers) in order to maintain heap invariants and avoid glitching out if the (build-)compiler or optimization options are changed. You can use refcounting, but multithreading around naïve refcounting requires an atomic dec-tst-(n)z instruction per shared upref/downref, which can be extremely inefficient (compared to non-atomic accesses) and may thoroughly wreck memory locality. Refcounting also gets hairier if you admit arbitrary circular reference chains, or if you have to bridge refcounted objects across spacetime (e.g., leases by networked nodes, interactions with untrustworthy code, or saving to a file).
The host-compiler (i.e., what compiles your language) is effectively in its own universe, so there’s ~no need to use the in-VM allocator ever, unless your compiler is actually running inside the VM.
An interpreter’s job is to anchor the VM in concrete reality, and where memory allocation is concerned that means you have to operate both within and without the VM. The interpreter needs to manage root pointers/objects, and to format and locate in-VM pointers properly so GC can deal. Often the VMaddress format underlying VMpointers will include some high or low bits not used by the interpreter’s own ptr/address format—e.g., a mutable bit mirroring C/++ const
-qualification, a lock bit so an object won’t be culled or relocated by GC, or a taint bit so data provenance can be tracked. Your build-language can’t use those things directly, just as in-VM code can’t(shouldn’t) touch raw interpreter innards directly. Any direct integration of VM concepts into the interpreter makes it much harder to change or improve the implementation without major rewrites.
How did you get a copy of the second edition? The Amazon listing says that it's the first edition.