The real problem that you're running into (been there, done that) is
that the MyISAM index code is only 32-bit clean. You can try to use
more than 4GB for the key_buffer, but it won't work; I don't recall
if the code implicitly limits the value, or just falls over.
One possible workaround is to use multiple (alternate) key_buffers
if your schema is such that that makes sense. You will want to
spend some time looking at the statistics to understand just
where you're running out of memory - if the "pressure" is
on the index or the tables themselves (or both).
As Dan said, MyISAM tries to get the system to do caching of
the table data. That works reasonably well on Linux and NetBSD
(at least), but joins and sorts will be slow above a certain
size, and there seems to be nothing that can be done about it.