When we talk about 32-bit or 64-bit what we are actually talking about is the width of the registers in the central processor that are responsible for storing data and addresses for a running program.
In order for a program to be able to get a piece of information from memory it first needs to supply an address for the data. It puts the address in a "register" and then it calls a command in the processor that will go to that address in memory and return with the value that it found there. The width of the register determines how many locations in memory are able to be queried in this manner.
A 32-bit address register gives a maximum of 2^32 (4,294,967,296) addresses and it is this number that gives us the hard limit of 4 gigabytes of memory per 32-bit process. I’m making the "per process" distinction because it is an important one in the world of memory managed task-switching operating systems.
A 64-bit address on the other hand gives a maximum of 2^64 (18,446,744,073,709,551,616) or approximately 16 Exabytes of memory. An exabyte is huge. 1024 gigabytes would give us 1 terabyte (the average hard drive these days), 1024 terabytes would give us 1 petabyte, and then 1024 petabytes gives us just 1 exabyte. We are several orders of magnitude away from even thinking of machines capable of making use of that much memory let alone becoming necessary, hopefully it will be long enough that we won’t be changing to 128-bit computers in the next few years.