I am not an expert on this but from my understanding the biggest issue, in a nutshell, is that a smaller cache is faster, but a larger cache will take less hits, meaning it is more likely to hold the sought after data. So to improve performance the CPU will go to the smaller faster L1 cache first, if the sought after data isn't there it will then move to the larger relatively slower L2 cache, then to the L3 (if present), then to main system memory, then finally to the HDD if necessary.
So basically there is a balance between smaller and faster, and larger and slower but taking less hits.
There is much more to this though. In fact you could probably write a masters thesis on cache alone.
8 Pro 64bit
AMD FX 8350
Asus Sabertooth 990FX R2.0
24GB GSkill/Corsair 1866MHz
2x XFX 6870 1GB
SSD - OCZ V4 128GB/Kingston 120GB
HDD - 3TB/2x 750GB/500GB/250GB
Cooler Master HAF 932 Red
CM Hyper 212 EVO w/2x SickleFlow 120mm Red
Logitech X540 5.1 Surround
2X Acer 23" LED - Eyefinity: 3840x1080
Microsoft Sidewinder x4 KB
"The significant problems we face cannot be solved at the same level of thinking we were at when we created them."
- Albert Einstein