I am not an expert on this but from my understanding the biggest issue, in a nutshell, is that a smaller cache is faster, but a larger cache will take less hits, meaning it is more likely to hold the sought after data. So to improve performance the CPU will go to the smaller faster L1 cache first, if the sought after data isn't there it will then move to the larger relatively slower L2 cache, then to the L3 (if present), then to main system memory, then finally to the HDD if necessary.
So basically there is a balance between smaller and faster, and larger and slower but taking less hits.
There is much more to this though. In fact you could probably write a masters thesis on cache alone.
8 Pro 64bit
AMD FX 8350
Asus Sabertooth 990FX R2.0
16GB G.Skill Sniper 1866MHz
OCZ Vertex 4 128GB SSD
HDD's 750GB x2/500GB/250GB
2x XFX 6870 1GB
PC P&C 750W PSU
Cooler Master HAF 932 Red LED
CM Hyper 212 EVO w/2x SickleFlow 120mm Red LED
Logitech X540 5.1 Surround
2X Acer 23" LED - Eyefinity: 3840x1080
"The significant problems we face cannot be solved at the same level of thinking we were at when we created them."
- Albert Einstein