I am not an expert on this but from my understanding the biggest issue, in a nutshell, is that a smaller cache is faster, but a larger cache will take less hits, meaning it is more likely to hold the sought after data. So to improve performance the CPU will go to the smaller faster L1 cache first, if the sought after data isn't there it will then move to the larger relatively slower L2 cache, then to the L3 (if present), then to main system memory, then finally to the HDD if necessary.
So basically there is a balance between smaller and faster, and larger and slower but taking less hits.
There is much more to this though. In fact you could probably write a masters thesis on cache alone.
Ultimate Edition Linux
AMD FX 8350
Asus Sabertooth 990FX R2.0
24GB GSkill/Corsair 1866MHz
EVGA GTX 970 SSC/ GTX 650 (PhysX)
SSD - OCZ V4 128GB/Kingston 120GB
HDD - 3TB/2x 750GB/500GB
CM HAF 932 Red
CM Hyper 212 EVO w/2x 120mm Red
Logitech X540 5.1
2X Acer 23" LED - Surround: 3840x1080
Microsoft Sidewinder x4 KB
"The significant problems we face cannot be solved at the same level of thinking we were at when we created them."
- Albert Einstein