Imma write these down and research them- I think it could be really interesting to have a magic-based server!
Thaumcraft, Blood Magic, and Witchery are really good ones. I'm not sure if they've fixed it or what, but as of last October, Ars Magica 2 would just break your server after a while. Very fun mod, but you just couldn't run it because there were certain things (new dimension, other stuff) that'd just kill the server.
EDIT: I've also heard fantastic things about the Direwolf20 1.7 pack. It's one of those mega-packs that just has everything-tech stuff, tons of magic, a few other dimensions (twilight forest, etc), pretty much everything. Seems like that and Resonant Rise 3 are the two big ones currently.