The add-on is so large because it (currently) ships with a pre-processed BrowsCap database of User-Agents. There are methods/libraries for parsing User-Agent-strings, but non of these provide the required information at the required level of quality for this add-on, atm. If I ever get to switch to any of these completely for “User-Agent Switcher” the size would likely drop to 100–300kB or so. So, no short-terms fixes for unfortunaltely, sorry.
You may like “Smart Referer” (also mostly be me): https://addons.mozilla.org/de/firefox/addon/smart-referer/
The UI isn't very advanced, but from a functional perspective they're almost at feature-pairity. Please open an issue or send me a mail if you have any (specific!) ideas on how to improve it.
Edit: Regarding the per-domain overrides: I know this is missing currently, but that's simply because nobody has asked for it yet :-D
Please open an issue in our issue tracker, including an outline of how you would want this to look like, and we'll see if this something that will be implemented.