Down vest is one of my favorite insulating pieces but not really what I‘d consider a mid layer. The vest is great for low activity, colder than balls conditions like shooting on a square range in the shade, in temps down around 15 or sitting in a shady belay while your buddy does the ice climbing above. As soon as you start moving around, it’s too hot, it doesn’t really breathe (especially if it’s covered in a “water resistant” coating) so if you start sweating, it’s the wrong choice and if it’s wet out, you are not psyched to be trying to keep down dry.
A mid layer, in my way of layering, is under the down vest so when it’s your turn to climb, off comes the vest and you have a breatheable, moisture wicking layer on. It stays warm if it gets wet, it wicks the perspiration away from your skin, it can be worn like a regular old sweatshirt if in the fall and under a heavy outer layer in the winter. When you sweat in a good mid layer fleece, there should be a dew of perspiration on the OUTSIDE of the fabric and your base layer underneath should be basically dry. Fleece has come a long way since the 80s. It smells better, for one thing. The “waffle-weave” type texture stuff is incredibly warm and dries out really fast.
Synthetic fleece does a little better at moisture control and drying out than wool, in my opinion but wool has astonishingly good temperature regulation and better wind resistance (density) but that also makes it heavier. Wool will smell better, longer too.
So, in a nutshell: wool for drier environs, fleece for wetter locales or in really high physical output situations where heavy perspiration will leave you with wet clothes that have to dry out. Down for really cold situations or if you expect dry conditions, low exertion, or want to prioritize packability and weight reductions.
I have skied and climbed (rock and ice and mountains) and hiked and trail-run and backpacked and sea kayaked, watched football and walked the dogs in every conceivable fleece and down and wool combination of base layer, insulating layer, mid layer, shell, etc. No one is really making gear that has no place in the market. Get whatever feels good to wear and meets your needs. Hell, a Pendleton Woolen Mills button up is pretty damn hard to beat for almost everything….and you’ll look looked a right proper hipster wearing it too.