After years of driving down fees, retail fund managers are nearing a wall as what they charge approaches zero. The chance to sell more funds that can charge higher fees would boost revenues for the likes of Vanguard and State Street, which pioneered exchange-traded funds.