Private-equity firm Bain Capital will buy Envestnet in a $4.5 billion deal, adding to its portfolio a financial software vendor that counts 16 of the 20 biggest U.S. banks among its customers.