Stormy markets have left quants reeling. Can a new flurry of machine-learning hedge funds whose robots learn from their own mistakes do any better?