Institutional evolution and multiple modes of cooperation