Quick Play goes with your party. Wait for the host to start one, or leave the party to play on your own.