Harvard Book Store is abandoning its plans to open a far larger location at the Prudential Center, saying in a message online that the expansion has proven “unsustainable.”