Merck & Co. Inc. will spend up to $610 million to acquire a five-year-old biotech startup out of Cambridge.