With the digitalization of the financial market, dealers are increasingly handling market-making activities by algorithms. Recent antitrust literature raises concerns on collusion caused by artificial intelligence. This paper studies the possibility of cooperation between market makers via independent Q-learning. Market making with inventory risk is formulated as a repeated general-sum game. Under a stag-hunt type payoff, we find that market makers can learn cooperative strategies without communication. In general, high spreads can have the largest probability even when the lowest spread is the unique Nash equilibrium. Moreover, introducing more agents into the game does not necessarily eliminate the presence of supra-competitive spreads.