I don’t see how the Colts is a rivalry to the Patriots except maybe in the eyes of Colts fans. Peyton is gone and when he left for the Broncos he pretty much took that rivalry with him.
Any team signing Brady (including the Patriots) might want to draft for the future just in case.