In addition to Jesus giving Peter the keys in Matthew 16, Peter is shown to be the leader of the church in another place.
After Paul’s first missionary trip in the 40’s AD, Paul went back to Antioch to see how things were going. While he was there a delegation from Jerusalem came to Antioch to tell the Christians there that all new Christians must hold to the law of Moses. All men had to be circumcised, they had to only eat kosher foods, and so on. The story is recorded in Acts 15
In Acts 15:2, Paul said, in effect, you’re nuts, it’s not going to happen. What did they do? Did everyone just say “Paul has spoken, we must obey”? No, they all went to Jerusalem to see what everyone else thought.
They all get together to discuss this very divisive topic (I wouldn’t want to get circumcised as an adult, but maybe that’s just me) and what happens? In Acts 15:7, who stands up to make this decision? James? Paul? No, it was Peter.
Catholics today don’t trace the line of popes back to Peter “just because”. We do so because he was the go-to guy for the early church, in addition to having the keys given to him by Jesus in Mt 16:18.