I think perhaps I was a bit unclear. He also said, before that, "I know what this place is. It's an illusion. It's a cruel, elaborate trick conjured by the weak to inspire fear. A desperate attempt at control." He could think that the TVA are a group that's captured him, that are preventing him from using magic, that could even kill him, while also believing that they're lying about who they are, what they do, and, in particular, that the images they're showing him of his mother being killed are fake. He thought (or wanted to believe) some or all of it was lies, in particular the (from his perspective) future timeline they were showing him.

So I felt the big significance of the infinity stones was convincing him that it was real. Personally I took the "is this the greatest power in the universe" line as more of a throwaway thing, especially in the context of Casey being the one having them, but I can see how it could be taken as Loki literally expressing the notion that the possession of the stones indicates the TVA are the greatest power in the universe. But even if you take it that way, I think the significance of convincing him of the reality of the situation is more significant; his immediate action right after he finds the Tesseract and the other infinity stones is to go and watch the rest of his timeline. And recognising the reality of the situation and the power of the TVA doesn't mean he respects them; he then messes around with the TVA hunter. I don't think Loki is really the respecting power type.