Yeah, you definitely need to be active working with your doctor to find the schedule that's right for you.
I've got the history in my family so I got my first earlier than normal; polyps the first and second time got me on a 3 year schedule; then clear for the next two put me back at a 5 year schedule.
And, as others have said, you gotta monitor your poop and make sure you see the doctor regardless of schedule if anything trends abnormal there.