I don't think this visit to Trenzalore was THE visit - i.e. the Fall of the Eleventh - because, well, he will clearly be there (again) at some time in his future where it becomes his grave, so I'm thinking it's that story that's still to come that the Silence were so afraid of.