Planning for performance is not a premature optimization
Donald Knuth's comments on premature optimizations have been with us since 1974. And while it is true that one shouldn't try to solve performance problems that they don't have, not planning for performance is something we should do. when is a performance related activity premature and when it is necessary. The answer depends on what phase of your project is in.