If the team is cross-functional and collaborative, the team can use agile principles.
You are discussing what I call "emergent" projects, where you may have to:
- Iterate on the requirements to see what you might want to define for release criteria. Then, you can proceed.
- Iterate on options, possibly with spikes (See https://www.agileconnection.com/article/need-learn-more-about-work-you-r...)
- Start the project and decide when you can define release criteria.
I have a number of suggestions in my book, Manage It! Your Guide to Modern, Pragmatic Project Management. See the review here: https://www.agileconnection.com/book/manage-it-your-guide-modern-pragmat... and info about the book here: http://www.jrothman.com/books/manage-it-your-guide-to-modern-pragmatic-p...
I have written about emergent projects. See http://www.jrothman.com/mpd/project-management/2013/12/do-you-have-an-em...
