Seth Gotlieb has an interesting blog post on CMSWire about the CMS software selection process. He argues that a requirements matrix, and evaluating and scoring against these requirements is a wasted effort.
He makes some valid points. With any analysis that involves numbers, it is always dangerous to assign to much value to the actual number and its precision. In addition, it is easy to get lost in the details, and losing oversight on the big picture.
From our experience, the requirements matrix still plays an important role. It ensures that all different areas are covered, from end user to IT needs. The matrix creation process also triggers important discussions with the different stakeholders.
However, equally important in the selection process are vendor demonstrations. They give users a good perspective on the solution, forces them to think about how a specific product would work within their process and environment. The follow on discussions then provide a forum to identify any red flags with specific software packages.
Once the leading products have been evaluated and scored, vendor demonstrations have taken place and discussed, we take a step back and look at the overall results. Based on these elements, a decision can be made that meets the majority of requirements, and also is the right fit for the organization.