How to choose a CMS with the right software ecosystem – these are the questions you should be asking potential vendors

Here, I’ll walk through the basics of what a software platform ecosystem is, as well as what kinds of questions you should be asking potential CMS vendors to ensure the solution is viable for your organization in the long-run.

The following chapter is excerpted from my book, "Things You Should Know: 25 Lessons I’ve Learned About Selecting Content Technology and Services". Download the full version of the book.

Lesson #11

A vendor’s ecosystem should be evaluated as a core feature

Eleven years ago, when Episerver had first entered the North American market and Blend Interactive was one of the few integrators working with it, a certain consultant I knew would always respond to my evangelism the same way: “Let me know when they have more than a half-dozen partners.”

This comment would annoy me, because in my head, we worked with the software, and that was enough.

But I’ve since come to understand that this wasn’t enough. Software doesn’t exist in a vacuum, and what happens outside of the vendor is just as important as the system and vendor themselves. Software needs to be supported by a network of resources, known as an “ecosystem.”

The ecosystem of software platform consists of:

1. An active community
2. Accurate, comprehensive documentation
3. Qualified integration partners

We’ll unpack these items individually.

Software ecosystem part 1: An active community

Community can be both formal and informal. The vendor probably provides a set of discussion forums, and good vendors encourage or even require their professional services team to spend time there answering questions. Many vendors reward members with some type of “MVP” status for helping other members of the community.

Informal communities matter too. A lot of problems can be resolved via social channels like Twitter or forum sites like Stack Overflow, provided there’s critical mass around the product. It can be instructive to search Stack Overflow for questions people are asking about the CMS you’re evaluating.¹ Evaluate both the volume of questions and their content. What problems are people having? How many people are willing to answer those questions?

Back in my corporate days, my company purchased a very expensive enterprise content management system. I spent months trying to figure it out, and all that time, I felt like I was working alone. In the back of my mind, I was sure that there was a group of people somewhere who were suffering through the same growing pains I was, but I just hadn’t been able to connect. Eventually, almost accidentally, I found a Yahoo! Group for this software, and there it was. I had finally stumbled upon another group of customers of this particular vendor, struggling with the same problems, and willing to talk about it. It was revelatory, and it changed how the software was perceived and implemented at my organization.

Software ecosystem part 2: Accurate, comprehensive documentation

Documentation ranges from auto-generated API docs to blog posts to sample code to accurate feature lists. Lack of documentation is probably Complaint #1 among developers trying to work with a vendor’s system.

I remember months of working with a particular vendor’s software without enough documentation. My last resort was simply to decompile their code (undoubtedly in gross violation of their license agreement), and wade through it line-by-line to figure out what was going on.

For many developers, sample code is the pinnacle of documentation. It tends to get vilified for encouraging blind copying and pasting, but reverse engineering is a common way to figure a system out – give a developer code that works, and they’ll examine it, take it apart, and modify it to do what they want. More software has been learned in this fashion than probably any other.

Many vendors offer “reference implementations” of their software. For CMS, this would be a sample website the vendor uses for demos, and offers as a “sandbox” or trial for customers who want hands-on experience with the system. The hope is that this implementation has enough breadth of functionality and represents good coding and usage practices so that it can be examined as reference.

Often, customers will start new projects from the reference implementation, modifying it for their particular needs. The advisability of this varies greatly, depending on the quality of the reference site and the applicability of it to the customer’s requirements. Some vendors are horrified that customers do this, while others have specifically designed their reference implementations for this purpose.

Look through this documentation and reference code before you buy. Ask your developers to look through it. Make sure it’s public.

I’ve seen vendors who lock away their documentation behind a login, which doesn’t make sense. If you can get a free account – by joining their developer community, for example – then that’s fine, but in some cases, vendors will only allow access to paying customers, as if their documentation was some trade secret.

Look for third-party documentation. Are people writing about the platform? Are there blog posts about how to implement or work with features? Go search Amazon. Has anyone written a book about it? Visit independent training sites like PluralSight or O’Reilly. Does anyone maintain video tutorials on YouTube?

The level of documentation and community activity is a reasonable proxy for how well that CMS has been received by the market and the level of adoption it’s received.

Software ecosystem part 3: Qualified integration partners

Finally, since that conversation with my analyst friend about Episerver eleven years ago, I’ve come to understand how much partner networks matter, the hard way. The partner network of a CMS vendor is your safety net. It’s the thing you can fall back on if the vendor or integrator lets you down. It’s a soft place to land.

Partner networks vary greatly in size and intensity. To be a “partner” of a vendor means an integrator gets some special perks – some extra licenses, recognition, more communication, and hopefully sales leads. In exchange, the vendor hopes that the integrator will evangelize their software and sell it for projects.

Sometimes an integrator only works with a single system and both sides benefit from a close affiliation. However, it’s not uncommon for the relationship to be asymmetrical, where an integrator is just looking for targets of opportunity. They’ll promote a vendor’s software if it works for them, otherwise they’ll gladly use something else. They’re “badge collectors” who will claim expertise in any system if it will help them obtain services work.

This means that the generic descriptor of “partner” is not absolute. Not all partners are equal, and you can’t trust this means an integrator actually has practical skill in a system.

A few years back, I watched a large customer purchase a CMS from a small, little-known vendor. Unfortunately, the vendor was small enough that this customer quickly overwhelmed the vendor’s professional services team.

The customer asked me to help them find a third-party integrator for help. The vendor’s “Partners” page listed a dozen companies, but every company I contacted had only done one implementation, and had only become a partner to get a development license (a common perk of partnership). Some of them didn’t even know their logo was on the vendor’s website.

We came to understand that there really was no partner network. And the vendor’s professional services staff was less than a dozen. There were less than 12 people in the world available as competent engineers of this system. And they were busy, all the time.

That’s not a good thing to find out when you’re six figures deep in licensing fees.

Honestly, I don’t enjoy emphasizing ecosystems, because the lack of one is the highest bar to entry for a new vendor to break into the mainstream. Newer vendors have to contend with building their system, supporting their customers, and, at the same time, convincing new people to use it to the extent that an ecosystem develops.

It’s a classic chicken and egg situation – few people are using a new system, so there’s little in way of ecosystem; and there’s little ecosystem because few people are using it. In the best-case scenario, it becomes a positive reinforcement loop. More users mean more ecosystem, and more ecosystem means more users. But the opposite is sadly more likely to be true. Smaller vendors can wallow in a tiny
market share and never break out because they can’t develop their ecosystem.

If you consider the major CMS vendors compared to all the others, the difference that stands out are the size of intensity of their partner networks. Look outside those systems, and you’ll still find some great software, but drastically fewer partners. Some very solid CMS platforms might only have one or two focused partners in North America.

I genuinely feel badly for smaller vendors that struggle with their ecosystem. The lack of one is a huge impediment, and they’re not easy to develop.

The fact remains that the lack of a viable ecosystem can put customers in a very bad position. You need to ask and verify questions about your vendor’s ecosystem, and vendors need to start cultivating their ecosystem and partner network like it was a core feature of the product.

^{1. Bonus points if the CMS is well-known enough to have earned its own tag.}