-
- North America
- EMEA
A Comprehensive Guide
a. Choose the right AI.
Not all AIs are equal, and many are designed for specific use cases, from finding the perfect restaurant for an offsite to developing revolutionary eco-friendly waterproof coatings. There are many types of AI (and it can get esoteric to try to agree on the number and divisions). Even within a category of AI, different AI offerings use different large language models (LLMs) with varying constraints and reach. You may use Midjourney or Canva if you need cosmic squirrel art or new LinkedIn banner images. Grammarly’s powerful AI features are great for getting your first novel out of your brain and to the finish line. StyleAI might help you with that special outfit for your cousin’s wedding.
For enterprise software shopping, you need a research-oriented AI that can perform web searches for the most up-to-date information possible.
Use an AI with a “research” mode turned on to make sure it is double-checking its work and providing sources for all facts that you can double-check and explore. It may take a while to generate output, but the quality will dramatically improve. Remember that cloud-based enterprise software like Arena continuously releases new features and updates. Using an AI with stale data may give you inaccurate information and should not be trusted. We’ve provided some AI recommendations in the table below based on our testing to get you started. You can also use AI to help you find new AI research tools.
b. Explore your AI’s settings.
Start by asking your AI if it has special settings or access to private data. Some AIs, like Microsoft 365 Copilot and Claude, may have access to special settings that are designed for your specific use case. Your AI may also have access to your company’s internal documents and communications, which can help it tailor its findings to your specific needs.
a. Requirements are, well, still required.
Before you can construct a good prompt, you need to identify business needs and capture your requirements. What your team needs is not identical to what other teams need. AI will provide generic non-actionable results if you don’t provide the context of your business needs and requirements. If you aren’t entirely sure if your requirement list is complete or accurate, consider also using AI to help review it. You can refer to our How to Buy Software Guide on the process of building a team and doing this work if you aren’t sure where to start.
b. Good input = better output.
Prompt engineering is a skill, and every AI has its own personalities and peculiarities. Getting one AI to generate useful information may require a very different prompt from another. Some may work best with a detailed, lengthy prompt while others generate more useful answers from shorter, general questions. Others are sycophantic and work best when you tell them to act as devil’s advocate. However, you don’t have to have a degree in machine learning or AI to construct a good prompt. You do need an understanding of what a good prompt is, an iterative experience, and the willingness in the community to share good prompts. And, we are a sharing people at Arena by PTC, so use these starter prompts and go from there.
c. Try multiple AI!
In our experiments, the results can vary dramatically. Go wild and try several. If this is your first time using AI for a structured research project, running the prompt through multiple will help you see the differences between tools and give you better overall content to use in your search.
a. AI is fallible and AI hallucinates.
Hallucinations are when our sensory perception does not correspond to the external stimuli. AI hallucinations occur when the algorithmic system gives information that seems plausible but is false, inaccurate, or misleading. (Dive deeper into this topic with researchers here.) This is particularly true when asking questions about enterprise software, for which publicly available documentation is often unavailable. You can mitigate this issue by turning on “Research mode,” but even this is not a guarantee. Software selection is important to your company and, therefore, to you and your team. AI hallucinations can cause real damage to yourself, your colleagues, and your business when used unchecked. If something sounds too good to be true, make sure to verify by reviewing the AI’s sources. See step 4 for more recommendations on how to mitigate this risk.
b. Iterate with the AI to improve the results.
Ask follow-up questions to dig deeper into a topic or have the AI consider the issue from a different perspective. You can have conversations with the AI to improve the results (until you run out of credits if using a free or limited version of the tool). You may also find that the AI’s clarifying questions or results give you improvements to the initial prompt that you want to re-run or run in another tool.
Software vendors do not publish everything about their companies, product roadmaps, functionality, and customer commitments. As AI-powered search and research tools have gained dominance and are fast replacing traditional web searches, all companies are now working to rapidly catch up to the new AEO (answer engine optimization) that is replacing traditional SEO (search engine optimization). What does this mean? All of us—including your own company—are busy re-evaluating the content breadth, depth, and structure on our public sites and other locations.
a. Don’t abdicate to AI.
Humans need to be in the loop at every step—from identifying business needs and requirements to running well-executed AI research to giving vendors the chance to respond to what you have learned so far in your research—answering your questions, providing customer references, and all the other critical information you need to make a great decision. Additionally, you may have a very detailed requirements list with weightings and other roll-up considerations. Generic AI research tools often lack access to the proprietary or nonpublic data needed to fully understand your requirements. As a result, they may misinterpret what you’ve uploaded or fail to assess whether a specific vendor can meet your needs. Just like working with a colleague, asking the AI to rephrase your requirements in its own words can help surface misunderstandings and clarify intent.
b. Help vendors help you.
Vendors can better respond to your search when you are transparent about your business needs and requirements. And, if you have used AI tools for research, let the vendors know and ask them to respond to the results and prove out solutions.
