We all know it's important to conduct a trial or do a proof of concept as part of the buying process.
Do you have any advice for our community about the best way to conduct a trial or PoC?
How would you conduct a trial effectively? Are there any mistakes which should be avoided?
I am not sure if this question comes from a vendor or customer so the response is somewhat generic. If you are the technical customer or end user, try to be involved in the process start to end. If possible, be the hands on the keyboard. No better way to understand the solution if you are going to be the user of it in the future. If you are the vendor promoting ease of use, there is no better way to sell your product to the technical team.
I have managed a lot of data replication, protection, and archiving POCs. Two requirements always stand out. Success criteria and POC type. As a vendor delivering the POC, you will fail 90% of the time without clearly defining these up front. As a customer, you should have a clear idea about why you are investing your time in POC and what you expect to gain from it.
POCs should not be a training exercise. They are a path to purchase a solution for a budgeted project. If you are just kicking the tires, consider the free or self-paced options provided by many vendors. These include on-line labs and downloadable virtual machines or trial software. These cannot be considered a POC in my book.
Now the two key components for a successful POC.
#1 - Define as a Functional or Performance POC
Decide whether you are running a functional or performance-based POC. If you are the vendor, make sure the customer is aware of the limitation of a functional POC in a limited resource environment. Don't allow a Functional POC to become a Performance POC. Been there. Done that. It's never a success.
Functional testing is easier. There is no requirement for measured performance so sizing the environment is a minor issue. Just has to be "fast enough" to keep your attention. They usually cover base installation, backup target configuration, agent configuration, test backups and restores, reporting, alerting, etc. Data sets are generally small. It can be executed in a limited environment with virtual machines. Sometimes the vendor can supply access to a remote lab environment such as the VMware vSAN lab. Sometimes it can be delivered as a preconfigured VM downloaded from the vendor.
Performance testing is complicated. Speeds and feeds matter. You will not be able to backup your entire live environment so you have to build a test environment to mimic it as close as possible if you are looking for GB/sec measurements. Success Criteria become golden in performance tests. You will be following the recommended hardware configuration supplied by the vendor.
#2 - Success Criteria
Define clear success criteria and stay with the plan. This will avoid scope creep where testing has no endpoint.
A test plan can be extremely difficult to create from scratch. Take the time because it is key to a fair and complete test. It will make you think about the purpose of the test. Most vendors have boilerplate POC documents. They are a good starting point but they almost always focus on the strength of the product. If you are the customer performing comparison testing, blend them into a single document.
Some or all of the success criteria should meet the "must have" requirements of a published RFP if it exists.
Test criteria should not be too detailed, especially to favor a particular solution UNLESS that is a pass/fail test.
Define a start and end date based on the testing requirements. Testing should be sequenced. Test backup of app A, app B, os C.. Don't jump back and forth between Oracle and Sharepoint for example. Complete one, deal with any issues, check the boxes, and move on.
DR, Performance, and SLA testing absolutely require detailed planning. Too much to detail in this short response. Imagine a POC where you are faced with "I need to recover my 50 TB Oracle server off-site with an RPO of 5 seconds and an RTO of 5 minutes".
In a large POC, you might have regularly scheduled meetings or conference calls for updates on the progress and to deal with issues.
Include a site survey covering security and the network configuration, Prepare to deal with fixed IPs, firewalls, ports, Active Directory, etc. Nothing like a backup solution to break a network and bring the testing to a standstill. Make sure you have a clear understanding of the environment. I once had a POC where they were migrating some AD domains that were part of the test infrastructure. Unknown to me. Needless to say, we faced constant failures.
Define the hardware and configuration requirements on a per server basis. OS, partition sizes, network, etc. This applies to the backup infrastructure servers and the servers that will be the source of the backup data.
Include all the key contacts with access information to servers.
Make sure you have ALL the required resources (human and compute) resources available on the start date. For example, you might need help from an Oracle DBA or SME on day 2 to continue the installation.
Define a process to modify the plan. I've seen cases where another department sees the shiny new object and wants to jump into testing their app after the plan was approved and tests begin. Plan to deal with this exception in the testing procedure but not deviate from accomplishing the original success criteria. It should be approved by management.
Define what is considered critical to the success of the test, what is a nice to have feature, and optionally, what doesn't matter at all. Be specific. Include application versions if it matters. You might judge the test completion as pass / partial pass / no pass or a percentage of how it meets the criteria. Don't use subjective rankings. Add a column next to the test for comments for subjective comments.
If you are comparison testing two or more solutions, make sure you can test "apples to apples" across the POC candidates. All vendors should be tested to the same standard. It can be difficult to compare an appliance to an enterprise software solution. The appliance will win the easy to install checkbox but might fail in the ease of expansion category because it requires a new, larger box.
Consider the future in a POC, not just how it functions today. For example, you should think about the process to add additional capacity locally or bring on new sites/servers.
NOTE: Content here subject to updates if I think of something new or helpful.
Was going to write a lengthy response but yours is spot on Gary. I will only add that the front end and back end of every SMART goal is to be Specific and Timely. Document what is important to test and what the criteria for passing are BEFORE you ever take delivery. Then put an expected time for this POC to complete and what would be a successful test.
The only other thing I would add is if the vendor is not providing technical resources to drive and/or assist during the POC...don't waste your time. But, if you expect the vendor to devote the resources, you can also expect the vendor to hold you to a purchasing decision when/if everything passes with flying colors.