Software Verification - All You Need to Know

Before we take a look at verification, let’s quickly re-visit software escrow so we understand just where verification fits in.

What is Software Escrow?

A developer creates a new application and wants to license it or offer it to a business that wants to use it.

The business – the end-user of the software, is concerned about relying on software from a developer with an unproven track record. What if the developer goes out of business? How will the end-user get bugs fixed or new features added?

If the end-user no longer has access to the software or support, their business would suffer or go out of business too.

To solve this problem the end-user wants the source code so they can keep using and modifying the application if the developer is no longer around to support them.

But the developer doesn’t want to give the user the actual source code because that’s their intellectual property. Giving away the source code is like giving away their secrets.

The solution they both agree to is an escrow arrangement. The developer gives their source code to an escrow agent.

The escrow agent keeps the code safe and only releases the code to the business when a release event occurs – like the developer going out of business.

What Should Be Held In Escrow?

The materials held in escrow can consist of many things; the source code files, scripts, libraries, build instructions, data files, third-party tools and more. What must be held in escrow is everything that is ever needed for an independent person to be able to build, modify and maintain the application.

What should not be held in escrow is:

Components readily available from the marketplace; like operating systems, databases, third-party toolsets. However each of these components should be documented in the build instructions.

Data;Usually data is owned by the customer and therefore should not be contained in the developer’s intellectual property or part of the escrow lodgment. The customer should have methods to regularly back-up and secure their data.

What is a Release Event?

A release event is a measurable indicator agreed by the parties to the escrow agreement. It is recorded within the escrow agreement. Once the indicator is met, the materials held by the escrow agent are released to the beneficiary – the end-user.

Examples of common release events include:

  • Bankruptcy of Developer
  • Developer is subject to any form of insolvency or involuntary administration
  • Developer has ceased to maintain or support the Materials
  • Developer ceases to conduct business
  • Key Person at Developer dies or is incapacitated in any way.

How do I know all the right materials are in my escrow?

In short, you don’t know. An escrow agent holds materials and releases the materials to the beneficiary when a release event is met. We have no insight into what is held in escrow.

Until we release the escrow materials to the end-user, no one can be sure exactly what the developer has included. Even if the developer is diligent with their escrow lodgements, there may be build instructions or a third party tools that they simply forgot to include. Waiting until the source code is released from escrow is too late to find out that the materials are incomplete and the application cannot be compiled or used.

Our experience warns us that over 70% of escrow lodgments are incomplete. They almost always needed additional input from the developer before they could be used.

That’s where verification comes in.

Escrow is a key insurance plan for any business that wants to reduce the risk from critical software that they rely on.

Verification ensures that your escrow arrangements will work you when you need them to.

Verification

To fix this, we can perform verification on escrow lodgements.

Trust your business partners     ….but verify

Verification is the process of making sure that all of the materials needed for an independent party to compile the application are contained within the escrow lodgement. That includes the instructions and processes that can be followed without the support of the developer.

We have developed a tiered process of verification. Each step undertakes a further level of analysis to provide an incrementally higher level of assurance, comfort and risk mitigation. We guide businesses through the process of considering which level of verification is right for them.

Level 0: Lodgment Validity

Each time the developer makes a lodgement, we validate the materials lodged into escrow for you by undertaking a physical media check, checking file compression methods and results, confirming encryption and passwords are valid, scanning for embedded threats and randomly sampling files to confirm human readability.

As the first step in verification, Level 0 will provide you with the comfort that your escrow lodgement is free from obvious flaws that could prevent its use when you need it.

Level 1: Escrow Review

This level of verification builds on Level 0 by comparing the lodgement with your escrow agreement. We make sure that the developer actually deposited what they said they would deposit. It includes ensuring all the correct files have been incorporated in the deposit like build instructions and third party tools. This may sound simple but there can be hundreds or even thousands of files associated with a complete set of the source code and they can be distributed across the complex directory structure.

Verifying that the right files are included in the right places is the first step towards making sure that the developer didn’t simply make a mistake when collecting the artefacts that are needed.

Level 2: Analysis and Test Compile

Level 2 verification often includes spending time on-site at the developer’s business. It includes an extensive review of the documentation and processes. It follows build instructions and test compiles the code. This is a major step towards making sure that a complete set of the source code has been deposited. If any file, script or library is missing, it will show up during compilation as an error. This step tells us that what the developer meant to deposit for source-code compilation included everything that they needed to deposit.

Level 3: Code Compilation and System Testing

Even if we can be sure that the source code that was lodged is complete, does it mean the software will function as required? While the analysis and test compiling of Level 2 Verification provides that comfort, a single file or line of code could be present that includes some bug, in-compatibility or broken functionality.

In this final Level 3 Verification, Harbinger will setup, configure and install the end-user’s production environment in a quarantine laboratory. We run agreed test cases to verify that all the software will function as expected and work correctly.

We also validate the source-code to uncover actual or potential coding issues. Source-code validation can reduce the degradation of the software’s usefulness over time.

What level of verification is right for my business?

As you can see, there are several verification options that you should consider. It is not usually necessary to always perform Level 3 Compilation for every escrow lodgement. Some businesses opt for Level 3 Compilation once annually and Level 1 Review for each other lodgement.

The Matrix below provides a side-by-side comparison of Harbinger’s standard verification services. Of course we are always happy to arrange a bespoke service to meet your particular needs.

Principally, the verification arrangement you choose should take into account your businesses appetite to accept risk and the impact of a software failure on your business reputation.  A simple risk matrix can help fine-tune your options:


What are the fees and costs for verification?

The cost of conducting verification has two components; A fixed fee to establish and manage the project (indicated in the table below) and an hourly chargefor the analyst’s time that conduct the verification.

The time it takes depends on the complexity of software, accuracy of instructions, number of files, components and systems required and several other factors.

We can of course undertake verification to your fixed budget.

Level 0 – Validate

$550

per audit plus T&M

    • Single technician by the hour
    • At Harbinger’s offices
    • Readable files have been received (random 5%)
    • Encryption and protection is in place
    • Encryption keys are effective
    • Files are virus-free
Enquire Now

Level 1 – Review

$1500

per audit plus T&M

    • Single technician by the hour
    • At Harbinger’s offices
    • Level 0 plus…
    • Readable files have been received (random 15%)
    • File listings are in accordance with the escrow agreement
    • Detailed review report
Enquire Now

Level 2 – Analysis

$2750

per audit plus T&M

    • Two technicians by the hour
    • On-site at Vendor’s premises
    • Level 0 & 1 plus…
    • Review of materials for required components
    • Comparison of materials with agreement
    • Test compile of source code
    • Review of build documentation
    • Detailed analysis report
Most Popular

Level 3 – Compilation

$5500

per audit plus T&M

    • Two technicians by the hour
    • On-site at Vendor’s premises
    • Level 0, 1 & 2 plus…
    • Construction of a build platform
    • Validation of the build instructions
    • Troubleshooting any problems
    • Review and manage rectifying for errors
    • Detailed compilation report
Enquire Now

Other Verification Options

Supervised Compilation

Supervised compilation is becoming recognised as a bona-fide method to assure that the materials held in escrow will work when called upon. We witness the developer as they compile the source code into working software and lodge the source code into escrow.

Pros:

  • Usually faster that traditional verification
  • Can provide a mid-range level of verification at a reduced cost

Cons:

  • Does not verify documentation, processes, blueprints or instructions
  • Requires the good-will and time of the developer to assist with the process
  • Must be carried out at the developer’s premises
  • May involve travel costs

Binary Comparison

This option can only be done when a release of the application is made to the end-user at the same time that a lodgement of the source code has been made with Harbinger.

A unique number called a hash-key is generated from both the released application by the end-user and escrow lodgement. Comparison of the two hash-keys from the compiled code and the version of the application that was released to the end-user guarantees that the escrowed material is identical to that which was used to compile the application currently being used by the end-user.

Pros:

  • Faster that traditional verification or witnessed compilation
  • Highly cost effective
  • Can be done without the involvement of the vendor

Cons:

  • Does not verify documentation, processes, blueprints or instructions
  • Will not disclose where errors may be
  • Few verification metrics available

When a problem arises, you might be ok or you might not.

If you don’t want to gamble with your company’s future, verification needs to be a key component in your escrow program.

Leave a Comment

Your email address will not be published.

*