Automated End-to-End Testing and Certification

Volume 3, Issue 2

Printer-friendly versionPDF version

Volume 3, Issue 2

US Office
Ascert, LLC
759 Bridgeway
Sausalito, CA 94965
Tel. +1 415-339-8500

UK Office
Ascert, Ltd
3rd Floor Signet House
49 - 51 Farringdon Road
London, EC1M 3JP
England, UK
Tel. +44 (20) 7488 3470

Crash Test Dummies

Don't be a "test dummy". Get testing smarts through VersaTest!

Where We Have Been          

Orlando, FL     
April 5-8

Electronic Transaction Association, Las Vegas, NV
April 21 - 23

EBUG - Prague, Czech Republic May 5

ACE - Washington, DC
June 1 - 2

HPTF - Las Vegas, NV
June 15-18                     
(See who won)

Where We Are Going 

Colorado TUG
Denver, CO,
August 20

Canadian TUG
Mississauga, ON, Canada
October 21 - 22

Boston, MA - BAI Retail Delivery Conference & Expo                    
November 4

London, UK - Ascert User Group
November 12 - 13
Welcome to this edition of the Testing Times.  Inside we continue our discussion on gaining efficiencies in testing through process and automation.  Have you ever thought about the costs saved by not testing?  Check out "The Cost of Not Testing" before you make any decisions on that one.  Let us know what you think at

The Cost of Not Testing

Have you ever thought that maybe you could cut costs by cutting testing?  This could be the biggest mistake you ever made!  Find out about the hidden costs of not testing...or almost as bad...testing inadequately.

More Details

RBS WorldPay Selects VersaTest Automator

RBS Worldpay chooses VersaTest Automator as the engine running testing automation.

More Details

Level Four and Ascert Win Recognition for Innovation

A combined end-to-end ATM testing solution was recognized in Finextra's Innovation Showcase for helping Barclays take a big-bang approach to their ATM refresh.

More Details

Questions & Answers

Ascert's product developers give answers to some of the questions they regularly get asked. See what other users are talking about and ask your own questions.

More Details

The Cost of Not Testing

Testing in an organization occurs on many levels and in many different departments. Measuring the costs of testing can be achieved and it can be boiled down to a dollar figure.  How many hours and what other types of hardware resources were used are typical elements that go into measuring this cost.  But what is the cost of not testing; or not testing adequately? 

The risk associated with not testing is, of course, that an application or system will fail in some way during production use. As John Watkins points out in his book "Testing It", "Many examples exist, particularly where systems have had safety critical, business critical, or security critical applications, where failure of the system has, either through litigation or loss of public confidence, resulted in the provider of the software going out of business."  So there it is, the ultimate cost of inadequate testing...the death sentence for an organization.  Many failure scenarios produce less drastic results, of course, but they all incur a direct cost to repair, and the later they are identified in the development cycle, the higher the cost associated with fixing them.

Bugs and errors in the code are all part of the development process.  You can't avoid them, but you must find them.  So, how do errors begin?  Picture layer upon layer of coding effort.  Each layer either is error free or it contains bugs.  The more layers there are, the more difficult it comes to discover were an error is occurring.  This is why developers run lots of unit testing before adding new code or modules. Unit testing will help to identify and correct problem areas that can be difficult or near impossible, without significant effort, to find later on. 

There are lots of different types of testing that can occur in an organization and a selection of those will be used for demonstration purposes.   Once modules are developed and begin to become connected, more integration testing occurs.  This ensures that modules operate together as well as they did separately!  Bringing the efforts of two or more developers together can be exciting...but also presents opportunities for disparity.  Integration testing can help to uncover these issues.

Next in the chain of testing to occur are system, stress, and regression testing.  Each of these types of testing is for different purposes and usually encompasses a much wider or larger set of features/functions that are being tested.

Perhaps an easier way to visualize the range of testing is to picture an inverted pyramid (see figure A).  As testing moves from the lowest level (unit) up to more encompassing testing (stress, functional) the area of the shape gets larger.  One might mistakenly surmise that the most important tests occur in this "larger" surface area.  They'd be wrong!

The inverted pyramid is not really about types of testing.  It's really about layers of cost.  The move from one level of development to the next adds layers of complexity and bugs or errors equal cost; cost of time to market, cost to find and fix the errors, cost of errors not fixed during development, and cost of product failing in service.  Not finding the errors in the early stages equals holding up the code/product release.  It means engineers' time wasted and the higher up the inverted pyramid the error/bug gets the higher is the cost of wasted man hours.If the errors aren't discovered during each phase of development, the difficulty of identifying and resolving them becomes greater.

So, the answer is testing, testing, and more testing!  The more testing conducted at the lowest levels the better.  This means that you must have a strong, repeatable testing discipline.  The best method handling this is through the use of testing software tools.  Home grown or self coded tools can work well but eventually the best course of action is to invest in externally produced tools from a reputable vendor.

Probably the most important reason to seek services from an outside vendor is that testing software is their core business and they likely see a greater variety of testing issues than you might in your own shop. The sophistication and operation of their software should reflect that.  Another reason to use testing tools from a third party is to avoid common mode failures. Building your own tools often means that the same assumptions are made in the testing tool as in the application being tested. These cancel each other out.  An example of this was NASA's mission to Mars where one group used imperial measurements and another used metric - they all tested their systems but using the same faulty assumptions (NASAMistake). 

Finally, increased testing probably means lots of repetition.  The more repetitive manual testing that occurs, the greater the chances for error.  Also, more testing means more man hours, which equates to increased costs...the very thing you're trying to cut out!  The only way to increase testing while decreasing costs is through test automation.  With a little bit of work, an experienced third party testing software provider can help you to calculate the ROI on implementing test automation in your operation.  It all comes down to the numbers.  It makes sense to test more to avoid the future costs of bugs or errors in your code.  But it doesn't make sense if the costs to test are greater than the benefits derived.

For more information on how software testing tools and testing automation tools can help your organization, send and email to

RBS WorldPay Selects VersaTest Automator for Implementing Testing Automation

RBS WorldPay, Inc. has joined a growing list of companies worldwide that have adopted the VersaTest Automator platform for internal and external testing.  VersaTest Automator, the latest in Ascert's family of software testing products adds new levels of user flexibility and automation that have never before existed.

"The new VersaTest Automator platform will take us to a whole new level of testing efficiencies," said Jim Tibbetts, vice president of product management for RBS WorldPay. "In addition, the Automator platform will enable RBS WorldPay to more rapidly deliver quality services and products to our merchants."

Andrew Mould, managing partner at Ascert, says, "The development of the Automator product really began more than two years ago when we realized that the industry was at a standstill as far as increasing efficiencies in testing POS, EFT, and ATM systems.  We decided that an entirely new approach to test environment development and methods of automating processes was the only way to take great leaps forward. Our VersaTest Automator product is the end result of those efforts and it addresses those issues by introducing a whole new generation of testing systems and methodologies."

"We are pleased with the early results of the VersaTest Automator product," said Tibbetts. "As one of the first companies in the U.S. to take advantage of this technology, we believe this product provides us with a significant competitive advantage; especially in the area of new POS development and certification with third-party vendors and customers. The Automator product provides us duplication and reliability of processes every time we run a test."

To learn more about VersaTest Automator, read the datasheet on our website.

Level Four and Ascert Win Recognition for Innovation


The implementation of a combined end-to-end ATM testing solution at Barclays Bank from Level Four, the provider of open standards-based ATM testing software, and Ascert, has been selected by Finextra to feature in its first 'Innovation Showcase'. Finextra, the independent newswire and information source for the financial technology community, sought to highlight some of the most interesting financial technology developments over the past 12 months and chose the project to feature in its 'Retail Channel Delivery and Management' category.

Finextra summarized the project in its showcase feature: "Barclays has deployed an end-to-end automated testing solution from Level Four and Ascert to support the biggest ATM refresh project by a UK bank in over a decade. The bank is moving away from proprietary ATM software and migrating more than 3,000 ATMs to an open standards environment, implementing Windows and packaged software such as BASE24 and ProCash."

"The project is also unique in the extent of its collaboration between the bank and its technology partners, Level Four and Ascert. Together, they have developed an advanced strategy for the rapid testing of the entire end-to-end transaction process, taking automated ATM testing to a completely new level. The testing environment went into pilot stage in September. To cope with this complex environment, Barclays has implemented BRIDGE:test from Level Four to automate the testing of the entire ATM stack and VersaTest from Ascert for message and host testing."

Elton Cane, strategy director at Finextra, commented: "What were we looking for? Newness, uniqueness and impact; clever ways of solving a problem; and projects at the leading edge of new industry trends." Commenting on the Barclays project, Cane added: "This kind of behind-the-scenes work, as well as being innovative in its scale and approach, will make it much easier for the bank to deliver new services via the ATM channel and be seen as innovative by its customers."

Martin Macmillan, Business Development Director at Level Four, added: "This is a real endorsement of the value that innovation can bring to banks' ATM networks. The complexities of open standards environments call for an end-to-end approach to ATM testing. Through automation, banks can carry out more extensive tests with greater frequency, enabling them to accommodate regular product upgrades and maximize the value of their ATM networks."

Mike Wainwright, Business Development Director in Ascert's London office, said: "The success of this project really validates the need for a rethink in end-to-end ATM testing to achieve the best possible results. We look forward to building on the successes already achieved by working collaboratively with Level Four and replicating that success with our other customers in the future."

Read the Finextra article.

Questions & Answers

 Q:  We are in the process of converting a VersaTest script running on our HP NonStop system to use TCP/IP communications instead of X.25.  After adding the TCP/IP construct and modifying the SESSIONS declaration to reference it, we're getting the error:

"Logical session identifier INTERFACE has not been assigned to a physical session...script not loaded"  

Our TCP/IP and SESSIONS declarations are show below; what's the problem?

TCP_IP mysock
    polarity client
    remote_address ENV_PARAM_GET("requester.remote.address","","")
    remote_port ENV_PARAM_GET("requester.port.number","","6061")
    message length at byte 0 field is decimal for 4

A: VersaTest/NSK allows the use of logical session identifiers in the SESSIONS declaration, in your SESSIONS declaration, this identifier is "INTERFACE". Specifying a session identifier in this way allows scripts to be written in a session independent manner and allows the assignment of the physical client session name to be deferred until runtime (facilitating the re-use of scripts without recompilation). The physical session name must then be specified at runtime using a file assignment. So, in your case, you simply need to set the assignment in your TACL environment before starting the script e.g "ASSIGN INTERFACE,$ztc0.#term1".

Q: We need to process messages that include a variable number of occurrences of a structure at the end of a message.  The occurrences are separated by delimiters with a value of HEX "1C" as shown in the following message structure. Unfortunately, the last occurrence of the sub-structure is followed by HEX "03" instead of HEX "1C", which causes the message processing to fail.  How can I deal with this situation?

+STRUCT resp_msg;
   field is int16             msg_len;
   field is ascii for 4   device_type;               
   field is ascii for 2   msg_num;                             
   field is ascii for 5   store_num;                             
   field is ascii for 2   register_num;                           
   field is ascii for 4   txn_num;                               
   field is ascii for 3   cashier_id;                             
   field is ascii for 1   request_type;                         
   field is ascii for 19  account_num;                           
   field is ascii for 1   originator_ind;                       
   field is ascii for 12  tran_date_time;                         
   field is ascii for 3   resp_code;                             
   string                 seperator;       
   STRUCT resp_fid OCCURS UPTO 15 UNTIL EOM;                       
      field is ascii for 1         fid;                                     
      field is ascii until {%H1C}  fid_data;                         
A: Sometimes message formats just aren't what you'd like and a bit of pre or post processing is required to deal with them.  As you point out, the last occurrence of fid_data won't be properly identified because it ends in HEX "03" rather than HEX "1C" as specified in the data structure.  This is a fairly common scenario and the way we've always dealt with it is to check the last byte of an incoming message to see if it has a value of "03" and, if so, replace that value with "1C", which allows the message to be handled appropriately using the defined structure.  Generally, this activity is performed in the "INBOUND" construct and a corresponding activity is performed in the OUTBOUND construct to replace "1C" in the final byte with "03".

Q: My company has just purchased VersaTest Automator.  We will be using Ascert's SPDH driver for regression testing and I have been assigned the task of defining the regression test cases for our environment.  What training is available?
A: Ascert offers the "VersaTest Automator Test Case Development Training" class, which provides one week of hands-on instruction in all aspects of test case definition.  This instructor-lead class is generally provided at the customer site and can accommodate up to 8 students.  For additional information or to schedule training,contact
Have a question? Send us an email to and ask the experts!

HPTF Raffle

Congratulations to Serge Temis from GE Healthcare for winning the new iPod Silver Shuffle.  He stopped by the Ascert booth, got a 3 minute sales pitch, and won!  

Dr. Ding Dong's Note:  This is cool!  Sometimes listening to sales people pays off!!


Copyright © Ascert, LLC. All rights reserved. Duplication of any portion of the Testing Times is permitted with prior written approval from Ascert, LLC.