
                          15APR89
                       AIPS DDT TAPE

     The accompanying tape contains the data files needed to execute
the "DDT" test on an AIPS installation.  The POPS scripts and HELP
files to run the DDT tests are distributed with AIPS in the RUNSYS and
HLPFIL areas.  The test procedures will not work on AIPS releases
before that of 15APR89 due to changes in the test data and the
organization on the tape.  The tape is at a density given on the
physical label on the tape reel.  1600-bpi tapes contain data for 2
tests: "SMALL" and "MEDIUM", while 6250-bpi tapes contain an
additional "LARGE" problem.  The text of the DDT.HLP file explains
the use of the DDT test and is shown below.

; DDT
;---------------------------------------------------------------
;! verifies correctness and performance using standard problems
;# RUN General
;-----------------------------------------------------------------------
;;  Copyright (C) 1995
;;  Associated Universities, Inc. Washington DC, USA.
;;
;;  This program is free software; you can redistribute it and/or
;;  modify it under the terms of the GNU General Public License as
;;  published by the Free Software Foundation; either version 2 of
;;  the License, or (at your option) any later version.
;;
;;  This program is distributed in the hope that it will be useful,
;;  but WITHOUT ANY WARRANTY; without even the implied warranty of
;;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;  GNU General Public License for more details.
;;
;;  You should have received a copy of the GNU General Public
;;  License along with this program; if not, write to the Free
;;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
;;  MA 02139, USA.
;;
;;  Correspondence concerning AIPS should be addressed as follows:
;;         Internet email: aipsmail@nrao.edu.
;;         Postal address: AIPS Project Office
;;                         National Radio Astronomy Observatory
;;                         520 Edgemont Road
;;                         Charlottesville, VA 22903-2475 USA
;-----------------------------------------------------------------------
;---------------------------------------------------------------
DDT       LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDT  :  Verification/timing test; see DDTLOAD.001, DDTEXEC.001
TCODE                              INIT,TEST,READ, or WRIT
TMODE                              T or M
TMASK           1.0       127.0    test selection bit mask
DDTSIZE                            'SMALL','MEDIUM','LARGE' test
DDISK           1.0         9.0    Disk drive #: master UVDATA
MDISK           1.0         9.0    Disk drive #: other masters
TDISK           1.0         9.0    Disk drive #: test images
IOTAPE          1.0         9.0    Input/Output tape drive #.
EDGSKP          0.0        64.0    Pixels to skip at edges
BADDISK         0.0         9.0       (1,3 good on CVAX)
VERSION
----------------------------------------------------------------
DDT
Type: RUN file
Use:  DDT performs the AIPS verification test of an AIPS system.
      See RUN files DDTLOAD.001 and DDTEXEC.001.  The purpose of
      the inputs file is to perform bounds checks on the
      parameters and to enable the use of TPUT and TGET.
Adverbs:
 TCODE    String*4      'INIT', 'TEST', 'READ', or 'WRIT'
 TMODE    String*1      'T' or 'M'
 TMASK    Scalar        test selection bit mask (1-127)
 DDTSIZE  String*8      'SMALL', 'MEDIUM', 'LARGE' test
 DDISK    Scalar        Disk drive # (1-9) master UVDATA
 MDISK    Scalar        Disk drive # (1-9) other master files
 TDISK    Scalar        Disk drive # (1-9) test files
 IOTAPE   Scalar        Input AND output tape drive #. (1-9)
 EDGSKP   Scalar        Number of pixels to skip at edges in
                        comparing test and master images
 BADDISK  Array         (1,3 good on CVAX)
 VERSION  String*40     OLD,NEW,OLDPSAP,NEWPSAP,LOCAL
                        (also private dirs; use | to separate)

DDT runs in four different modes: (TCODE)
   INIT = starts from the original master UV dataset, and
           (re)computes master files (does ALLDEST).
           The "mask" TMASK controls which steps execute.
           An output tape is required: each result must go to
           tape and be restored to disk before the next step.
   TEST = re-does the INIT computation but stores the new
           files under different names and compares against
           masters. The "mask" TMASK controls which steps
           execute. TEST steps use either the TEST run output
           of the previous step (TMODE=T) or the INIT run
           output (TMODE=M).  On TMODE=T, an output tape is
           required: each result must go to tape and be
           restored to disk before the next step.
   WRIT = write out all master files to   tape.
   READ = read  in  all master files from tape.
A binary mask controls which steps of the test execute:
   TMASK +1=>SORT,  +2=>UVMAP, +4=>APCLN,
         +8=>ASCAL, +16=>MXmap,  +32=>MXclean,
         +64=>VTESS
   For example, TMASK=54 would execute UVMAP,APCLN,MXmap,and
   MXclean.
Input data files for test steps are selected by the TMODE
   adverb which has two possible values: 'T' for those made
   in previous steps of the present test or 'M' for the
   master "correct" data sets input from tape.
The adverb DDTSIZE controls the size of the test performed:
   'S'    256x256 image, 2000 clean components, 8000 vis
   'M'    512x512 image, 5000 clean components, 15000 vis
   'L'    1024x1024 image, 15000 clean components, 75000 vis
----------------------------------------------------------------
Explanation of DDT:               (DCW, Feb/Mar85, EWG Nov85,
                                   CF Mar89)

DDT exercises the "Dirty Dozen" of AIPS tasks using data for
one of three sources.  The three data sets provide three sizes
of "real" AIPS reduction: 256x256 images with 8000 vis. and
2000 components, 512x512 with 13000 vis. and 5000 components,
and 1024x1024 with 77000 vis. and 15000 components.  The sources
for the "small" and "medium" problems were originally background
sources for an HI absorption experiment by Liszt and Greisen
while the "large" problem uses data supplied by Alan Bridle.

The DDT problem is intended to be used to certify the correct
execution of all AIPS systems. (The smallest is small enough to
run in less than one day on all machines which will ever run
AIPS.)  Its execution time is significant and easily measurable
on VAX+AP systems, and approximate timings can also be obtained
for machines 10-100x faster, like the Cray. In order to properly
test the performance of these machines which are much faster
than a VAX+AP, the larger data sets may be used or a still
larger problem may have to be developed and added to DDT.
================================================================
Steps to execute the DDT Test on a new computer:

1. Log in to AIPS under the user number which will own the data
   files and the SAVE/GET files associated with the test.  It is
   probably convenient to clear your AIPS message file before
   proceeding: PRNUM=-1; (PRTMSG possibly) ; CLRMSG

2. Now do the POPS command "RUN DDTLOAD". This initiates the
   compilation process, which can take as long as 15 minutes on
   a heavily loaded VAX-11/780. You may want to execute the POPS
   command "PRTMSG" afterwards to document the compilation of
   the test procedures.

3. The next step is to read the data files from the tape.  These
   are files 1 - 11 for the small problem, files 12 - 22 for the
   medium problem, and files 23 - 33 for the large problem.  The
   first one in each set of 11 is the original visibility data
   set of the DDT problem.  Subsequent files are the "master"
   versions of files computed from the original dataset.  The
   read process consists of the following steps:
      CLRMSG                       (prepare for tape read steps)
      GET DDTINIT
      Specify DDISK, MDISK, TDISK, and IOTAPE
      Also specify BADDISK as needed
      DDTSIZE='SMALL' or 'MEDIUM' or 'LARGE'  (desired problem)
      TCODE='READ'; TPUT DDT         (TPUT saves adverb values)
      MOUNT the tape
      RUN DDTEXEC
      DISMOUNT the tape
      PRTMSG                       (documents tape read process)
   The READ mode of DDT will rewind the tape and will skip over
   the data for the smaller problems (if 'SMALL' wasn't given).
   It will then execute UVLOD and IMLOD in the appropriate
   pattern to read the UV and map files to disk.  At the end of
   the process, it rewinds the tape and prints a directory of
   the disk files.

   NOTE: DDTEXEC will insist on having non-default values for
   IOTAPE, DDISK, MDISK, and TDISK.

4. Now it is time to actually perform the test.  The 'TEST' mode
   recomputes the various files.  All map results are compared
   to the master versions which were read from tape in the
   previous step. There are two ways to perform the
   calculations: with each step computed from the result of the
   previous test calculation in the sequence, or else from the
   result of the previous MASTER computation in the sequence.
   The first mode (TMODE='T') tests how well the system under
   test is able to reproduce a complex sequence of calculations
   beginning from a designated starting point.  That is, it
   measures the accumulation of error in the system.  In order
   to match everything, including the loss of accuracy from
   going to tape, this mode should use an out/in tape to write
   each result to tape and read it back before going on to the
   next step in the test.  The second mode (TMODE='M') tests
   the steps of the sequence separately so that errors created
   in each step can be isolated.  It also should use tape.  Both
   tests are important, and both should be performed to certify
   fully the scientific adequacy of an AIPS installation, but it
   is generally best to use TMODE='M' as the initial (and
   primary) test of all systems.  The steps are:

      CLRMSG                       (prepare for test results)
      GET DDTINIT
      TGET DDT
      TCODE='TEST'; TMODE='M'; TMASK=127
      DDTSIZE='SMALL' or 'MEDIUM' or 'LARGE'  (desired problem)
      TPUT DDT                     (saves the adverb values)
      RUN DDTEXEC
      PRTMSG                       (print out the test results)

   The message file from DDTEXEC will be more than 2000 lines
   long.  It is intended to be the major item documenting the
   test procedures and results.

   Making a second run with TMODE='T' is generally advisable to
   evaluate the accumulation of error. This process also checks
   the manipulations of visibilities, antenna files, and gain
   files; these processes are not really tested in the 'M' mode
   because AIPS currently has no comparison operators for files
   of these types. The 'T' mode tests them by their effect on
   subsequent map results.  This is particularly important for
   the output of the self-calibratiion program ASCAL.

   One can also rerun the tests with a different value for the
   VERSION adverb.  Generally this is most interesting in the
   comparison of the NEWPSAP library with the AP library.  One
   can also use this option to test experimental versions of
   specific tasks.

5. The 'WRIT' mode tests task FITTP, while 'READ' tests IMLOD
   and UVLOD.  The basic idea is to produce a tape, and then to
   read that tape on an AIPS system to verify that it yields
   correct results.  It follows that generating a new tape with
   the 'WRIT' mode and then executing 'READ' and 'TEST' on it
   will also effectively test the tape tasks.  Better yet, take
   the tape to another AIPS and run DDT on it!  This verifies
   the intercommunication of the systems.

   Note that DDT assumes that the tape consists of the 11 SMALL
   data files, followed by 11 MEDIUM data files, followed by 11
   LARGE data files.  You should first run DDT in WRIT mode for
   the SMALL problem.  Having done this, you may then (option-
   ally) run DDT in WRIT mode for the medium problem and, having
   done all this, you may then WRIT the LARGE problem.

   The steps are:

      CLRMSG
      GET DDTINIT
      TGET DDT
      DDTSIZE='SMALL' or 'MEDIUM' or 'LARGE'  (desired problem)
      TCODE='WRIT';TMODE='T'
      TPUT DDT
      RUN DDTEXEC
      PRTMSG                       (print out the test results)

6. Finally, one wants to know the accounting statistics from the
   test runs.  The accounting listing can be correlated with the
   message file listings in order to analyze the performance of
   the tested system.  DDT now runs task PRTAC to produce the
   desired information.  Note that PRTAC has exclusive access to
   the accounting file until it finishes, and so various tasks
   may complain while they are waiting to get access to the
   file.

   The listing produced by PRTAC will have one line for each
   execution of any task or program AIPS.  The columns of the
   listing give the taskname, POPS#, user#, start-time, real-
   time, CPU-time, and "IOCNT".  Asterisks mark any run that
   aborted.  After this basic sequential listing comes a listing
   by task name giving the totals accumulated by each task.  The
   listing is given in two sort orders.

================================================================
Regarding "cloning":

It is obvious that DDTLOAD and DDTEXEC are cloneable.  First, a
clone of DDT could process another dataset; this would be done
by adding additional allowed string values for DDTSIZE and
renaming the input UV data set DDDT?, where ? is the first
character in the new allowed string value.  The adverb values
needed for the new data set, are initialized in the DDTPROC
procedure which is SAVEd in DDTSAVE.  Second, the general
pattern of executing and testing the tasks in succession will
work for a wide variety of AIPS tasks and data types.

================================================================
Interpretation of Test Results

This section will be added later and will discuss issues of
numerical precision.  It will also address various issues
associated with benchmarking.  We need some beta-testing before
this section can be constructed.
