%-----------------------------------------------------------------------
%;  Copyright (C) 1995
%;  Associated Universities, Inc. Washington DC, USA.
%;
%;  This program is free software; you can redistribute it and/or
%;  modify it under the terms of the GNU General Public License as
%;  published by the Free Software Foundation; either version 2 of
%;  the License, or (at your option) any later version.
%;
%;  This program is distributed in the hope that it will be useful,
%;  but WITHOUT ANY WARRANTY; without even the implied warranty of
%;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%;  GNU General Public License for more details.
%;
%;  You should have received a copy of the GNU General Public
%;  License along with this program; if not, write to the Free
%;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
%;  MA 02139, USA.
%;
%;  Correspondence concerning AIPS should be addressed as follows:
%;          Internet email: aipsmail@nrao.edu.
%;          Postal address: AIPS Project Office
%;                          National Radio Astronomy Observatory
%;                          520 Edgemont Road
%;                          Charlottesville, VA 22903-2475 USA
%-----------------------------------------------------------------------
% Summary of latest DDT results and forms
% last edited by  Glen Langston on 92 September 03
% last edited by  Ernie Allen   on 92 September 03
\documentstyle{article}
% Semi-general AIPS memo format
\newcommand{\lastedit}{{\it 92 September 3}}
\newcommand{\memodate}{92 September 3}
\newcommand{\memono}{{77}}
\newcommand{\subject}{{Summary of DDT Accuracy Results}}
\newcommand{\byline}{{Ernest Allen and Glen Langston}}
%
\parskip 3mm             \parindent 0in
\textwidth 6.5in         \linewidth 6.5in
\oddsidemargin -.25in    \evensidemargin -.25in
\topmargin -.5in         \headheight 0in
\headsep 0.25in          \textheight 9.5in
\headheight 0.25in       \marginparsep 0in
\pretolerance=10000

%general definitions
\newcommand{\beq}{\begin{equation}}       % start equation
\newcommand{\eeq}{\end{equation}}
\newcommand{\beddes}{\begin{description} \leftmargin 2cm} % description list
\newcommand{\eeddes}{\end{description}} % description list
\newcommand{\backs}{$\backslash$}
\newcommand{\myitem}[1]{\item{\makebox[3cm][l]{\bf {#1}}}}
\newcommand{\mybitem}[1]{\item{\makebox[0.65cm][l]{\sc {#1}}}}
\newcommand{\AIPS}{{$\cal AIPS$}}
\newcommand{\uv}{{\it uv}} \newcommand{\etal}{{\it et al.}}
\newcommand{\AIPONE}{{$\cal AIPS$}}    \newcommand{\AIPTOO}{{$\cal AIPS$++}}
\newcommand{\Ape}{{$\cal APE$}}
\newcommand{\XAS}{{$\cal XAS$}}
\newcommand{\CPP}{{$\cal C++$}}
\newcommand{\APEIN}[1]{{\normalsize \sc {#1}}}
\newcommand{\CL}{{\normalsize \sc CL}~}
\newcommand{\normalstyle}{\baselineskip 7mm \parskip 1mm \large}
\newcommand{\tablestyle}{\baselineskip 4mm \parskip 0mm \normalsize }
\newcommand{\ftp}{{\tt ftp}}
\newcommand{\bits}{{\tt bits}}
\newcommand{\baboon}{{\tt baboon}}
\newcommand{\uvdata}{{{\it uv}-data}}

\begin{document}
\pagestyle{myheadings}
\newcommand{\HEADING}{{\it \AIPS\ Memo} \memono \hfill \subject
\hfill Page~~}
\markboth{\HEADING}{\HEADING}
\vskip -1.5cm

\listparindent 0cm
\labelsep 0cm
\setcounter{page}{1}

\begin{center}
\Large
\newcounter{cms}
\setlength{\unitlength}{1mm}
\begin{picture}(50,10)                % make a memo number box
\thicklines \put(0,0){\framebox(50,10){\AIPS\ Memo \memono}}
\end{picture}

\subject

\large \byline

\memodate
\end{center}
\normalstyle
\begin{quote}
The current results of the \AIPS\ DDTs (Dirty Dozen Tests) are presented.
The DDTs consist of a set of inputs and outputs to a few of the most
commonly used \AIPS\ tasks.
The purpose of the DDTs is to determine whether the \AIPS\ software
output is changing with time.
The DDTs also measure the relative performance of the CPUs running
\AIPS.
The changes to the \AIPS\ task outputs have two types of causes,
{\it 1)} improvements and {\it 2)} bugs.
The latter should be detected by these tests and the bugs fixed.

Three types of tests have been developed,
{\it 1)} continuum imaging (DDTs),
{\it 2)} continuum \uvdata\ calibration (VLAC), and
{\it 3)} spectral line \uvdata\ calibration (VLAL).
A DDT for VLBI software is needed, but has not yet been implemented.

A new continuum calibration (VLAC) master tape has been released,
to accommodate \AIPS\ calibration changes.
\end{quote}

\section{INTRODUCTION}
In order to assure the continuity of the \AIPS\ tasks results,
sets of test procedures have been developed.
These tests are important for tracking changes in the software
and detection of bugs.  The results are presented in three sections,
continuum imaging, continuum calibration and spectral line
calibration.
Current results of these tests are described in turn below.

The Small, Medium and Large DDT results are presented in Appendix A,
the VLAC results in Appendix B, and the VLAL results in Appendix C.
The forms useful for reporting the DDT results to the \AIPS\ group
are presented in Appendix D.
A listing of \AIPS\ messages produced by the VLAC test are shown
in Appendix E and the VLAL test messages are shown in Appendix F.

\section{Continuum Imaging, DDT}
The Continuum imaging DDTs consist of three size tests, Small, Medium
and Large.  The small test creates 256x256 pixel images, medium
512x512 and large 1024x1024.
The method for running these tests has been described in
\AIPS\ memo 73.

The latest results are presented in the tables in Appendix A.
These results compare the current operation of the \AIPS\ tasks
with the results of these tasks when the
last master tape was created.
The current master tape was created in October 1989.
Three different types of CPUs are listed in the results tables.
Table 1 lists the names of the CPUs tested.


\begin{table}\begin{center}
\begin{tabular}{ll}
\hline \hline
Nrao1  &  A Convex C1 computer \\
Baboon &  A Sun Sparc2 Workstation  \\
Lemur  & An IBM 6000 (Power Station), model 530  \\
\hline \hline
\end{tabular}
\end{center}
\caption{NRAO CPU's tested}
\end{table}

Two types of tests are executed, imaging and \uvdata\ tests.
The imaging test results are expressed as the number of \bits\
of agreement between the master and the test images.
In modern CPUs, the floating point numbers are
represented with 32 \bits, 8 \bits\ of exponent and 24 \bits\
of mantissa.
In the tables, two \bits\ numbers are given for each image compared,
{\it a)} the number of \bits\ of agreement in the pixel with the
maximum difference, and
{\it b)} the average number of \bits\ of agreement between the master
and test images.

The \uvdata\ tests compare differences between the master and test
\uvdata\ using the \AIPS\ task UVDIF.
This task looks for three kinds of \uvdata\ differences,
{\it 1)} differences in flux density of the visibilities,
{\it 2)} differences in {\it u,v and w}, and
{\it 3)} and other differences, when there does not appear to
be any relation between the master and test visibilities.
The third type of difference often occurs when the \uvdata\ end
up with a different sort order.  This happens frequently because
several of the visibilities in the master dataset have nearly identical
{\it u} values.  The test resorts are reported as list of the three
numbers of differences, ie. the 0,0,2 in column 6 of the first
DDT results table.

The typical accuracy now expected for the DDT tests is 10 to 16 \bits.
The \AIPS\ accuracy is calculated as the number of \bits\ that
match exactly between the master and test images.
The number of \bits\ of accuracy is a function of the internal
floating point algorithm of the CPU, the improvements
of the software and errors introduced in inadvertently.
Since most CPUs now conform to IEEE floating point standards,
the first of these should have little affect.
The second of these confuses the test results and requires
periodic creation of new master images.
The inadvertent introduction of errors is detected by strangely
low numbers of \bits.

The algorithm for the \AIPS\ task MX has improved for large
images to the point where the reference image has only 9 \bits\
matching when compared to current results.

{\bf Note that because the computers were often very heavily loaded during
the DDT tests, the relative CPU times are NOT useful in comparing
the performance of the different types of computers.}

The master DDT tape will probably be updated sometime in the
near future. The current Small and Medium DDT master data are
available via {\it anonymous} \ftp. Below is the README.DDT file
in the \baboon\ \ftp\ area.

\vskip 1in

\input{README.DDT}

\clearpage
\section{Continuum Calibration, VLAC}
The VLA continuum calibration test is designed to check the tables
produced by the \AIPS\ calibration process for a continuum, two IF
observation.
There is only one size of calibration test (unlike the DDTs which
come in three sizes, small, medium and large).

The VLAC test was created by Bill Cotton in 1990 and the current master
data and images were previously updated on December 6, 1990.
On 1992 June 16, a new master VLAC tape was created using the 15APR92
release of \AIPS.  This tape is now available for distribution and
may be requested with the latest releases of \AIPS.

The VLA continuum calibration test executes in a manner
similar to the DDT tests.
The test \AIPS\ procedures are created by running the RUNFIL VLACLOAD.
The test master images are loaded from tape into \AIPS\ using the
RUNFIL VLACEXEC.
The VLAC test parameters are specified by specifying the adverbs
in VLAC (ie. within \AIPS\ typing \APEIN{INPUT VLAC}).
After the adverbs are correctly set, the adverb values must be
saved in \AIPS\ by typing \APEIN{TPUT VLAC}.
Then the VLAC test is executed by running the RUNFIL VLACEXEC.

The VLAC tasks tested are listed in table 1.
\vskip .25in
\begin{table}[h]\begin{center}
\begin{tabular}{cl}
VLAC Task & Purpose \\
\hline
\hline
UVCOP & Copy a selection of \uvdata\ \\
UVFLG & Flag bad \uvdata \\
SETJY & Set the flux density in the source table. Defines the
observation gain \\
CALIB & Assuming point source observations, the antenna complex gains
are estimated \\
GETJY & Measure the calibration source flux densities \\
CLCAL & Interpolate antenna gain measurements into CL table for all
observations \\
PCAL  & Calibrate the intrinsic antenna polarization \\
CLCOR & Correct the CL table for intrinsic antenna polarization \\
SPLIT & Apply calibration and create single source data sets \\
HORUS & Fourier Transform the calibrated \uvdata.
Creates Beam I and Q images \\
\hline
\hline
\end{tabular}
\end{center}
\caption{Continuum Calibration Tasks tested by VLAC}
\end{table}

The VLAC results are presented in Appendix B, and the
entire \AIPS\ messages for the VLAC test are listed in Appendix E.
The execution times for the VLAC test are not recorded.
The DDTs must be used to check the execution times of
the CPUs.

\clearpage
\section{Spectral Line Calibration, VLAL}
The Spectral Line Calibration process is tested by VLAL.
The test is similar to the VLAC test with the addition of several
tasks for the relative calibration the spectral channels (the band
pass calibration).
The VLA spectral Line calibration (VLAL) tests were created by Doug
Wood with modifications and improvements by Phil Diamond and Dave
Adler.

The tasks tested by VLAL are listed in the table 2.
\begin{table}\begin{center}
\begin{tabular}[h]{cl}
VLAL Task & Purpose \\
\hline
\hline
UVCOP & Copy a selection of \uvdata\ \\
AVSPC & Average spectral line channels to create a new \uvdata set \\
UVFLG & Flag erroneous \uvdata \\
SETJY & Set the flux density in the source table. Defines the
observation gain \\
CALIB & Assuming point source observations, the antenna complex gains
are estimated \\
GETJY & Measure the calibration source flux densities \\
CLCAL & Interpolate antenna gain measurements into CL table for all
observations \\
BPASS & Band pass calibration of the spectral line channels \\
CLCOR & Correct the CL table for intrinsic antenna polarization \\
SPLIT & Apply calibration and create single source data sets \\
HORUS & Fourier Transform the calibrated \uvdata.
Creates a cube of images\\
\hline
\hline
\end{tabular}

\end{center}
\caption{Spectral Line Calibration Tasks tested by VLAL}
\end{table}

The current Spectral Line DDT master tape was created on 1991 November
4, using the 15APR92 version of \AIPS.
The Spectral Line DDT is still being refined and the master
data and images will be re-released shortly.
The Spectral Line DDT results are presented in Appendix C.
A listing of the \AIPS\ messages produced by the VLAL test
are shown in Appendix F.
\section{Results Forms}
The DDT results are useful for comparing the IEEE floating point
compliance of CPUs running \AIPS\ as well as comparing
the speed of the CPUs.
\AIPS\ System managers running the DDTs on new types of CPUs
are requested to report their results to the \AIPS\ group.
In Appendix D, the DDT reports forms are presented.

\clearpage
\baselineskip .3cm
\renewcommand{\HEADING}{{\it \AIPS\ Memo} \memono \hfill Appendix A
\hfill Page~~}
\markboth{\HEADING}{\HEADING}
\input{DDTUDDT.TEX}    % DDT Results
\clearpage
\renewcommand{\HEADING}{{\it \AIPS\ Memo} \memono \hfill Appendix B
\hfill Page~~}
\markboth{\HEADING}{\HEADING}
\input{DDTUVLAC.TEX}   % VLAC Results

\clearpage

\renewcommand{\HEADING}{{\it \AIPS\ Memo} \memono \hfill Appendix C
\hfill Page~~}
\markboth{\HEADING}{\HEADING}
\input{DDTUVLAL.TEX}   % VLAL Results

\end{document}



