\documentstyle [twoside]{article}
%-----------------------------------------------------------------------
%;  Copyright (C) 2007
%;  Associated Universities, Inc. Washington DC, USA.
%;
%;  This program is free software; you can redistribute it and/or
%;  modify it under the terms of the GNU General Public License as
%;  published by the Free Software Foundation; either version 2 of
%;  the License, or (at your option) any later version.
%;
%;  This program is distributed in the hope that it will be useful,
%;  but WITHOUT ANY WARRANTY; without even the implied warranty of
%;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%;  GNU General Public License for more details.
%;
%;  You should have received a copy of the GNU General Public
%;  License along with this program; if not, write to the Free
%;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
%;  MA 02139, USA.
%;
%;  Correspondence concerning AIPS should be addressed as follows:
%;         Internet email: aipsmail@nrao.edu.
%;         Postal address: AIPS Project Office
%;                         National Radio Astronomy Observatory
%;                         520 Edgemont Road
%;                         Charlottesville, VA 22903-2475 USA
%-----------------------------------------------------------------------
\newcommand{\DoMemo}{T}
\newcommand{\doFig}{T}
\newcommand{\showwork}{T}
\newcommand{\workyes}{F}
%
\newcommand{\memnum}{112}
\newcommand{\whatmem}{\AIPS\ Memo \memnum}
\newcommand{\AIPS}{{$\cal AIPS\/$}}
\newcommand{\VPOPS}{{$\cal VPOPS\/$}}
\newcommand{\RANCID}{{$\cal RANCID\/$}}
%
\newcommand{\memtit}{Capabilities of the VLA pipeline in AIPS}
%
\newcommand{\POPS}{{$\cal POPS\/$}}
\newcommand{\Cookbook}{${{\cal C}ook{\cal B}ook\/}$}
\newcommand{\TEX}{\hbox{T\hskip-.1667em\lower0.424ex\hbox{E}\hskip-.125em X}}
\newcommand{\AMark}{AIPSMark}
\newcommand{\AMarks}{AIPSMarks}
\newcommand{\figyes}{T}
\newcommand{\uv}{{\it uv}}
\newcommand{\eg}{{\it e.g.},}
\newcommand{\ie}{{\it i.e.},}
\newcommand{\daemon}{d\ae mon}
\newcommand{\Aipsletter}{{${\cal AIPSL}etter\/$}}
\newcommand{\ust}{{\rm st}}
\newcommand{\uth}{{\rm th}}
\newcommand{\und}{{\rm nd}}
\newcommand{\urd}{{\rm rd}}
%
\newcommand{\boxit}[3]{\vbox{\hrule height#1\hbox{\vrule width#1\kern#2%
\vbox{\kern#2{#3}\kern#2}\kern#2\vrule width#1}\hrule height#1}}
%
\title{
   \vskip -35pt
   \if T\DoMemo
      \fbox{{\large\whatmem}} \\
      \fi
   \vskip 28pt
   \memtit\\}
\author{Lorant O. Sjouwerman}
%
\parskip 4mm
\linewidth 6.5in
\textwidth 6.5in                     % text width excluding margin
\textheight 8.81 in
\marginparsep 0in
\oddsidemargin .25in                 % EWG from -.25
\evensidemargin -.25in
\topmargin 0.3in
\headsep 0.25in
\headheight 0.25in
\parindent 0in
\newcommand{\normalstyle}{\baselineskip 4mm \parskip 2mm \normalsize}
\newcommand{\tablestyle}{\baselineskip 2mm \parskip 1mm \small }
%
%
\begin{document}

\pagestyle{myheadings}
\thispagestyle{empty}

\if T\DoMemo
   \newcommand{\Rheading}{\whatmem \hfill \memtit \hfill Page~~}
   \newcommand{\Lheading}{~~Page \hfill \memtit \hfill \whatmem}
\else
   \newcommand{\Rheading}{L.~O.~Sjouwerman\hfill \memtit \hfill Page~~}
   \newcommand{\Lheading}{~~Page \hfill \memtit \hfill L.~O.~Sjouwerman}
   \fi
\markboth{\Lheading}{\Rheading}
%
%

\vskip -.5cm
\pretolerance 10000
\listparindent 0cm
\labelsep 0cm
%
%

\vskip -30pt
\maketitle
%\vskip -30pt
\normalstyle

%\begin{abstract}
%
%\end{abstract}
%
%\section{Introduction}
%
%\section{Acknowledgements}
%
%\end{document}
%\documentclass[11pt,twoside]{article}
%\begin{document}
%\title{\bfseries Capabilities of the VLA pipeline in AIPS}
%\author{Lor\'ant O. Sjouwerman\\ \itshape National Radio Astronomy Observatory}
%\maketitle
\begin{abstract}
\noindent This document describes the VLA pipeline procedure. The
procedure runs in AIPS, though a system has been set up to process VLA
data with this pipeline from a UNIX command line. The latter and an
analysis of a pilot project on 1999/2000 VLA B-array 5 and 8 GHz
continuum data run using this pipeline are
described in another document; most of this document is to explain
about the capabilities of the pipeline in combination with AIPS input
parameters, choices and limitations, as invoked from inside
AIPS. Starting conditions and suggestions for success are given. This
document covers the status of the VLA pipeline as per mid-November
2006.
\end{abstract}
\vfill
\hrule
\tableofcontents
\bigskip\hrule
%\clearpage
\section{Introduction to the early versions}
An early version (the ``first'' version) of the VLA pipeline was
simple and created for private use. It was intended to perform
calibration and imaging of high-frequency multi-IF spectral line VLA
survey data in any array configuration using NRAO's classic AIPS
package. It assumed that all bad data was removed and that one of the
standard VLA flux calibrators was included in the observations. A
cleaned-up version of this was released to the general public as an
AIPS run-file procedure named ``VLARUN'' in 2003, and used for
demonstrations at the Santa Fe ``xraydio'' meeting in February
2004. This pipeline was a very useful tool to examine the $u,v$-data and
create images within minutes of acquiring the raw VLA data, point out
the trouble spots (i.e.\ bad data), and obtain first order self-cal
source models.

Since then the VLA pipeline has been further developed for automated
pipelining, with emphasis on X- and C-band continuum in
B-array. Enhanced features such as wide-field imaging, automatic
detection of source models, correcting for fast-switching source
names, etc, are now also
included for processing of any frequency over 1 GHz. This is the latest procedure known as VLARUN currently in
AIPS (since March 2006). Further developments, also because of encountering
different types of observations and user specified observing setups
(and features..) are currently done in a private version until they
are robust enough to be put back. This allows us to fiddle with the
code, implement suggestions and, e.g., test out solar system mode
without introducing bugs to VLARUN users that run the AIPS midnight
job.

Current imperfections include non-robust automated self-cal and most
added features have remained untested in spectral-line mode. No
attempt is made to calibrate and image polarization data or mosaics.

VLARUN, the AIPS VLA pipeline procedure, was announced to the public
at several occasions over the last few years, including the
aforementioned ``xraydio'' meeting, the AIPS newsletters, and, e.g,
the January 2005 AAS meeting in San Diego. Furthermore, VLARUN was
used for several professional publications and explicitly mentioned in
those articles.

\section{The VLA pipeline setup}
Although a description and manual of the fully automated VLA pipeline
is the subject of another document (the ``user manual'' by Jared
Crossley), a brief insight of the automated pipeline setup helps to
understand the capabilities of the pipeline as described in the next
section below.

The use of the VLA pipeline, VLARUN or our modified private version,
can be divided up in two different modes. First, it can be run by a
competent user that knows the data or has a drive to understand its
details, and is capable of making decisions regarding, e.g., bad data,
solution interval, image size, confusing sources, etc. Details about
capabilities and input parameters are commented on below. Or, if a
user is not familiar with the data, it can be run in a {\bfseries
best-effort} non-radio-expert mode, i.e., the pipeline can run without
astronomical knowledge with the use of default ``dummy'' input
parameters also outlined below. These two modes assume the user has
started up AIPS and loaded the data into AIPS. Another mode, which is
the automated pipeline mode, uses c-shell scripts, Perl programs and
AIPS run files and default inputs analogous to the second ``dummy'' case.

All automated pipeline processing is done through command line
instructions that interact with the NRAO archive, cron jobs and AIPS
procedures. The operator of the automated pipeline only selects a data
set to be processed and determines whether a run was successful. Other
cron jobs and scripts automatically fill and index the image archive
web pages (currently at http://www.aoc.nrao.edu/$\sim$vlbacald). The
operation is in three steps: \newcounter{opns}
\begin{list}{\arabic{opns}: }{\usecounter{opns}}
\setlength{\baselineskip}{-1pt}\setlength{\parskip}{\baselineskip}
\item  query the archive;
\item prepare a data set to be downloaded, checked, loaded into AIPS,
and prepare a script to run the pipeline on it; and
\item after the pipeline has run validate the results and release the
images and calibrated $u,v$-data to the NRAO and VLA image archive.
\end{list}

In order to run the automated pipeline smoothly, some preparations are
done before the actual AIPS pipeline procedure is started. For
example, all data sets are converted to J2000 coordinates, if needed,
source and frequency checks are done and, if passed, to avoid confusion
with another pipeline process each data set is processed under a
separate AIPS user number. There are three different AIPS run
procedure scripts: \newcounter{auto}
\begin{list}{\alph{auto}: }{\usecounter{auto}}
\setlength{\baselineskip}{-1pt}\setlength{\parskip}{\baselineskip}
\item prepare and check the set for pipelining (step 2 above);
\item if approved allow VLARUN to process it with standard parameters
(without operator intervention after step 2 above); and
\item post-collect the images and data into a format standardized for
release in the NRAO and VLA image archive (step 3 above).
\end{list}

\section{The VLA pipeline in AIPS}
This section describes the capabilities of the VLA pipeline procedure
script VLARUN. Essentially it summarizes the VLARUN {\tt HELP} file in
AIPS....

VLARUN can be used to calibrate $u,v$-data and make investigative plots
to refine the calibration, or it can be used to calibrate and
automatically image the sources in one go. This switch is set by the
user, specifying the parameter {\tt DOIMAGES}. The request for images to
be made by setting {\tt DOIMAGES$>$0} activates an extra set of
parameters for the imaging (see below), and setting {\tt SLFCAL} to a
non-zero value activates interactive self-calibration. The only other
interactive setting is {\tt NOPAUSE} which allows for a pause in the
calibration process to check the flux densities and their associated
errors of the secondary (phase) calibrators. It is very useful to look
at these {\tt ``GETJY''} values to spot possible calibration problems
(such as residual bad data). None of the other settings below modify
the interactiveness of the pipeline procedure, although a request to
make diagnostic plots ({\tt AUTOPLOT$\ge$0}) may slow down the pipeline
considerably.

Note that many of the parameters are used in a non-intuitive way
(i.e., not as usual in AIPS). Here the non-obvious parameters are
described one-by-one, including their ``dummy'' value.

\subsection{VLARUN \underline{calibration}\ input parameters}
{\tt CATNUM} {\bfseries or} {\tt INNAME} (etc) specify the file to
calibrate. For the automated pipeline this is always {\tt WORKDISK=1},
{\tt CATNUM=1}, and {\tt INNAME='~'}, {\tt INCLASS='~'} and {\tt
INSEQ=0}.

For spectral line data, {\tt CATNUM} or {\tt INCLASS} must point at
the spectrally averaged {\tt 'CH 0'} file and a spectral line file
{\tt 'LINE'} must exist with the same {\tt INNAME} and {\tt INSEQ},
and on the same {\tt WORKDISK}.

{\tt FASTSW=1} checks the source table for sources that are within 3
mas of each other, declares them the same source, and reassigns the
shortest name found for them. This typically is needed when
fast-switching is used at the VLA. The automated pipeline does this by
default in an earlier stage (in the preparation stage 2 above), so for
the automated pipeline {\tt FASTSW=$-$1}.

If a flagging table is present it is assumed that removal of bad data
has been done (either by hand in an earlier stage or in a previous
run). Otherwise, setting {\tt AUTOFLAG$\ge$0} activates some automated flagging of
bad data with {\tt FLAGR}. If positive, {\tt QUACK} is used to clip
the beginning of each scan. However, for frequencies over 18 GHz this
option is disabled to avoid clipping the majority of short scans - the
user is warned to perform this outside of the pipeline if desired. For
the automated pipeline {\tt AUTOFLAG=2}, i.e., use all flagging options
but disable {\tt QUACK} for high frequencies.  See also the section on
improvements.

{\tt PHAINT} is the parameter for the time interval used in {\tt
CALIB}, the core calibration task in AIPS. Typically {\tt PHAINT} is
short (a couple of minutes or a small fraction thereof for
high-frequency observations) to obtain phase solutions. For amplitude
solutions one would prefer to use a longer interval, which is
implemented by specifying {\tt AMPINT}. VLARUN first separately
calibrates phases, then also the amplitudes.  The automated pipeline
uses {\tt PHAINT=1} and {\tt AMPINT=5} minutes.

One can specify a reference antenna with {\tt REFANT}, but AIPS will
make an educated choice if left zero as is done in the automated
pipeline.

The {\tt DOMODEL} switch allows {\tt CALIB} to use AIPS provided
source models to be used for calibration. This is usually good
practice, though at this moment not all standard VLA flux calibration
source models are available. Earlier versions of VLARUN would crash if
a model was requested but did not exist; currently a point source (as
was practice before the availability of these source models) is used
in such cases. {\tt DOMODEL} is set to use the models available in the
automated pipeline.

If for some reason a standard flux calibrator is missing in the
observation, the user can designate an alternate source observed in
the observing run as point-like flux calibrator by setting {\tt
AMPCAL} to the source name, {\tt FLUX} to its presumed flux density
and when necessary {\tt UVRANGE} to the range of {\it u,v}-values for
which this flux density is defined. {\tt AMPCAL} and {\tt FLUX} are
only used if no models are used ({\tt DOMODEL$\le$0}), whereas {\tt
UVRANGE} may also apply to the default flux calibration models (though
setting {\tt UVRANGE} for a model is not recommended).

The {\tt PHACAL} parameter captures the names of all phase
calibrators to be used, up to 20. If more are needed or if the
explicit calibrator names are unknown, as is the case in the automated
pipeline, set {\tt PHACAL='*','~'} and it will presume point sources
for everything with a ``calcode'' in the source list. The latter
usually works surprisingly well (for VLA data). Sources specified here
are used as secondary amplitude (gain) calibrators and their flux
densities will be determined by {\tt GETJY}.

{\tt BNDCAL} is only used in spectral line mode and captures up to
five explicit names of bandpass calibrators to be used (simultaneously).

In interactive mode it is very useful to set {\tt NOPAUSE=$-$1}. This
will pause the pipeline just after {\tt GETJY}, the task that derives
the flux densities of the secondary phase calibrators. Next to the
flux densities {\tt GETJY} reports the errors in determining the flux
densities, which together with the flux densities themselves may point
out the success of, or residual bad data in, the calibration. In the
automated pipeline this switch is turned off; {\tt GETJY} flux
densities can be examined in the automated pipeline log file.

The parameter {\tt AUTOPLOT} is used to determine the amount of
diagnostic plots. Setting this parameter however may slow the pipeline
down considerably. Set {\tt AUTOPLOT=$-$1} to skip diagnostic plots. The
automated pipeline uses {\tt AUTOPLOT=0} to make {\it u,v}-plots only for
the calibrated single source files, i.e, the {\it u,v}-coverage, the
visibility Real versus Imaginary and visibility amplitude versus {\it
u,v}-distance. These {\it u,v}-plots are only made in the imaging
stage (see below) when calibrated single source files are
available. Positive settings will create plots during the calibration
stage and include plots of phase and amplitude calibration solutions
of the multi-source file and bandpass calibration solution in the case
of a spectral line observation (see {\tt HELP}).

Set {\tt DOIMAGES$\le$0} to skip imaging, otherwise study the next two
sections.

\subsection{VLARUN \underline{imaging}\ input parameters}
These parameters are only of interest if imaging is requested (i.e.,
{\tt DOIMAGES$>$0}).

The {\tt ARRYSIZE} parameter fixes the resolution by taking the value
as the maximum baseline length in kilometers. Alternatively, if zero
it will figure out the maximum unprojected baseline from the antenna
file, or, if set negative it will leave it up to the task {\tt SETFC}
to determine the image pixel size. The latter, using SETFC to
determine the image pixel size, is done in the automated
pipeline.

The first argument of {\tt IMSIZE} specifies the output image size
with a minimum of 128 (square). For calibrators this defaults to 256
and can only be disabled when imaging the full primary beams (as
determined by {\tt SETFC}). To do this set {\tt IMSIZE} negative, or
large negative to include full primary beam imaging for the
calibrators in addition to the targets. The automated pipeline images
the full primary beam for all sources.

{\tt NITER} sets the maximum number of clean components to be
used. Note that zero means zero, i.e., make a non-cleaned dirty
image. Again, for calibrators this defaults to 500 (if {\tt ALLIMG$>$0}
below), unless the full primary beam is imaged for the calibrators too
({\tt IMSIZE} large negative). If set negative, as in the automated
pipeline, {\tt NITER} is set to a huge number effectively allowing
{\tt CUTOFF} (see next) to take over the stopping criterion.

If the image RMS noise reaches {\tt CUTOFF} stop cleaning. This
parameter heavily interacts with {\tt NITER}. If {\tt CUTOFF} is
negative, stop cleaning after the first negative clean component (or
{\tt NITER} if it is reached first). If both {\tt NITER} and {\tt
CUTOFF} are set negative, as is the case for the automated pipeline,
the pipeline tries to make a conservative estimate for the RMS noise
in the map and cleans to a conservative level above this estimated RMS
noise level (and beyond the first negative), regardless whether this is a target source or a strong
calibrator.

{\tt ALLIMG} determines whether to treat the calibrators the same as
the targets or not. Normally some information on the calibrators is
desirable to obtain some quality assessment of the data. This can be
an image (or total flux density, the source structure and position),
though deep cleaning is not necessary. Set {\tt
ALLIMG} positive to include images of the continuum calibrators, larger
than one for the spectral line version too and maybe also pay
attention to {\tt NITER} and {\tt CUTOFF}. This mode, to make no
distinction between calibrators and targets, is used in the
automated pipeline.

\subsection{VLARUN \underline{self-cal}\ input parameters}
This parameter is only relevant for radio continuum data of the target
sources if imaging is requested (i.e., {\tt CATNUM} {\bf or} {\tt
INCLASS} point to an AIPS catalog entry with a class other than {\tt
'CH 0'/'LINE'}, and {\tt DOIMAGES$>$0}).

The automated pipeline sets this parameter to zero, which at the
moment is probably the best value. {\tt SLFCAL} was included in an
earlier version of the VLARUN code to attempt self-cal for strong
point-like sources, allowing the task {\tt SCIMG} to do the imaging
instead of {\tt IMAGR}. However, since the focus on the pipeline
project and the many changes to the code, especially for continuum, it
is unclear how everything has affected this option. No testing has
been performed since the original implementation, so currently values
other than zero are highly discouraged. The original implementation
uses the absolute value of {\tt SLFCAL} as the number of cycles ({\tt
NMAPS} in {\tt SCIMG}) and the sign determines the level of
interactive (TV) display (see {\tt HELP}).

We hope to have an implementation for automated self-cal in the near
future, though this may not use an adjusted or improved version of the
current ({\tt SCIMG}) implementation.

\subsection{VLARUN \underline{pipeline}\ parameters}
If unknown or in doubt what to set for one or
more of the parameters, try the automated pipeline ``dummy''
inputs. The results of using these inputs in the pilot automated
pipeline imaging are described in a different document. The ``dummy''
inputs are below. Note that the ones with a $\dagger$ are different
from the pipeline (as shown between brackets), but strongly
recommended for interactive use. Also check your situation for the ones marked with
$\ast$, as these are automatically imposed by the automated pipeline
system:
%\newpage

{\tablestyle
\begin{table*}[h]
\begin{tabular}{lc|lc|lc}
$\phantom{0000}${\tt BADDISK} = 0   & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt AMPINT} = 5         &  & $\phantom{0000}${\tt AUTOPLOT} = 2 (0) & $\dagger$$\phantom{0000}$\\
$\phantom{0000}${\tt WORKDISK} = 1  & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt REFANT} = 0         &  & $\phantom{0000}${\tt DOIMAGES} = 1     & \\
$\phantom{0000}${\tt CATNUM} = 1    & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt DOMODEL} = 1        &  & $\phantom{0000}${\tt ARRYSIZE} = $-$1  & \\
$\phantom{0000}${\tt INNAME} = '\ ' & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt AMPCAL} = '\ '      &  & $\phantom{0000}${\tt IMSIZE} = $-$1    & \\
$\phantom{0000}${\tt INCLASS} = '\ '& $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt FLUX} = 0           &  & $\phantom{0000}${\tt NITER} = $-$1     & \\
$\phantom{0000}${\tt INSEQ} = 0     & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt UVRANGE} = 0, 0     &  & $\phantom{0000}${\tt CUTOFF} = $-$1    & \\
$\phantom{0000}${\tt FASTSW} = $-$1 & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt PHACAL} = '*', '\ ' &  & $\phantom{0000}${\tt ALLIMG} = 1       & \\
$\phantom{0000}${\tt AUTOFLAG} = 2  & $\ast$$\phantom{0000}$ & $\phantom{0000}${\tt BNDCAL} = '\ '      &  & $\phantom{0000}${\tt SLFCAL} = 0       & \\
$\phantom{0000}${\tt PHAINT} = 1    &                           & $\phantom{0000}${\tt NOPAUSE} = $-1$ (1) & $\dagger$$\phantom{0000}$ &            & \\

\end{tabular}
\end{table*}

}

\section{Running the VLA pipeline}
\subsection{Starting conditions}
In order for the pipeline to run smoothly it is important to pay
attention not only to the input parameters, but also to the input $u,v$-file
and its structure. For the automated pipeline, many of possible
problems with the input $u,v$-file are dealt with in the preparation
stage (2, or a), though it cannot handle every case. A simple scan
listing using {\tt LISTR} after loading the data into AIPS will
indicate possible problems. Another problem saver is to use a flagging
program such as {\tt TVFLG} or {\tt WIPER} to manually take care of
bad data ahead of time as the automated flagging is not yet
perfect. Also, currently all data at frequencies below 1 GHz (L-band)
are disregarded by the pipeline.

Possible problems that can be spotted in the scan listing can be
related to the source list or the observing setup itself. The pipeline
cannot handle multiple frequency-ID's and calibration of multiple
subarrays is untested. For each of these, split the data in single
frequency files ({\tt FILLM} does a lot of this already), and either
reset all frequency-ID's to one (with {\tt DFQID}) or split each
frequency-ID (and subarray) in its own separate file. For spectral
line data do this in the same way on both the {\tt 'CH 0'} and {\tt
'LINE'} data.

In the source list, look for a standard flux density calibrator (e.g.,
3C286 which is also known as 1331+305, etc.) and whether it has enough
visibilities; typically this should be a few integration times
multiplied by the number of baselines. If data on all standard flux
density calibrators are missing, find another source with enough
visibilities for which the flux density information can be figured out
- use this in {\tt AMPCAL} and {\tt FLUX}. Furthermore, verify
whether all the phase calibrators have calibrator codes - this is
usually the problem with phased-array data. The latter can be
processed fine, though the setting {\tt PHACAL='*','~'} cannot
be used. Check that the calibrators have enough visibilities, and if
possible, look up their approximate flux densities to compare with the
{\tt GETJY} values. Sometimes blocks of data may be missing due to
weather or hardware problems. Be sure to be aware of any of such
problems.

Bad data is best removed as soon as spotted to prevent the pipeline from
running when bad results are to be expected. However, bad data can also
be removed after calibration as it may be easier to recognize. If
after the pipeline ran, it turns out that more data needs to be
flagged, simply add the flags to the highest flagging table of the
original input $u,v$-data set, delete the files that were created by
the pipeline, and restart the pipeline. If a flagging table is
present, automated flagging is disabled to retain the (manual)
flagging. To enable the automated flagging either delete the flagging
information (but this is probably not the intention), or apply the
flagging by making a copy of the data with the flagged data
removed. Note also that for high frequencies (i.e., over 18 GHz), the
automatic {\tt QUACK} flagging is disabled. If needed, run {\tt QUACK}
manually and if other automated flagging is requested, make a copy of
the file with flagging applied.

\subsection{Suggested use}
The automated pipeline runs VLARUN blindly with the parameters set as
described in Section 3. Most of the time there are no
problems, but some exceptions are mentioned below. However, since the
pipeline runs very fast it is not a big waste of time to run it again,
especially if one is already working within AIPS. Because the pipeline
relies on a sequence of commands, the AIPS prompt is inactive during
the pipelining process - it is useful to open up another window with
AIPS running using the same AIPS user number (see below).

A personal recommendation is to do the following; this is just an
empirically tested mode of operation aimed to obtain a high success
rate on a random data set - specific data sets may run with less, or
perhaps more complicated schemes. After loading and quickly inspecting
the data scans (but no flagging yet), run the pipeline with automatic
flagging enabled ({\tt AUTOFLAG$\ge$1}), with {\tt NOPAUSE=$-$1}, and
without imaging requested ({\tt DOIMAGES=$-$1}). Check which antenna is
used as reference antenna and check whether this is proper. This will
yield first order flagging, and fluxes with errors for the phase
calibrators.  Match large errors with additional flags in the highest
numbered flagging table, and delete the files created by the pipeline
(which should be a single {\tt TASAV}-class file only). It is
suggested to leave the pipeline window for the pipeline and to use the
other window for these non-pipeline operations.

Rerun the pipeline with some plotting options (e.g., {\tt AUTOPLOT=2}),
perhaps with a different reference antenna and perhaps with a modified
solution interval. Request imaging of a reasonable field of view, with
a reasonable (low) number of iterations and a reasonable estimated
cutoff flux set. Inspect the calibration plots and images generated to
determine phase stability, residual bad data, (calibrator) source
structure, etc. Delete the files created by the pipeline (a single
{\tt 'TASAV'}-class file, and several $u,v$-files, e.g., with class
{\tt 'X-BAND'}, and corresponding image files with class {\tt
'ICL001'}).

Finally rerun the pipeline in the way the final images should be. One
can always clean deeper afterward by rerunning {\tt IMAGR} if {\tt
NITER} and {\tt CUTOFF} were set too conservatively. If {\tt SETFC} is
used, make sure to make a copy of the {\tt BOXFILE} before {\tt IMAGR}
finishes. At the time of imaging the target, a simple ``tget imagr'' in
the additional AIPS window may be very useful for later execution with
an adjusted number of iterations, field size, etc..

\section{Common problems and fixes}
Most problems relate to either bad data, bad observing conditions, or
bad pipeline constraints. In particular the automated flagging is not
yet perfected and occasionally obviously bad data may not be flagged
automatically. However, bad data is easily removed by interactive
flagging programs and rerunning the pipeline with additional flagging
information in the flagging table is trivial.

In some cases the automated flagging deletes too much data, in
particular in fast-switching observations. Or, when slewing of the
telescope leaves too little data on the first (calibrator) scan,
flagging can remove most or all of the data of those scans. If not
enough data remains, calibration solutions may be too noisy or
undetermined. Remove the flagging table, flag obvious bad data by
hand, maybe apply the flags (Section 4), and rerun the pipeline.

Other problems may occur because the input $u,v$-data file does not
have the expected structure. For example, none of the standard flux
calibrators observed has enough visibilities, the data contains
multiple frequency-ID's or subarrays, or calibrator codes are
missing. Make sure the starting conditions are ``clean'' (see
above). Occasionally unused qualifiers are attached to the source,
which can be reset using {\tt DQUAL}. Nothing can be done to correct
for ``unfortunate'' observing conditions such as weather, bad
pointing, faulty hardware and pilot errors.

Finally, sometimes the pipeline gets confused when files exist with
similar names (but, e.g., only with a different sequence). Make sure
before starting the pipeline that no files remain from a previous run
- rename or delete them to avoid confusion.

Be aware that in a few cases, a data set just cannot be
calibrated, period!
If a fix still can't be found, try to email lsjouwerman@aoc.nrao.edu
for help.

\section{Improvements}
Currently, improvements are implemented in the next version of VLARUN in the area of self-cal
algorithms and dynamic range problems, as well as processing of
``moving target'' (i.e., solar system) data. Furthermore, it should be
possible to use more than one standard flux calibrator, or a model of
the source specified in {\tt AMPCAL}. Longer time-scale improvements
are expected to
include more robust flagging and perhaps polarization calibration. It
is also possible to check and update calibrator positions and
calibrator codes on the fly and perhaps create models for the
calibrators prior to the target calibration. However the priority and
implementation of part or all of this depends on local priorities.

\section*{Acknowledgments}
Many thanks go to Eric for making some of my wishes come true, and to
Amy, John, Ed and Jared for all their suggestions and other input.

\vfill\tiny
\section*{\tiny Disclaimer}
Any information or advice given in this document is meant for general
convenience only; one oneself is held responsible for determining
whether such information or advice applies to one's particular
situation. The information and advice in this document should not be
relied upon as statements or representations of facts.

\end{document}
