%-----------------------------------------------------------------------
%;  Copyright (C) 2018
%;  Associated Universities, Inc. Washington DC, USA.
%;
%;  This program is free software; you can redistribute it and/or
%;  modify it under the terms of the GNU General Public License as
%;  published by the Free Software Foundation; either version 2 of
%;  the License, or (at your option) any later version.
%;
%;  This program is distributed in the hope that it will be useful,
%;  but WITHOUT ANY WARRANTY; without even the implied warranty of
%;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%;  GNU General Public License for more details.
%;
%;  You should have received a copy of the GNU General Public
%;  License along with this program; if not, write to the Free
%;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
%;  MA 02139, USA.
%;
%;  Correspondence concerning AIPS should be addressed as follows:
%;          Internet email: aipsmail@nrao.edu.
%;          Postal address: AIPS Project Office
%;                          National Radio Astronomy Observatory
%;                          520 Edgemont Road
%;                          Charlottesville, VA 22903-2475 USA
%-----------------------------------------------------------------------
%Body of intermediate AIPSletter for 31 December 2018 version

\documentclass[twoside]{article}
\usepackage{graphics}

\newcommand{\AIPRELEASE}{June 30, 2018}
\newcommand{\AIPVOLUME}{Volume XXXVIII}
\newcommand{\AIPNUMBER}{Number 1}
\newcommand{\RELEASENAME}{{\tt 31DEC18}}
\newcommand{\NEWNAME}{{\tt 31DEC18}}
\newcommand{\OLDNAME}{{\tt 31DEC17}}

%macros and title page format for the \AIPS\ letter.
\input LET98.MAC
%\input psfig

\newcommand{\MYSpace}{-11pt}

\normalstyle

\section{Happy 39$^{\rm th}$ birthday \AIPS}

\subsection{\Aipsletter\ publication}

We have discontinued paper copies of the \Aipsletter\ other than for
libraries and NRAO staff.  The \Aipsletter\ will be available in
PostScript and pdf forms as always from the web site listed above.
New issues will be announced in the NRAO eNews mailing and on the
bananas and mnj list server.

\subsection{Current and future releases}

We have formal \AIPS\ releases on an annual basis.  While all
architectures can do a full installation from the source files,
Linux (32- and 64-bit), and MacIntosh OS/X (Intel) systems may install
binary versions of recent releases.  Binary versions for Solaris and
MacIntosh (PPC) are available for the {\tt 31DEC17} release but not
for \NEWNAME\@.  The last, ``frozen'' release is called \OLDNAME\
while \RELEASENAME\ remains under active development.  You may fetch
and install a copy of these versions at any time using {\it anonymous}
{\tt ftp} for source-only copies and {\tt rsync} for binary copies.
This \Aipsletter\ is intended to advise you of improvements to date in
\RELEASENAME\@.  Having fetched \RELEASENAME, you may update your
installation whenever you want by running the so-called ``Midnight
Job'' (MNJ) which copies and compiles the code selectively based on
the changes and compilations we have done.  The MNJ will also update
sites that have done a binary installation.  There is a guide to the
install script and an \AIPS\ Manager FAQ page on the \AIPS\ web site.

The MNJ for binary versions of \AIPS\ now uses solely the tool {\tt
  rsync} as does the initial installation.  For locally compiled
(``text'') installations, the Unix tool {\tt cvs} running with
anonymous ftp is used for the MNJ\@. Linux sites will almost
certainly have {\tt cvs} installed; but other sites may have to
install it from the web.  Secondary MNJs will still be possible using
{\tt ssh} or {\tt rcp} or NFS as with previous releases.  We have
found that {\tt cvs} works very well, although it has one quirk. If a
site modifies a file locally, but in an \AIPS-standard directory,
{\tt cvs} will detect the modification and attempt to reconcile the
local version with the NRAO-supplied version.  This usually produces a
file that will not compile or run as intended.  For local versions,
use a copy of the task and its help file in a private disk area
instead.

\AIPS\ is now copyright \copyright\ 1995 through 2018 by Associated
Universities, Inc., NRAO's parent corporation, but may be made freely
available under the terms of the Free Software Foundation's General
Public License (GPL)\@.  This means that User Agreements are no longer
required, that \AIPS\ may be obtained via anonymous ftp without
contacting NRAO, and that the software may be redistributed (and/or
modified), under certain conditions.  The full text of the GPL can be
found in the \texttt{15JUL95} \Aipsletter, in each copy of \AIPS\
releases, and on the web at {\tt http://www.aips.nrao.edu/COPYING}.


\section{Improvements of interest in \RELEASENAME}

We expect to continue publishing the \Aipsletter\ approximately every
six months, but the publication is now primarily electronic.  There
have been several significant changes in \RELEASENAME\ in the last six
months.  Some of these were in the nature of bug fixes which were
applied to \OLDNAME\ before and after it was frozen.  If you are
running \OLDNAME, be sure that it is up to date; pay attention to the
patches and run a MNJ any time a patch relevant to you appears.
New tasks in \RELEASENAME\ include {\tt TLCAL} to convert VLA telcal
files into an initial {\tt SN} table, {\tt XYDIF} to solve for the X
minus Y phase difference for linear polarization data, {\tt PLOTC} to
display the colors used by various tasks on {\tt DO3COLOR TRUE} with
labeling to show which polarization, spectral window, channel, and/or
source goes with which color, and {\tt SPFIX} to make a spectral cube
from the intensity and spectral index images produced by {\tt
  SPIXR}\@.

{\tt 31DEC14} contains a change to the ``standard'' random parameters
in $uv$ data and adds columns to the {\tt SN} table.  Note, however,
that the random parameters written to FITS files have not been changed.
Older releases of \AIPS\ cannot handle the new {\it internal} $uv$
format and might be confused by the {\tt SN} table as well.  {\tt
  31DEC09} contains a significant change in the format of the antenna
files, which will cause older releases to do wrong things to data
touched by {\tt 31DEC09} and later releases.  You are encouraged to
use a relatively recent version of \AIPS, whilst those with recent VLA
data to reduce should get release \OLDNAME\ or, preferably, the latest
release.

\subsection{UV-data}

\subsubsection{{\tt EDITA}}

The edit class was changed to allow viewing and editing upon solution
weights for {\tt SN} and {\tt CL} tables.  The weight becomes another
variable that can be displayed and edited along with amplitude, phase,
and delay.  {\tt EDITA} and {\tt SNEDT} also offer the option with
these tables of displaying and editing only multi-band delay.  That
option had corrections made to the menu display and to force the
3-color option to false.  Dispersion was added as another
one-parameter only option like multi-band delay.

{\tt EDITA} will, when {\tt ANTENNAS} is not specified, check the
antenna table and mark any antennas labeled {\tt OUT} or with zero for
coordinates as ones to be avoided.  Code was added to determine the
median and median-absolute-deviation of all plotted parameters.  If
{\tt FLAG ABOVE} or {\tt FLAG BELOW} as specified will flag too close
to the median, the user is prompted to make sure the command is not in
error.  The display of flagged data on {\tt DO3COLOR} displays was
changed to white and the logic governing the plotting of valid and
invalid data was altered to insure correctness.

\subsubsection{Linear polarization}

The VLA receivers at around 1 and 4 meters wavelength (P and 4 bands)
have linearly polarized feeds.  To support this, code was added to
{\tt 31DEC17} to do a proper conversion from linear polarizations
to true Stokes.  Then in 2018, \AIPS\ was corrected to apply the
parallactic angle to linears only in this new routine and not also in
the routine where circular polarizations are corrected for parallactic
angle.  The ubiquitous warning message about linear polarization not
being calibrated was changed to appear only if Stokes Q, U, and/or V
are produced with {\tt DOPOL} false.

{\tt PCAL} has been found to work with linear polarizations rather
better using {\tt SOLTYPE = 'APPR'} than with the linear-polarization
specific model.  Why this should be joins the other mysteries of
polarization calibration.  Like circular polarization, the phase
difference between the two parallel polarizations has to be calibrated.
A new task called {\tt XYDIF} was written to perform this operation.
{\tt RLDIF} uses the measured RL and LR phases after calibration plus
the known position angles of certain calibration sources to measure
the R-L phase correction.  {\tt XYDIF} uses instead the fact that this
phase correction in linear polarization rotates signal from Stokes U
to Stokes V.  If the calibration source has no circular polarization
($V = 0$) but has a measurable U, then this rotation allows the phase
difference to be determined.  There are limitations to this because
there are few calibration sources that are polarized at these low
frequencies and these may have a significant rotation measure, causing
Stokes U to be very small over some ranges of frequency.  Because of
these limitations and uncertainties of 180 degrees in the answers, {\tt
  XYDIF} should be viewed as experimental rather than well
established.  In developing {\tt XYDIF} it became apparent that the
error bars in the spectral mode of {\tt RLDIF} were too large and that
was corrected.

\subsubsection{Miscellaneous}

\begin{description}
\myitem{TLCAL} is a new task to read the text file containing VLA
         ``telcal'' gain and delay solutions done in real time at the
         telescope.  They are converted to a solution ({\tt SN}) table
         which may be used for editing, {\tt GETJY}, and application
         to the calibration ({\tt CL}) table with {\tt CLCAL}\@.
\myitem{UVRANGE} has BEEN applied DIRECTLY to the $uv$ values recorded
         with each sample.  These are correct only for the reference
         frequency given in the header and so can be a factor of 2 off
         at the high end of current spectra.  This option was changed
         to apply on a per spectral-window basis which, while not
         perfectly accurate, is much closer to what is intended.
\myitem{UVSUB} had difficulties computing the model as the sole
         output.  Several patches were required to get all facets into
         the model, to get correct weights, and to work on both
         compressed and uncompressed data.
\myitem{SETJY} was corrected to quote error bars only when
         appropriate.  The explain file was overhauled to describe the
         current state of the code in detail.
\myitem{FITLD} attempted to correct the input for identical frequency
         IDs.  Unfortunately, it only correct for one identical ID and
         only corrected the visibility data.  It now corrects an
         unlimited number of identical FQIDs in both the visibility
         data and the many tables.  The logic to handle {\tt ANTNAME}
         when concatenating was defective and has been re-written in a
         manner that appears to work.
\myitem{FRING} was corrected to fill in multi-band delay when doing
         the dispersion option.  The default when writing out a
         single-source output file is now to write all channels, not
         just a single average channel.
\myitem{CALIB} now allows the use of {\tt SMODEL} with multi-source
         data sets.
\myitem{DBAPP} was corrected to use the modern structure for the $uv$
         parameter common, to write an proper index ({\tt NX}) table,
         and to continue working even when {\tt UT1UTC} differs
         between input files.
\myitem{Dispersion} corrections are now made independently for the two
         polarizations.  The values were independent but only the R or
         X value was used in calibration.
\myitem{RLDLY} was given a true solution interval using {\tt SOLINT},
         with {\tt DETIME} to indicate integration time.  {\tt
           TIMERANGE} is honored even with scans and, if appropriate,
         scans are broken up into equal intervals approximately equal
         to {\tt SOLINT}\@.  This allows for investigations to see of
         the R-L delay difference is a function of time.
\myitem{DOBAND} mode 3 was found to take blanking of one of the two
         bandpass solutions too seriously.  It now takes one of the
         two applying to a time if the other was blanked.
         Optimization failed also.
\myitem{Antenna} table writing can require a large number of keywords
         so the initialization routine was changed to request them,
         but only when creating a brand new file, not when rewriting
         an existing table.
\myitem{SNFLG} can now flag on delay and rate values out of range and
         can flag on failed solutions.
\myitem{FGSPW} now offers control of the averaging interval {\tt
         SOLINT} rather than simply doing scan averages.  The cutoff
         values can be in Jy or in scaled Jy/flux (\ie\ they are to be
         multiplied by the source table fluxes before being applied).
\myitem{PBEAM} accepts an option to state that the values come from
         two moving antennas and hence are in units of power rather
         than voltage.
\myitem{UVHOL} now does amp/phase by default rather than
         real/imaginary.  It offers an {\tt OPTYPE = 'PLOT'} option to
         plot amplitude, phase, real, or imaginary with all of the
         usual plot control adverbs.  It can plot multiple plots for
         multiple input antenna pairs (if they are not averaged).  It
         can handle plotting every single included time or averaging
         the data at each pointing position.
\end{description}

\subsection{Display}

\subsubsection{DO3COLOR and PLOTC}

Numerous tasks offer the {\tt DO3COLOR} option, as that adverb is now
spelled, to distinguish polarizations, spectral channels, spectral
windows (``IFs''), sources, and other matters.  The help files for all
of these tasks were revised to describe the color schemes used in more
detail.  Those schemes, particularly in {\tt VPLOT}, were also
adjusted.  In general, the schemes now start at pure red and go
nearly, but not quite, to pure blue.  Pure blue tends to be too dark
on workstation monitors, so color values go from 0.0 to 0.97.  Red is
used for lower channels/IFs, blue for higher and the later channels
can over-write the earlier.  Thus, if all channels have the same
value, the plot will be blue.

Since it is still not terribly clear what color goes with what
parameter in these tasks, a new task {\tt PLOTC} was written.  It
makes a TV image or plot file showing all the colors used by the
specified task, with appropriate labeling to show which is which.
Supported tasks include {\tt ANBPL}, {\tt EDITA}, {\tt EDITR}, {\tt
  ELFIT}, {\tt SNEDT}, {\tt SNIFS}, {\tt SNPLT}, {\tt UVPLT}, and {\tt
  VPLOT}\@.  Tasks that are not supported include {\tt GCPLT} and {\tt
  TARPL} which are not in common use and which employ color to
differentiate plots of different inputs.  Tasks {\tt BPLOT} and {\tt
  PCPLT} use color to distinguish times or antennas or to distinguish
intensities.  The meaning of colors in these cases should be obvious.

%\subsubsection{Miscellaneous}

\begin{description}
\myitem{SNPLT} has new options to plot phase or delay re-referenced to
          a specified {\tt REFANT}\@.
\myitem{ISPEC} has new options to plot flux or average on a log-log
          plot, suitable for spectral index studies.
\myitem{}
\end{description}

\subsection{Imaging}

\begin{description}
\myitem{IMAGR} now offers an option in the TV menu to determine and
          display robust and normal statistics for the current
          residual image.  It now also avoids re-computing image
          facets that are already up to date.
\myitem{Gridded} models require more memory in large cases at stages
          later than when the memory was obtained.  Corrected the code
          to check and increase the memory if needed.
\myitem{DFT} model computation can now do all four polarizations.  The
          addressing of the parameter telling the routines whether to
          compute the model or the data minus the model required
          correcting.
\myitem{SPIXR} was corrected for errors in frequency when {\tt BLC(1)}
          $> 1$ and for the output header frequency.  Options to flag
          solutions for excessive uncertainties in the fit parameters
          were added.
\myitem{SPFIX} is a new task to convert the outputs of {\tt SPIXR}
          into a well-behaved image cube or a residual image cube.
\myitem{CONVL} will now determine values for the beam parameters from
          the {\tt CG} table if they are not specified in {\tt OPCODE
            = 'GAUS'}\@.
\myitem{COMB} was changed to report in the messages any use of
          beam-area correction factors; they appeared only in the
          history file previously.  Said factors are no longer used in
          the {\tt MULT} operation.
\myitem{OMFIT} was changed to allow many antennas and the usage of
          dynamic memory was corrected.
\myitem{FQUBE} now writes at most one output {\tt CG} table.  It wrote
          many tables when the input file had none.
\end{description}


\subsection{General}

Chapter 4 of the \Cookbook\ was completely re-written to describe the
modern JVLA\@.  Appendix E was removed since it was both obsolete in
some of its wording and replaced entirely by the new Chapter 4.  A new
Appendix O (``obsolete'') was added to describe data reduction for the
old VLA using sections from the old Chapter 4.  VLB MK III and IV
details were also moved from Chapter 9 to Appendix O\@.  Appendix L
(low frequency) was given a new, short section on polarization.

\begin{description}
\myitem{Batch} in \AIPS\ was found to be broken due to bad call
           sequences.  I guess few people use this, but it now works
           again.
\myitem{BADDISK} was added to a large number of tasks which might have
           to sort large tables, such a flag, pulse-cal, or SysPower
           tables.
\myitem{MacIntosh} compilation procedures were given new case
           statements allowing for gfortran/gcc compilers.  A Mac
           version with gfortran was tested and shown to work.  The
           binaries we provide are still compiled with the Intel
           compiler.  This allows us to ship binaries which will work
           on older versions of Mac OSX\@.
\end{description}

\section{Patch Distribution for \OLDNAME}

Important bug fixes and selected improvements in \OLDNAME\ can be
downloaded via the MNJ or from the Web beginning at:
\hspace{3em}{\tt http://www.aoc.nrao.edu/aips/patch.html}\\
Alternatively one can use {\it anonymous} \ftp\ to the NRAO server
{\tt ftp.aoc.nrao.edu}.  Documentation about patches to a release is
placed on this site at {\tt pub/software/aips/}{\it release-name} and
the code is placed in suitable sub-directories below this.  As bugs in
\NEWNAME\ are found, they are simply corrected since \NEWNAME\ remains
under development.  Corrections and additions are made with a midnight
job rather than with manual patches.  Because of the many binary
installations, we now actually patch the master version of \OLDNAME,
meaning that a MNJ run on \OLDNAME\ after the patch will fetch the
corrected code and/or binaries rather than failing.  Also,
installations of \OLDNAME\ after the patch date will contain the
corrected code.  The \OLDNAME\ release has had a number of patches:
\begin{enumerate}
  \item\ Linear polarization handling applied the parallactic angle
      twice. {\it 2018-01-09}
  \item\ Batch has not worked for some years. {\it 2018-01-16}
  \item\ {\tt CALIB} would not use {\tt SMODEL} on multi-source data
      sets  {\it 2018-01-16}
  \item\ {\tt UVSUB} did not do {\tt OPCODE 'MODL'} correctly for
      multiple facets.  {\it 2018-01-20}
  \item\ {\tt SETJY} printed erroneous error bars with the latest 2017
      flux scale. {\it 2018-03-01}
  \item\ {\tt UVSUB} did not do {\tt OPCODE 'MODL'} correctly; weights
      came out zero.  {\it 2018-03-14}
  \item\ Large gridded models did not allocate sufficient memory. {\it
      2018-04-04}
  \item\ Antenna file reformatting in place failed on files with large
      numbers of IFs. {\it 2018-04-04}
  \item\ {\tt FITLD} did not correct duplicate FQ IDs properly. {\it
      2018-04-04}
  \item\ {\tt DBAPP} had an old version of the data pointers and
      failed to copy modern data properly.  {\it 2018-04-09}
  \item\ {\tt FQUBE} wrote many {\tt CG} tables when none were input.
      {\it 2018-04-23}
  \item\ {\tt FITLD} did not handle antenna files properly when
      concatenating. {\it 2018-04-24}
  \item\ {\tt DOBAND 3} interpolation flagged too much data with
      failed bandpass solutions. {\it 2018-04-27, 2018-06-26}
  \item\ Model computation made errors in DFT mode when 4
      polarizations were being computed. {\it 2018-06-20}
  \item\ {\tt UVSUB} did not do {\tt MODL} correctly with compressed
      data. {\it 2018-06-20}
\end{enumerate}

\vfill\eject

% mailer page
% \cleardoublepage
\pagestyle{empty}
 \vbox to 4.4in{
  \vspace{12pt}
%  \vfill
\centerline{\resizebox{!}{3.2in}{\includegraphics{FIG/Mandrill.eps}}}
%  \centerline{\rotatebox{-90}{\resizebox{!}{3.5in}{%
%  \includegraphics{FIG/Mandrill.color.plt}}}}
  \vspace{12pt}
  \centerline{{\huge \tt \AIPRELEASE}}
  \vspace{12pt}
  \vfill}
\phantom{...}
\centerline{\resizebox{!}{!}{\includegraphics{FIG/AIPSLETS.PS}}}

\end{document}


\section{\AIPS\ Distribution}

We are now able to log apparent MNJ accesses and downloads of the tar
balls.  We count these by unique IP address.  Since some systems
assign the same computer different IP addresses at different times,
this will be a bit of an over-estimate of actual sites/computers.
However, a single IP address is often used to provide \AIPS\ to a
number of computers, so these numbers are probably an under-estimate
of the number of computers running current versions of \AIPS\@. In
2017, there have been a total of 315 IP addresses so far that have
accessed the NRAO cvs master.  Each of these has at least installed
\AIPS\@.  During 2018 more than 290 IP addresses have downloaded the
frozen form of \OLDNAME, while more than 395 IP addresses have
downloaded \RELEASENAME\@.  The binary version was accessed for
installation or MNJs by 269 sites in \OLDNAME\ and 346 sites in
\RELEASENAME\@.  A total of 728 different IP addresses have appeared
in one of our transaction log files.  These numbers are slightly lower
than last year, which was lower than the year before.
