\input COOK82.MAC
\def\begintt{\let\par=\endgraf \ttverbatim \parskip=0pt
   \catcode`\|=0 \rightskip=-5pc \ttfinish}
\def\ttverbatim{\begingroup \catcode`\\=\other \catcode`\{=\other
     \catcode`\}=\other \catcode`\$=\other \catcode`\&=\other
     \catcode`\#=\other \catcode`\%=\other \catcode`\~=\other
     \catcode`\_=\other \catcode`\^=\other
     \obeyspaces \obeylines \tenpoint\tt}
{\catcode`\|=0 |catcode`|\=\other % | is temporary escape character
  |obeylines % end of line is active
  |gdef|ttfinish#1^^M#2\endtt{#1|global|setbox9=|vbox{#2}|endgroup}}
\catcode`\|=\active
% Needed to get literal equations correct.
\def\text#1{\hbox{\it#1}}
\def\chapt{Y}
\sect{Appendix Y.  FILE SIZES}
\pd


Your data reduction strategy will be more effective if you have an idea of
how big the data and map file sizes will be on disk.  Also, it will help
you estimate just how many files you can backup to tape.


\subsect{Y.1.  Blocks {\it versus} Bytes}

Conventionally, we are used to working in units of {\it blocks} for the sizes
of files.  Unfortunately, this is not a very useful unit since {\it blocks}
mean different things in different places \ie\ a {\it block} is a completely
arbitrary unit.  On the other hand, a {\it byte} is a well defined unit;  it
is eight bits regardless of context.  It makes much more sense to measure the
sizes of files in terms of {\it bytes}.  You are encouraged to develop this
healthier way of thinking; it makes for more meaningful communication when you
request disk space!  (Just to be awkward, a block on a Convex, SUN or
VAX is {\bf now} 1024 bytes!)

To calculate the size of the data files (in bytes, of course), you must first
understand just how data is stored on the data disks.


\subsect{Y.2.   Compressed format for \uv\ data}

The use of ``compressed'' data can make substantial savings in the amount of
disk space that you require, particularly so for spectral line databases.  All
of the calibration routines can accommodate either the compressed or the
uncompressed formats.
Compressed data files can be identified by the dimension of 1 for the
``COMPLEX'' in the database header.  (Uncompressed data will have a dimension
of 3.)  The savings can be close to a factor of three for spectral line
observations.

How is this achieved?  Essentially, this is done by assigning only one each of
the ``WEIGHT'' and ``SCALE'' random parameters per visibility record and
packing the real and imaginary terms into one REAL value with magic value
blanking.  This is to be compared with the uncompressed format in which each
of the real, imaginary and weight terms are stored in a REAL.


\subsubsect{Y.2.1. When NOT to use compressed format}

In general, compressed data should be used and this is the default in the
task {\tt FILLM}.  At worst (single frequency, single IF, single polarisation
data), you will
not save any disk space.  In all other cases, there are respectable savings
to be made.  However, the use of a packed REAL for the real and imaginary
parts of the visibility function along with magic value blanking imposes a
restriction on the ``spectral dynamic range'' of the data set of around 32000:1.
Consequently, there are some situations where compressed data should {\bf not}
be used.  For example, if the spectral dynamic range in the \uv\ database is
likely to be greater than 32000:1, then you must use {\bf uncompressed}
data format.  A situation where this might occur is where you have a spectrum
in which there are maser lines of 1~Jy and $ >32000$~Jy; in this case, you
should not use compressed data.  {\bf In general, continuum datasets should
be loaded with data compression} since the above spectral dynamic range limit
will not be encountered normally.

\vfill
\eject

\subsect{Y.3.  How is data stored on disk?}

There are two kinds of file types in \AIPS.  The first are {\it map} files
--- this is a generic term for \uv\ and image files.  The second are regular
disk files; these are generally what are known as ``extension files'' in \AIPS\
\eg\  PL files or CL tables.

\uv\ data files contain information about the
coherence function of a wavefront at random locations and random times and
consequently, the way this information is stored on disk is different from
that for images where the pixels are on a regular grid.  AIPS \uv\ data is
stored on disk in a manner similar to the way that it would be organised
on a FITS ``random group'' tape.  The data is stored as logical records; each
record (a ``visibility'') contains {\bf all} the data taken on one baseline at
a given time.
Consequently, a record may contain information for several IFs, several
frequencies at each of those IFs and more than one polarisation
combination for each frequency/IF.  The first part of each logical record
contains what are known as ``random parameters'' \eg\ spatial frequency
co-ordinates and time.  After the random parameters, there is a small,
regular array of data.

    For an uncompressed, multi-source data set such as might be created by
{\tt FILLM}, the random parameter group will contain the following:

    UU-L-SIN, VV-L-SIN, WW-L-SIN, TIME1, SOURCE, BASELINE

``UU-L-SIN'' {\it etc.}\ indicate that the spatial frequency coordinates are
computed
with a sine projection and are given in terms of wavelengths at the reference
frequency.  ``TIME1'' is the label for the time for historical reasons.
``BASELINE'' and ``SOURCE'' are self-obvious labels.  If you have frequency
table identifiers (which is always the case for current data) then there will
be an additional random parameter, ``FQSEL''.  For a compressed data
base, two additional random parameters will be required --- ``WEIGHT'' and
``SCALE''.

The regular data array is similar to an image array in that the order of axes
is arbitrary.  However, the convention is for the first axis to be of type
``COMPLEX''.  This will have a dimension of 3 for uncompressed data (real,
imaginary, weight) and be of dimension 1 for compressed data.  The other axes
of the regular array are ``RA'', ``DEC'', ``FREQ'' and ``STOKES''.



\subsect{Y.4.  \uv\ database sizes}

The size of the database to be loaded to disk with {\tt FILLM} is given by
the following formula:

  $$\eqalignno{   \Bigr\lbrack \lbrace (\#~random~parameters)
                  + \lbrack (dimension~of~COMPLEX~axis)
                            \times (\#~polns)
                            &\times (\#~freqs)
                             \times (\#~IFs) \rbrack \rbrace  \cr
                            & \times (\#~vis)
                              \times 4 \Bigr\rbrack
                                        \rm\,bytes } $$

The number of visibilities is given by
  $${length~of~observation \over integration~time} \times
                                          number~of~baselines $$
where the number of baselines is the usual ${1\over2}n(n-1)$.  This is equal to
351 for the VLA for the usual 27 antenna array.

So, for example, a 12 hour observation with 30 second integrations,
1 frequency and 2 IFs with RR, RL, LR and LL written in compressed format
({\tt DOUVCOMP = TRUE} in {\tt FILLM}) will occupy about 34 Mbytes on disk.
In practice, the \uv\ file will usually be a little larger due to the way the
system allocates space on the disks. You must also remember to allow room for
the extension tables --- see \Sec Y.5.   If this database had been written
in uncompressed format, the \uv\ data would have occupied around 62~Mbyte.

Consider another example.  {\us IMHE} generated the listing below:
\beginc{50pt}
\begintt

Image=MULTI     (UV)         Filename=20/07/90    .L BAND.   1
Telescope=VLA                Receiver=VLA
Observer=AFTST               User #= 1364
Observ. date=20-JUL-1990     Map date=23-JUL-1990
# visibilities    105198     Sort order  TB
Rand axes: UU-L-SIN  VV-L-SIN  WW-L-SIN  BASELINE  TIME1
           SOURCE  FREQSEL  WEIGHT  SCALE
--------------------------------------------------------------
Type    Pixels   Coord value  at Pixel    Coord incr   Rotat
COMPLEX      1   1.0000000E+00    1.00 1.0000000E+00    0.00
STOKES       4  -1.0000000E+00    1.00-1.0000000E+00    0.00
IF           2   1.0000000E+00    1.00 1.0000000E+00    0.00
FREQ         1   1.4524000E+09    1.00 2.5000000E+07    0.00
RA           1    00 00 00.000    1.00      3600.000    0.00
DEC          1    00 00 00.000    1.00      3600.000    0.00
--------------------------------------------------------------
Maximum version number of extension files of type HI is   1
Maximum version number of extension files of type AN is   1
Maximum version number of extension files of type NX is   1
Maximum version number of extension files of type SU is   1
Maximum version number of extension files of type FQ is   1
Maximum version number of extension files of type CL is   1
Maximum version number of extension files of type SN is   2
\endtt
\endc

\vskip 50pt
How much space does this compressed ({\tt COMPLEX Pixels = 1}) \uv\ database
occupy?  There are 9 random parameters, 4 polarisations, 1 frequency and
2 IFs.  The listing also shows that the database contains 105198
visibilities.  So, the size of the \uv\ part of the database is:
$$ \Bigl\lbrack \lbrace 9 + \lbrack 1 \times 4 \times 1 \times 2 \rbrack
         \rbrace \times 105198 \times 4 \Bigr\rbrack \rm\,bytes $$
$$ = 7.153 \rm\,Mbytes $$



\subsect{Y.5.  How big are the extension tables?}

Some extension tables --- principally the CL tables --- can become
rather large.

The CL (calibration) table contains the total model of the interferometer at
each interval.  Many of these logical records are blank in
the case of most interferometers but, for the VLBA, these records will contain
essential information.  The CL table contains a logical record for each antenna
in the array at each CL time stamp.  The CL time stamps are set by the user
when loading the data.  The default for VLA data is every~5 minutes.  For VLBI
data, it is every 1~minute.  Each CL logical record requires

  $$\lbrace 15 + \lbrack 14 \times  (\#~IFs) \times
                       (\#~pols)\rbrack\rbrace \times 4 \rm\, bytes$$

For a 12 hour observation with the VLA with 2 IFs and 4 polarisation pairs,
with entries every 5 minutes, the CL table will occupy about
$$ \Bigl\lbrack\lbrace 15 + \lbrack 14 \times 2 \times 4 \rbrack\rbrace
   \times 4 \times {(12 \times 60) \over 5 } \times 27 \Bigr\rbrack
   \rm\, bytes$$
$$ = 1.975 \rm\, Mbytes$$

Since the other files are noticeably smaller, their sizes can be ignored.


\subsect{Y.6.  How big are my image files?}

Since images are regular arrays, the sizes of image files are easier to
calculate than for \uv\ data files.
Generally, the images are stored as floating point numbers \ie\ 32 bits per
pixel so in bytes the image file size is given by
   $$ 4 \times \prod_{i} length(i) $$
where $length(i)$ is the number of pixels on the $i^{th}$ axis.  For example, a
128 channel cube with each plane a $256 \times 256$ image will require around
34 Mbytes of disk storage space.  This will be increased a little due to the
way in which the system allocates space for files.



\subsect{Y.7.  Storing data on tape}

Eventually, you will wish to write your images or \uv\ databases to tape for
archival purposes and for transfer to another machine.  How much data can you
reasonably expect to fit on a tape?

Current tape drives are {\it nine track} devices \ie\ the data is recorded
on eight tracks and the ninth track is used to record {\it parity bits} for
error checking.  Standard recording densities are 800, 1600 and 6250 bits per
inch per track ({\it bpi}).  Generally, data is recorded at 6250 bpi wherever
possible.

Each tape has a reflective metallic marker near the start of the tape to
signify {\it beginning-of-tape} or BOT.  The BOT marker is used to allow
the tape drive to position the tape at the starting position.  Similarly, an
EOT ({\it end-of-tape}) maker is located near the physical end of the tape.
This is sensed by the tape drive and control circuitry prevents the device
spooling the tape completely off the main spool;  this is very useful for
self-loading tape drives!

Note that plot (PL) files are not saved to tape with the task {\tt FITTP}.


\subsubsect{Y.7.1.  Tape sizes}

The standard tape length on a regular size reel is 2400 feet long.  However,
by the use of a thinner tape substrate for the oxide layer (made of Mylar),
the regular spool can accomodate up to 3600 feet of tape.

\vfill
\eject

\subsubsect{Y.7.2.  Blocking density}

Data are written on to the tape in {\it blocks}.  In general, the block size is
2880 bytes.  After each block, an {\it inter-record gap} is left before the
next block is written.  The {\it tape blocking efficiency} is a measure of the
amount of essentially wasted space.  It is defined as
$$ { {block~size \over recording~density} \over
           { \lbrack \lbrace {block~size \over recording~density} \rbrace
           + \text{\it{length~of~inter-record~gap}}} \rbrack } $$
The lengths of the inter-record gaps are roughly $3\over5$ of an inch at
recording densities of 800 and 1600 bpi and $3\over10$ of an inch at 6250 bpi.
This gives a tape blocking efficiency of around 0.61 for a 6250 bpi tape
using 2880 byte blocks.

It is possible to squeeze more data on to tape by reducing the amount of space
given over to the inter-record gaps.  This is what the \AIPS\ adverb {\tt
BLOCKING} lets you control in the task {\tt FITTP}.  {\us BLOCKING = {\it n}}
assembles the data into block sizes of {\it n} $\times$ 2880 bytes before
writing the
data to tape.  So, for {\us BLOCKING = {\it 10}}, the block sizes are 28,800
bytes long and the tape blocking factor improves to around 0.95 for a 6250
bpi tape.  In other words, you can fit more than half again on to the same
tape.

In round numbers, the capacity of a standard 2400 feet tape is therefore about
110 Mbytes using {\us BLOCKING = 1} and around 170 Mbyte using
{\us BLOCKING = 10}
(the maximum).  The default value is {\us BLOCKING = 2} and this will allow you
to record around 135 Mbyte on the standard tape at 6250 bpi.


\subsubsect{Y.7.3.  Tape formats}

Several formats are available within the task {\tt FITTP} and are controlled
through the adverb {\tt FORMAT}.  The formats are:

\itemitem{(1) }  16 bit integer
\itemitem{(2) }  32 bit integer
\itemitem{(3) }  32 bit IEEE floating point

For the integer {\it n} formats, \AIPS\ finds the maximum and minimum value in
the image or data file and maps the range on to the integers between
$ 2^{n-1}-1 $ and  $ -(2^{n-1}-1)$.  (A sign bit is also required.)  This is
what is happening when \AIPS\ reports ``First find the scaling parameters''
when using the integer formats when the task {\tt FITTP} starts up.

For the 16 bit integer format, the effective ``dynamic range'' with which
images or \uv\ datasets can be stored on tape is thus about 65000:1.  For
many situations, this format is perfectly adequate. If your image or
data has a dynamic range of greater than this then you should use one of the
32 bit formats.  Which one should you use?  For most purposes, it is better
to use the 32 bit floating point format.  For host computers which support
the IEEE 32 bit standard, the acts of reading from and writing to tape then
become simple copies; no bit manipulation has to be performed.  Consequently,
the tape reading and writing tasks are executed much faster.  A more minor
benefit of the floating point format is that the numbers representing your data
are recorded exactly on tape as they are stored on disk; there are no
``quantisation errors''.  This may be important for software development.

Note, too, that for the integer formats the numbers representing the time
stamp of each integration have to be mapped on to the available integer
range.  For the 32 bit integer format, there is sufficient range to handle
all conceivable situations.  However, in the case of the 16 bit integer format,
the precision with which the timestamp can be represented is $2^{-16}$ of a
day {\it i.e.} 1/65536 of a day or 1.318 seconds.  This means that if your
data have integration times of less than this, you will lose precision when you
write your data to tape with the 16 bit integer format. For most VLA observers,
the basic integration time is 10~seconds or greater.  However, solar observers
often use integration periods much less than this; in general, they should
use one of the 32 bit formats when archiving their data.

    The default behaviour of {\tt FITTP} is to use the 32 bit IEEE
floating point format.

\subsubsect{Y.7.4.  How \AIPS\ writes the data on tape}

The preceding sections do not tell the full story, however.  \AIPS\ does
currently does not write compressed \uv\ data to tape in a compressed format.
Instead, it effectively expands the data into the uncompressed form first
then writes the data on tape (with scaling for the integer formats, if
required).

In the conversion, the real and imaginary values that were stored in one
packed real number are expanded into three real values --- one each for
real, imaginary and weight terms --- and the weight and scale random
parameters are removed since these are required no longer. If the data is
written in 16 bit format, the `TIME1' random parameter will be
stored in {\bf two} random parameters to maintain the required precision.
Consequently, for the 16 bit format, the compressed data is expanded to
$$
\lbrace (\#~random~parameters - 1 ) + \lbrack (\#~pol) \times (\#~IFs) \times
                                  (\#~frequencies) \times 3 \rbrack \rbrace
\over
\lbrace (\#~random~parameters) + \lbrack (\#~pol) \times (\#~IFs) \times
                                      (\#~frequencies) \rbrack \rbrace
$$
the original size  (where $\#~random~parameters$ is the original number in
the compressed database).  After the expansion, the data is scaled for the
integer formats, if required.  For the 16 bit format, the size of the database
on tape is half of the expanded size since the 32 bit format on disk is mapped
to 16 bits on tape.  (For the 32 bit formats, the number of random parameters
in the expanded database is 2 less than were in the original database.)

Consider the example of the following multi-source spectral line data base
stored on disk in compressed format.  There were seven channels at 2 IFs,
each with 2 polarisations.  The random parameters were --- UU-L-SIN, VV-L-SIN,
WW-L-SIN, TIME1, SOURCE, BASELINE, FREQSEL, WEIGHT and SCALE.  The database
contained 834031 visibilities.  From \Sec Y.4, we can calculate the size
of the \uv\ file to be 123~Mbytes.  (Remember, this doesn't include any  of
the extension files, some of which might be several Mbytes in size.)   Before
the file is written to tape in 16 bit integer format, it is first expanded by
a factor of
$$
 \lbrace (9 - 1) + \lbrack 2 \times 2 \times
                                      7 \times 3 \rbrack \rbrace
\over
\lbrace 9 + \lbrack 2 \times 2 \times
                                      7 \rbrack \rbrace $$
$$ = 2.486 $$
after which it is effectively reduced by a factor of two by the conversion
to the 16 bit format.  Consequently, the data will occupy
$$ {{123 \times 2.486}  \over 2} \rm\, Mbytes$$
$$ = 153\rm\, Mbytes $$
on tape.  In other words, this data base and all the associated extension
files are likely to fit on one standard, 6250~bpi tape using {\us BLOCKING = 10}
(\Sec Y.7.2) and {\us FORMAT = 1} (\Sec Y.7.3).



\subsect{Y.8.  Help!  I have a very large database that I can't
fit on one tape.}

{\tt FITTP} can not write multi-volume tapes.  Some spectral line databases
(and perhaps some continuum databases) may be so large that the file can not
fit on one tape even with {\us FORMAT = 1} and {\us BLOCKING = 10}.  What can
you do to backup your data?  There are several approaches that you can adopt.

First, you could {\tt SPLIT} out the database into {\it single source}
databases and back each of these up individually.  Alternatively, you could
subdivide the large database in several smaller databases with {\tt UVCOP}
by specifying a different time range for each of the smaller databases, then
back these up individually.  Another way to solve the problem is to realise
that the calibration and flagging information that you have carefully generated
during the calibration is contained in the extension tables   --- the raw data
that you loaded is not modified until you finally {\tt SPLIT} out the individual
sources.  Consequently, you can write create a dummy \uv\ database to which all
the extension tables are attached with the task {\tt TASAV}, then save this
``database'' on tape with {\tt FITTP}.  The raw visibilities can be saved
in the form of copies of the archive tapes.


\subsubsect{Y.9.  An example}

Images are stored in floating point as 4 bytes/pixel.  This means that a
typical image of $512 \times 512$ pixels will occupy around 1 Mbyte on disk.
Backing up such images to tape in 32 bit IEEE floating point ({\us FORMAT = 3})
with {\us BLOCKING = 10} will allow you to fit over 150 such images to a
standard 2400 feet tape at 6250 bpi.



\vfil\eject

\pgskip

\end
