\chapter{Graphics}
\label{chap:graphs}

\section{Gnuplot graphs}
\label{gnuplot-graphs}

A separate program, \app{gnuplot}, is called to generate graphs.
Gnuplot is a very full-featured graphing program with myriad options.
It is available from \href{http://www.gnuplot.info/}{www.gnuplot.info}
(but note that a suitable copy of gnuplot is bundled with the packaged
versions of gretl for MS Windows and Mac OS X).  Gretl gives you
direct access, via a graphical interface, to a subset of gnuplot's
options and it tries to choose sensible values for you; it also allows
you to take complete control over graph details if you wish.

With a graph displayed, you can right-click on the graph window (or
use the ``hamburger'' toolbar button) for a pop-up menu with the
following options.

\begin{itemize}
\item \textsf{Save as PNG}: Save the graph in Portable Network
  Graphics format (the same format that you see on screen).
\item \textsf{Save as postscript (EPS)}: Save in encapsulated
  postscript format.
\item \textsf{Save as PDF}: Save in PDF format.
\item \textsf{Save as Windows metafile}: Save in Enhanced Metafile
  (EMF) format (with color and monochrome options).
\item \textsf{Copy to clipboard}: with color and monochrome options.
\item \textsf{Save to session as icon}: The graph will appear in
  iconic form when you select ``Icon view'' from the View menu.
\item \textsf{Zoom}: Lets you select an area within the graph for
  closer inspection (not available for all graphs).
\item \textsf{Display PDF}: view a PDF version of the graph.
\item \textsf{Edit}: Opens a controller for the plot which lets you
  adjust many aspects of its appearance.
\item \textsf{Close}: Closes the graph window.
\end{itemize}

If you select \textsf{Save as postscript} or \textsf{Save as PDF} you
get a dialog box that lets you adjust several aspects of the graph,
and also preview the result.

\subsection{Displaying data labels}
\label{plot-labels}

For simple X-Y scatter plots, some further options are available if
the dataset includes ``case markers'' (that is, labels identifying
each observation).\footnote{For an example of such a dataset, see the
  Ramanathan file \verb+data4-10+: this contains data on private
  school enrollment for the 50 states of the USA plus Washington, DC;
  the case markers are the two-letter codes for the states.} With a
scatter plot displayed, when you move the mouse pointer over a data
point its label is shown on the graph.  By default these labels are
transient: they do not appear in the printed or copied version of the
graph.  They can be removed by selecting ``Clear data labels'' from
the graph pop-up menu. If you want the labels to be affixed
permanently (so they will show up when the graph is printed or
copied), select the option ``Freeze data labels'' from the pop-up
menu; ``Clear data labels'' cancels this operation.  The other
label-related option, ``All data labels'', requests that case markers
be shown for all observations.  At present the display of case markers
is disabled for graphs containing more than 250 data points.


\subsection{GUI plot editor}
\label{plot-editor}

Selecting the \textsf{Edit} option in the graph popup menu opens an
editing dialog box, shown in Figure~\ref{fig-plot}.  Notice that there
are several tabs, allowing you to adjust many aspects of a graph's
appearance: font, title, axis scaling, line colors and types, and so
on.  You can also add lines or descriptive labels to a graph (under
the Lines and Labels tabs).  The ``Apply'' button applies your changes
without closing the editor; ``OK'' applies the changes and closes the
dialog.

\begin{figure}[htbp]
  \begin{center}
    \includegraphics[scale=0.6]{figures/plot_control}
  \end{center}
  \caption{gretl's plot controller}
  \label{fig-plot}
\end{figure}


\subsection{Publication-quality graphics: advanced options}
\label{plot-advanced}

The GUI plot editor has two limitations.  First, it cannot represent
all the myriad options that \app{gnuplot} offers. Users who are
sufficiently familiar with \app{gnuplot} to know what they're missing
in the plot editor presumably don't need much help from gretl,
so long as they can get hold of the \app{gnuplot} command file that
gretl has put together.  Second, even if the plot editor meets
your needs, in terms of fine-tuning the graph you see on screen, a few
details may need further work in order to get optimal results for
publication.

Either way, the first step in advanced tweaking of a graph is to get
access to the graph command file.

\begin{itemize}
\item In the graph display window, right-click and choose ``Save to
  session as icon''.
\item If it's not already open, open the icon view window---either
  via the menu item View/Icon view, or by clicking the ``session icon
  view'' button on the main-window toolbar.
\item Right-click on the icon representing the newly added graph and
  select ``Edit plot commands'' from the pop-up menu.
\item You get a window displaying the plot file
  (Figure~\ref{fig:plot-edit}).
\end{itemize}

\begin{figure}[htbp]
  \centering
  \includegraphics[scale=0.6]{figures/plotedit}
  \caption{Plot commands editor}
  \label{fig:plot-edit}
\end{figure}

Here are the basic things you can do in this window.  Obviously, you
can edit the file you just opened.  You can also send it for
processing by gnuplot, by clicking the ``Execute'' (cogwheel) icon in
the toolbar.  Or you can use the ``Save as'' button to save a copy for
editing and processing as you wish. And please note that the Help
button on the toolbar (a lifebelt in Figure~\ref{fig:plot-edit}) gives
you access to the \app{gnuplot} manual.

One relatively simple editorial job would be to set a chosen driver
(or ``terminal'' in gnuplot parlance) and output filename. For
example, to get PDF output you could insert lines like the following
at the top:

\begin{code}
# PDF, slightly amended (the default size is 5in x 3in)
set term pdfcairo font "Sans,6" size 5in,3.5in
set output 'mygraph1.pdf'
# or small size
set term pdfcairo font "Sans,5" size 3in,2in
set output 'mygraph2.pdf'
# or with size given in centimeters
set term pdfcairo font "Sans,6" size 6cm,4.2cm
set output 'mygraph3.pdf'
\end{code}

Or substitute \texttt{epscairo} for \texttt{pdfcairo} (and change the
filenames) if you want EPS output. However, such changes may be more
easily made via the \textsf{Save as PDF} and \textsf{Save as
  postscript} options in the plot menu.\footnote{A ``traditional''
  \texttt{postscript} terminal may also be available in gnuplot, with
  an \texttt{eps} option. The defaults in this case are quite
  different from \texttt{epscairo}, and to make use of the alternative
  you'll have to consult the \app{gnuplot} manual.}

The real payoff to editing the plot code can be obtained if you dive
into the details and employ \app{gnuplot} features that are not
accessible via gretl, and/or use one of the terminal types not
directly supported by gretl, such as \texttt{context} (ConTeXt),
\texttt{mp} (MetaPost), \texttt{lua} (Lua) or \texttt{pslatex}
(\LaTeX\ picture environment with PostScript specials). The
\texttt{lua} terminal with the \texttt{tikz} option is especially
useful for \LaTeX\ users, because it produces a \texttt{tikzpicture}
environment, which offers almost unlimited customization possibilities
(note that in order to use plots produced in this way you'll also need
the \texttt{gnuplot-lua-tikz} \LaTeX\ package).

To find out more about \app{gnuplot} visit
\href{https://gnuplot.sourceforge.net/}{gnuplot.sourceforge.net}. This
site has documentation for the current version of the program in
various formats along with a large collection of demonstration plots.

\subsection{Additional tips}
\label{subsect-graph-tips}

To be written.  Line widths, enhanced text.  Show a ``before and
after'' example.  

\section{Plotting graphs from scripts}
\label{sec:plotenv}

When working with scripts, you may want to have a graph shown onto
your display or saved into a file. In fact, if in your usual workflow
you find yourself creating similar graphs over and over again, you
might want to consider the option of writing a script which automates
this process for you. gretl gives you two main tools for doing
this: one is a command called \cmd{gnuplot}, whose main use is to
create standard plot quickly. The other one is the \cmd{plot} command
block, which has a more elaborate syntax but offers you more control
on output.

\subsection{The \cmd{gnuplot} command}
\label{sec:gnuplot-cmd}

The \cmd{gnuplot} command is described at length in the \GCR\ and the
online help system. Here, we just summarize its main features:
basically, it consists of the \cmd{gnuplot} keyword, followed by a
list of items, telling the command \emph{what} you want plotted and a
list of options, telling it \emph{how} you want it plotted.

For example, the line
\begin{code}
gnuplot y1 y2 x   
\end{code}
will give you a basic XY plot of the two series \texttt{y1} and
\texttt{y2} on the vertical axis versus the series \texttt{x} on the
horizontal axis. In general, the arguments to the \cmd{gnuplot}
command is a list of series, the last of which goes on the x-axis,
while all the other ones go onto the y-axis. By default, the
\cmd{gnuplot} command gives you a scatterplot. If you just have one
variable on the y-axis, then gretl will also draw a the OLS
interpolation, if the fit is good enough.\footnote{The technical
  condition for this is that the two-tailed $p$-value for the slope
  coefficient should be under 10\%.}

Several aspects of the behavior described above can be modified. You
do this by appending options to the command. Most options can be
broadly grouped in three categories:
\begin{enumerate}
\item Plot styles: we support points (the default choice), lines,
  lines and points together, and impulses (vertical lines). 
\item Algorithm for the fitted line: here you can choose between
  linear, quadratic and cubic interpolation, but also more exotic
  choices, such as semi-log, inverse or loess (non-parametric). Of
  course, you can also turn this feature off.
\item Input and output: you can choose whether you want your graph on
  your computer screen (and possibly use the in-built graphical widget
  to further customize it --- see above, page \pageref{plot-editor}),
  or rather save it to a file. We support several graphical formats,
  among which PNG and PDF, to make it easy to incorporate your
  plots into text documents.
\end{enumerate}

Listing~\ref{awm-plots} shows examples of some traditional plots in macroeconomics,
using time series from the ``area-wide model'' dataset produced by the 
European Central Bank, which is shipped with gretl in the file \texttt{AWM.gdt}.
\texttt{PCR} is aggregate private real consumption and \texttt{YER} is real GDP.

\begin{script}
  \caption{Plotting macroeconomic data}
  \label{awm-plots}
\begin{scode}
open AWM.gdt --quiet

# --- consumption and income, different styles ------------

gnuplot PCR YER
gnuplot PCR YER --output=display
gnuplot PCR YER --output=display --time-series
gnuplot PCR YER --output=display --time-series --with-lines

# --- Phillips' curve, different fitted lines -------------

gnuplot INFQ URX --output=display
gnuplot INFQ URX --fit=none --output=display
gnuplot INFQ URX --fit=inverse --output=display
gnuplot INFQ URX --fit=loess --output=display
\end{scode}
\end{script}

The first command line in the listing
plots consumption against income as a kind of Keynesian 
consumption function. More precisely, it produces a simple scatter plot with
an automatically linear fitted line. If this is executed in the gretl console
the plot will be directly shown in a new window, but if this line is contained
in a script then instead a file with the plot commands will be saved for later
execution. The second example line changes this behavior for a script command
and forces the plot to be shown directly. 

The third line instead asks for a plot of the two variables as two separate
curves against time on the x-axis. Each observation point is drawn separately
with a certain symbol determined by gnuplot defaults. If you add the option
\cmd{--with-lines} the points will be connected with a continuous line and the
symbols omitted.

The second batch of examples demonstrate how the fitted line in the 
scatter plot can be controlled from gretl's side. The option \cmd{--fit=none}
overrides gnuplot's default to draw a line if it deems the fit to be ``good
enough''. The effect of \cmd{--fit=inverse} is to consider the variable on the
y-axis as a function of $1/X$ instead of $X$ and draw the corresponding
hyperbolic branch. For the workings of a Loess fit (locally-weighted polynomial 
regression) please refer to the documentation of the \cmd{loess} function.

For more detail, consult the \GCR.


\subsection{The \cmd{plot} command block}
\label{sec:plotblock}

The \cmd{plot} environment is a way to pass information to
\app{Gnuplot} in a more structured way, so that customization of basic
plots becomes easier. It has the following characteristics:

The block starts with the \cmd{plot} keyword, followed by a required
parameter: the name of a list, a single series or a matrix. This
parameter specifies the data to be plotted. The starting line may be
prefixed with the \verb|savename <-| apparatus to save a plot as an icon
in the GUI program. The block ends with \cmd{end plot}.

Inside the block you have zero or more lines of these types, identified 
by an initial keyword:
\begin{description}
\item[\normalfont \texttt{option}:] specify a single option (details below)
\item[\normalfont \texttt{options}:] specify multiple options on a single line; if
  more than one option is given on a line, the options should be
  separated by spaces.
\item[\normalfont \texttt{literal}:] a command to be passed to gnuplot literally 
\item[\normalfont \texttt{printf}:] a printf statement whose result will be passed
  to gnuplot literally; this allows the use of string variables
  without having to resort to \verb!@!-style string substitution.
\end{description}

The options available are basically those of the current \cmd{gnuplot} 
command, but with a few differences. For one thing you don't need the 
leading double-dash in an "option" (or "options") line. Besides that,
\begin{itemize}
\item You can't use the option \option{matrix=whatever} with \cmd{plot}:
  that possibility is handled by providing the name of a matrix on the
  initial \cmd{plot} line.
\item The \option{input=filename} option is not supported: use
  \cmd{gnuplot} for the case where you're supplying the entire plot
  specification yourself.
\item The several options pertaining to the presence and type of a
  fitted line, are replaced in \cmd{plot} by a single option \cmd{fit} which
  requires a parameter. Supported values for the parameter are: none,
  linear, quadratic, cubic, inverse, semilog and loess. Example:
\begin{code}
  option fit=quadratic
\end{code}
\end{itemize}

As with \cmd{gnuplot}, the default is to show a linear fit in an X-Y
scatter if it's significant at the 10 percent level.

Here's a simple example, the plot specification from the ``bandplot''
package, which shows how to achieve the same result via the
\cmd{gnuplot} command and a \cmd{plot} block, respectively---the
latter occupies a few more lines but is clearer

\begin{code}
   gnuplot 1 2 3 4 --with-lines --matrix=plotmat \
   --fit=none --output=display \
   { set linetype 3 lc rgb "#0000ff"; set title "@title"; \
     set nokey; set xlabel "@xname"; }
\end{code}

\begin{code}
   plot plotmat
     options with-lines fit=none
     literal set linetype 3 lc rgb "#0000ff"
     literal set nokey
     printf "set title \"%s\"", title
     printf "set xlabel \"%s\"", xname
   end plot --output=display
\end{code}

Note that \option{output=display} is appended to \cmd{end plot}; also
note that if you give a matrix to \cmd{plot} it's assumed you want to
plot all the columns. In addition, if you give a single series and the
dataset is time series, it's assumed you want a time-series plot.

\subsubsection{Example: Plotting an histogram together with a density}

\begin{script}[htbp]
  \scriptinfo{mroz-logwage}{Plotting the log wage from the Mroz example dataset}
\begin{scode}
set verbose off
open mroz87.gdt --quiet

series lWW = log(WW)
scalar m = mean(lWW)
scalar s = sd(lWW)

###
### prepare matrix with data for plot
###

# number of valid observations
scalar n = nobs(lWW)
# discretize log wage
scalar k = 4
series disc_lWW = round(lWW*k)/k
# get frequencies
matrix f = aggregate(null, disc_lWW)
# add density
phi = dnorm((f[,1] - m)/s) / (s*k)
# put columns together and add labels
plotmat = f[,2]./n ~ phi  ~ f[,1]
strings cnames = defarray("frequency", "density", "log wage")
cnameset(plotmat, cnames)

###
### create plot
###

plot plotmat
    # move legend
    literal set key outside rmargin
    # set line style
    literal set linetype 2 dashtype 2 linewidth 2
    # set histogram color
    literal set linetype 1 lc rgb "#777777"
    # set histogram style
    literal set style fill solid 0.25 border
    # set histogram width
    printf "set boxwidth %4.2f\n", 0.5/k
    options with-lines=2 with-boxes=1
end plot --output=display
\end{scode}
\end{script}

Listing \ref{ex:mroz-logwage} contains a slightly more elaborate
example: here we load the Mroz example dataset and calculate the log
of the individual's wage. Then, we match the histogram of a
discretized version of the same variable (obtained via the
\cmd{aggregate()} function) versus the theoretical density if data
were Gaussian.

There are a few points to note:
\begin{itemize}
\item The data for the plot are passed through a matrix in which we
  set column names via the \cmd{cnameset} function; those names are
  then automatically used by the \cmd{plot} environment.
\item In this example, we make extensive use of the \cmd{set literal}
  construct for refining the plot by passing instruction to
  \app{gnuplot}; the power of \app{gnuplot} is impossible to
  overstate. We encourage you to visit the ``demos'' version of
  \app{gnuplot}'s website (\url{http://gnuplot.sourceforge.net/}) and
  revel in amazement.
\item In the \cmd{plot} environment you can use all the quantities you
  have in your script. This is the way we calibrate the histogram
  width (try setting the scalar \verb|k| in the script to different
  values). Note that the \cmd{printf} command has a special meaning
  inside a \cmd{plot} environment.
\item The script displays the plot on your screen. If you want to save
  it to a file instead, replace \verb!--output=display! at the end
  with \texttt{-{}-output=\textsl{filename}}.
\item It's OK to insert comments in the \cmd{plot} environment;
  actually, it's a rather good idea to comment as much as possible (as
  always)!
\end{itemize}
The output from the script is shown in Figure \ref{fig:mroz-logwage}.


\begin{figure}[htbp]
  \centering
  \includegraphics{figures/Mroz-logwage}
  \caption{Output from listing \ref{ex:mroz-logwage}}
  \label{fig:mroz-logwage}
\end{figure}

\subsubsection{Example: Plotting Student's $t$ densities}

\begin{script}[htbp]
  \scriptinfo{student_plot}{Plotting $t$ densities for varying degrees of freedom}
\begin{scode}
set verbose off

function string tplot(scalar m)
    return sprintf("stud(x,%d) title \"t(%d)\"", m, m)
end function

matrix dfs = {2, 4, 16}

plot
    literal set xrange [-4.5:4.5]
    literal set yrange [0:0.45]
    literal Binv(p,q) = exp(lgamma(p+q)-lgamma(p)-lgamma(q))
    literal stud(x,m) = Binv(0.5*m,0.5)/sqrt(m)*(1.0+(x*x)/m)**(-0.5*(m+1.0))
    printf "plot %s, %s, %s", tplot(dfs[1]), tplot(dfs[2]), tplot(dfs[3])
end plot --output=display
\end{scode}
\end{script}

The power of the \cmd{printf} statement in a \cmd{plot} block becomes
apparent when used jointly with user-defined functions, as exemplified
in Listing \ref{ex:student_plot}, in which we create a plot showing
the density functions of Student's $t$ distribution for three
different settings of the ``degrees of freedom'' parameter (note that
plotting a $t$ density is very easy to do from the GUI: just go to the
\emph{Tools $>$ Distribution graphs} menu).

First we define a user function called \texttt{tplot}, which returns a
string with the ingredients to pass to the gnuplot \texttt{plot}
statement, as a function of a scalar parameter (the degrees of freedom
in our case). Next, this function is used within the \cmd{plot} block
to plot the appropriate density. Note that most of the 
statements to mathematically define the function to plot are
outsourced to gnuplot via the \cmd{literal} command.

The output from the script is shown in Figure \ref{fig:StudentPlot}.

\begin{figure}[htbp]
  \centering
  \includegraphics{figures/StudentPlot}
  \caption{Output from listing \ref{ex:student_plot}}
  \label{fig:StudentPlot}
\end{figure}


\pagebreak[4]

\section{Boxplots}
\label{sect-boxplots}

These plots (after Tukey and Chambers) display the distribution of a
variable. Its shape depends on a few quantities, defined as follows:

\begin{center}
\begin{tabular}{rl}
  $x_{\mathrm{min}}$ & sample minimum \\
  $Q_1$ & first quartile \\
  $m$ & median \\
  $\bar{x}$ & mean \\
  $Q_3$ & third quartile \\
  $x_{\mathrm{max}}$ & sample maximum\\
  $R = Q_3 - Q_1$ & interquartile range\\
\end{tabular}
\end{center}

The central box encloses the middle 50 percent of the data, i.e.\ goes
from $Q_1$ to $Q_3$; therefore, its height equals $R$.  A line
is drawn across the box at the median $m$ and a ``\texttt{+}'' sign
identifies the mean $\bar{x}$.

The length of the ``whiskers'' depends on the presence of
outliers. The top whisker extends from the top of the box up to a
maximum of 1.5 times the interquartile range, but can be shorter if
the sample maximum is lower than that value; that is, it reaches
$\min[x_{\mathrm{max}}, Q_3 + 1.5 R]$. Observations larger than
$Q_3 + 1.5 R$, if any, are considered outliers and represented
individually via dots.\footnote{To give you an intuitive idea, if a
  variable is normally distributed, the chances of picking an outlier
  by this definition are slightly below 0.7\%.} The bottom whisker
obeys the same logic, with obvious adjustments.
Figure~\ref{fig-boxplot} provides an example of all this by using the
variable \texttt{FAMINC} from the sample dataset \texttt{mroz87}.

\begin{figure}[htbp]
  \begin{flushleft}
    \hspace{1cm}
  \input{figures/boxplot_sample}
  \end{flushleft}
  \caption{Sample boxplot}
  \label{fig-boxplot}
\end{figure}

In the case of boxplots with confidence intervals, dotted lines show
the limits of an approximate 90 percent confidence interval for the
median.  This is obtained by the bootstrap method, which can take a
while if the data series is very long. For details on constructing
boxplots, see the entry for \cmd{boxplot} in the \GCR\, or use the
\textsf{Help} button that appears when you select one of the boxplot
items under the menu item ``View, Graph specified vars'' in the main
gretl window.

\subsection{Factorized boxplots}

A nice feature which is quite useful for data visualization is the
conditional, or factorized boxplot.  This type of plot allows you to
examine the distribution of a variable conditional on the value of
some discrete factor.

As an example, we'll use one of the datasets supplied with
gretl, that is \cmd{rac3d}, which contains an example taken from
\cite{cameron-trivedi13} on the health conditions of 5190 people. The
script below compares the unconditional (marginal) distribution of the
number of illnesses in the past 2 weeks with the distribution of the
same variable, conditional on age classes.

\begin{scode}
open rac3d.gdt
# unconditional boxplot
boxplot ILLNESS --output=display
# create a discrete variable for age class: 
# 0 = below 20, 1 = between 20 and 39, etc
series age_class = floor(AGE/0.2)
# conditional boxplot
boxplot ILLNESS age_class --factorized --output=display
\end{scode}

After running the code above, you should see two graphs similar to
Figure \ref{fig:fact-boxplots}. By comparing the marginal plot to
the factorized one, the effect of age on the mean number of illnesses
is quite evident: by joining the green crosses you get what is
technically known as the conditional mean function, or regression
function if you prefer.

\begin{figure}[htbp]
  \centering
  \begin{tabular}{cc}
    \includegraphics[width=0.475\textwidth]{figures/uboxplot} & 
    \includegraphics[width=0.475\textwidth]{figures/fboxplot}
  \end{tabular}
  \caption{Conditional and unconditional distribution of illnesses}
  \label{fig:fact-boxplots}
\end{figure}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

