Up : MAXI Data Analysis Next : Data Analysis

Preparation of data analysis (before HEAsoft 6.25 release)

This page is a step-by-step guide of software install and settings for MAXI archive data analysis. We have confirmed this MAXI software package would work in Linux CentOS Vers.5, 6, and 7. It may work in other Linux distributions, too, but we can not give a guarantee at the moment. We are working to support more platforms.

The MAXI package can be installed in a normal user's directory without any administrator's privilege, in principle. However, it requires some modification in the configuration files of the two external packages, HEAsoft and CALDB. If these packages you use in your system have been installed by an administrator, you may need ask your administrator to modify their configuration files as explained below (Section 2). The MAXI team and NASA/GSFC are now working to implement the MAXI FTOOLS in the official FTOOLS package, and hopefully it will happen soon.

Contents

The MAXI team
Please send questions, comments, and bug reports to : maxihelp@ml.ac.jaxa.jp

1. External software packages required

The following external software packages are required, before the MAXI-specific software is installed and run. Once these have been installed, set up the relevant environmental variables and run the relevant initial-setup scripts. We recommend the installation be made in the following order due to the dependency issue.

  1. HEAsoft of HEASARC (Version 6.20 as in March 2017; NASA)
    https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/

    Note: At "STEP 2 - Download the desired packages" in installation of HEAsoft, you are asked to select which packages are downloaded and installed. Make sure to include Mission-Specific Tools for Suzaku, ASCA, and Swift, General-Use FTOOLS, and XANADU at least.

  2. CALDB for HEASARC (NASA)
    https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_install.html

In addition to the HEAsoft building environment, especially CC (gcc), make, and Perl 5 (see HEAsoft installation guide for detail), the MAXI software package requires the computer languages of Python 2 and Ruby 2 (At the time of writing, the default version of Ruby installed on Cent OS Ver.7 is 2, whereas those in Vers.5 and 6 are still 1.8, which is not compatible with one of the MAXI software). Also, you may need install OpenSSL and rebuild openssl library in Ruby, if OpenSSL has not been installed in your environment. OpenSSL is (one of) the standard library to access websites like the MAXI website via secure connection (SSL).

2. Implementing the MAXI part to HEAsoft

The MAXI software package is distributed in the source-code format only (no binary distribution). You need up to 80 MB of free disk space to install it.

The environmental variable HEADAS should have been set (part of the HEAsoft setting), which points to the top directory of HEAsoft. Append the following in the HEAsoft file $HEADAS/bin/xselect.mdb (prior to Section 4. “Installation and set-up of the MAXI software package”): 

MAXI:submkey       NONE
MAXI:instkey       INSTRUME
MAXI:dmodekey      DATAMODE
MAXI:mkf_def_expr  mx*.mkf*
MAXI:mkf_rel_dir   .
MAXI:time          TIME
MAXI:tunits        s
MAXI:binsize       16.
MAXI:x             X
MAXI:y             Y
MAXI:xsiz          TLMAX
MAXI:ysiz          TLMAX
MAXI:detx          DETX
MAXI:dety          DETY
MAXI:detxsiz       TLMAX
MAXI:detysiz       TLMAX
MAXI:rawx          NONE
MAXI:rawy          NONE
MAXI:rawxsiz       NONE
MAXI:rawysiz       NONE
MAXI:phamax        TLMAX
MAXI:gti           STDGTI
MAXI:events        EVENTS
MAXI:timeorder     yes
MAXI:instruments   GSC GSC_0 GSC_1 GSC_2 GSC_3 GSC_4 GSC_5GSC_6 GSC_7 GSC_8 GSC_9 GSC_A GSC_B SSC SSC_H SSC_Z
MAXI:spbn          1
MAXI:ecol          PI
MAXI:ccol          NONE
MAXI:gcol          NONE
MAXI:imagecoord    SKY
MAXI:wmapcoord     DETECTOR
MAXI:catnum        1
MAXI:wtmapb        yes
MAXI:wtmapfix      no
MAXI:lststr        *.evt
MAXI:fbin          1
MAXI:hbin          1
MAXI:GSC:modes     64BIT 32BIT 16BIT
MAXI:SSC:modes     64BIT
MAXI:extract       extractor
MAXI:catcol        OBJECT TELESCOP INSTRUME DATAMODE DATE-OBS DATE-END TIME-OBS
MAXI:dispcol       OBJECT DATE-OBS DATE-END TIME-OBS
MAXI:extendresp    yes

3. Implementing the MAXI part to CALDB

The environmental variable CALDB should have been set (part of the CALDB setting). Download the two MAXI-CALDB files (caldb_maxi_gsc_20170614.tar.gz, caldb_maxi_ssc_20170324.tar.gz) from https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/ and put them in your CALDB directory (prior to Step 4. “Installation and set-up of the MAXI software package”). The following is an example of how to do it from a terminal: 

 cd $CALDB
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/caldb_maxi_gsc_20170614.tar.gz
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/caldb_maxi_ssc_20170324.tar.gz
 tar zxvf caldb_maxi_gsc_20170614.tar.gz
 tar zxvf caldb_maxi_ssc_20170324.tar.gz

Also, add the following in one of the CALDB configuration files: $CALDB/software/tools/caldb.config 

#
# MAXI
#
MAXI GSC   CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_0 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_1 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_2 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_3 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_4 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_5 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_6 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_7 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_8 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_9 CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_A CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
MAXI GSC_B CALDB data/maxi/gsc caldb.indx CALDB data/maxi/gsc
#
MAXI SSC   CALDB data/maxi/ssc caldb.indx CALDB data/maxi/ssc
MAXI SSC_H CALDB data/maxi/ssc caldb.indx CALDB data/maxi/ssc
MAXI SSC_Z CALDB data/maxi/ssc caldb.indx CALDB data/maxi/ssc

Run caldbinfo in FTOOLS, as follows, to check if they have been successfully implemented: 

 caldbinfo INST MAXI GSC
   ** caldbinfo 1.0.2
   ... Local CALDB appears to be set-up & accessible
   ** caldbinfo 1.0.2 completed successfully

4. Installation and set-up of the MAXI software package

The MAXI software consists of the main package (MAXI FTOOLS) (Section 4.1), which needs compiling, and an auxiliary Ruby script, mxdownload (Section 4.2).

4.1. Installation of the main MAXI software package (MAXI FTOOLS)

The installation of MAXI FTOOLS has been tested on Cent OS 5, 6, and 7 of Linux.

4.1.1. Downloading MAXI FTOOLS

MAXI FTOOLS consists of four separate tar-archive files:

  1. maxitask-1.0.17.tar.gz
  2. maxitool-4.0.4.tar.gz
  3. perl5-Maxitool-2.1.0.tar.gz
  4. mxproduct-1.1.0.tar.gz

Fetch them from the same URI with username/password as described in the CALDB setting-up section, and put them in your chosen top directory for MAXI FTOOLS. The following is an example of how to do it: 

 cd ((Your-MAXI-FTOOLS-Top-Directory))
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/maxitask-1.0.17.tar.gz
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/maxitool-4.0.4.tar.gz
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/perl5-Maxitool-2.1.0.tar.gz
 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/mxproduct-1.1.0.tar.gz
4.1.2. Setup of the environmental variables to build the package

Your HEAsoft environment, including the environmental variable HEADAS, must be set up first. Then, set the environmental variables MXFTOOLS to point to your chosen top directory for MAXI FTOOLS, and modify the two environmental variables of LD_LIBRARY_PATH and PERL5LIB, as follows:

in B-sh (bash, zsh)
 export MXFTOOLS=/data/opt/maxi/mxftools # Your chosen top directory for MAXI FTOOLS.
 export LD_LIBRARY_PATH=${MXFTOOLS}/lib:$LD_LIBRARY_PATH
 export PERL5LIB=$MXFTOOLS/lib/perl5:$PERL5LIB
in C-sh (tcsh)
 setenv MXFTOOLS /data/opt/maxi/mxftools # Your chosen top directory for MAXI FTOOLS.
 setenv LD_LIBRARY_PATH ${MXFTOOLS}/lib:$LD_LIBRARY_PATH
 setenv PERL5LIB $MXFTOOLS/lib/perl5:$PERL5LIB
4.1.3. Compiling and Installing

Uncompress all the MAXI FTOOLS packages at the top directory $MXFTOOLS as follows, 

 cd $MXFTOOLS
 tar zxvf maxitool-4.0.4.tar.gz
 tar zxvf maxitask-1.0.17.tar.gz
 tar zxvf perl5-Maxitool-2.1.0.tar.gz
 tar zxvf mxproduct-1.1.0.tar.gz

and follow the procedure of installation of each package in the following order.

Install maxitool (Library collection)
in B-sh
 cd $MXFTOOLS/maxitool-4.0.4
 ./configure --prefix=${MXFTOOLS} 
 make install > install.log 2>&1 &
in C-sh
 cd $MXFTOOLS/maxitool-4.0.4
 ./configure --prefix=${MXFTOOLS} 
 make install >& install.log &

Several files should have been installed in the directories $MXFTOOLS/lib and $MXFTOOLS/include now. Check if they are.

Install maxitask (Task collection)
in B-sh
 cd $MXFTOOLS/maxitask-1.0.17
 ./configure --prefix=${MXFTOOLS} --with-maxitool=${MXFTOOLS}
 make install > install.log 2>&1 &
in C-sh
 cd $MXFTOOLS/maxitask-1.0.17
 ./configure --prefix=${MXFTOOLS} --with-maxitool=${MXFTOOLS}
 make install >& install.log &

Several files should have been installed in the directory $MXFTOOLS/bin now. Check if they are.

Install perl5-Maxitool (Perl task collection)

The procedure differs slightly between the systems of Cent OS Ver.6 & 7 and Ver.5. 

 cd $MXFTOOLS/perl5-Maxitool-2.1.0
in both B-sh and C-sh
in Cent OS 6 and 7
 perl Makefile.PL INSTALL_BASE=${MXFTOOLS} MAXITOOL=${MXFTOOLS}
in Cent OS 5
 perl Makefile.PL PREFIX=${MXFTOOLS} MAXITOOL=${MXFTOOLS}
in B-sh
 make > make.log 2>&1 &
 make install > install.log 2>&1 &
in C-sh
 make >& make.log &
 make install >& install.log &

The file $MXFTOOLS/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Maxitool.pm should have been generated now (Note the actual path may be dependent on the OS and environment). Check if it is.

Furthermore, the following setup is required before you proceed to install mxproduct (see the next subsection), if your system is Cent OS 5.

only in Cent OS 5
NOTE: The part of the path, lib64, in the following example is dependent on your environment, and can be, specifically, lib. Please adjust them accordingly (A tip: you can run perl -V:sitearch and see what the architecture of your environment is).
in B-sh
 export PERL5LIB=$MXFTOOLS/lib64/perl5/site_perl:$PERL5LIB
in C-sh
 setenv PERL5LIB $MXFTOOLS/lib64/perl5/site_perl:$PERL5LIB
Install mxproduct (Pipe-line command)

Prior to this step, make sure the environmental variable PERL5LIB is correctly set, especially if your system is Cent OS 5 (see the previous subsection).

in B-sh
 cd $MXFTOOLS/mxproduct-1.1.0
 ./configure --prefix=$MXFTOOLS
 make > make.log 2>&1 &
 make install > install.log 2>&1 &
in C-sh
 cd $MXFTOOLS/mxproduct-1.1.0
 ./configure --prefix=$MXFTOOLS
 make >& make.log &
 make install >& install.log &

The executable file $MXFTOOLS/bin/mxproduct should have been generated now. Check if it is.

4.2. Installation of the auxiliary command mxdownload

Download a helper tool, mxdownload, written in Ruby, from 

 wget https://darts.isas.jaxa.jp/pub/maxi/mxftools/20171002/mxdownload

Put mxdownload in one of your command-line search paths, such as /usr/local/bin/ and $HOME/bin/. You may need to execute a shell command rehash, or to rerun the init script for your shell environment, if you want to activate it immediately (or you may simply open a new terminal).

Note that mxdownload is an independent and stand-alone command, and hence to put it in one of your command-search paths is not a requirement by MAXI FTOOLS. Instead, you can put it anywhere and call it with the absolute (or relative) path, if you prefer.

Alternatively, you can choose to download the entire MAXI archive in your local disk, using the following command (or similar). The total data size is roughly 1.4TB as in February 2017. 

 wget -q -nH --cut-dirs=1 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks \
 https://darts.isas.jaxa.jp/pub/maxi/mxdata/obs/

4.3. Setting up the MAXI software package

You need to set some environmental variables to use MAXI FTOOLS, in addition to those required for the external packages, such as HEADAS. You may want to write the setting at the end of your shell start-up file, which is typically, ~/.bashrc for bash, ~/.profile for (login-shell of) Bourne Shell variants (sh, ksh, ash), or ~/.cshrc for C-shell variants (csh, tcsh).

The MAXI-specific settings are as follows (the PERL5LIB paths may be different for Cent OS 5; see the description above).

in B-sh
 export MXFTOOLS=((Installed-Directory))
 export PATH="${MXFTOOLS}/bin:$PATH"
 export LD_LIBRARY_PATH="${MXFTOOLS}/lib:$LD_LIBRARY_PATH"
 export PFILES="$HOME/pfiles;${MXFTOOLS}/syspfiles:$HEADAS/syspfiles"
 export PERL5LIB="${MXFTOOLS}/lib/perl5:$PERL5LIB"
in C-sh
 setenv MXFTOOLS ((Installed-Directory))
 set path = (${MXFTOOLS}/bin $path)
 setenv LD_LIBRARY_PATH "${MXFTOOLS}/lib:$LD_LIBRARY_PATH"
 setenv PFILES "$HOME/pfiles;${MXFTOOLS}/syspfiles:$HEADAS/syspfiles"
 setenv PERL5LIB "${MXFTOOLS}/lib/perl5:$PERL5LIB"

In the above examples, ((Installed-Directory)) is the place holder of the directory in which the MAXI package is installed.

In summary, your set-up script should be more or less similar to the following (adjust them according to your environment; in particular, the PERL5LIB paths may be different for Cent OS 5; see the description above):

B-sh (bash, zsh)
## Set-up HEAsoft
export HEADAS=$HOME/ASTRO/heasoft-6.20/x86_64-unknown-linux-gnu-libc2.12
. $HEADAS/headas-init.sh

## Set-up CALDB
export CALDB=$HOME/ASTRO/caldb
export CALDBCONFIG=$CALDB/software/tools/caldb.config
export CALDBALIAS=$CALDB/software/tools/alias_config.fits

## Set-up MAXI FTOOLS
export MXFTOOLS=$HOME/ASTRO/maxi/installed
export PATH=${MXFTOOLS}/bin:$PATH
export LD_LIBRARY_PATH="${MXFTOOLS}/lib:$LD_LIBRARY_PATH"
export PFILES="$HOME/pfiles;${MXFTOOLS}/syspfiles:$HEADAS/syspfiles"
export PERL5LIB="$MXFTOOLS/lib/perl5:$PERL5LIB"
C-sh (tcsh)
## Set-up HEAsoft
setenv HEADAS $HOME/ASTRO/heasoft-6.20/x86_64-unknown-linux-gnu-libc2.12
source $HEADAS/headas-init.sh

## Set-up CALDB
setenv CALDB $HOME/ASTRO/caldb
setenv CALDBCONFIG $CALDB/software/tools/caldb.config
setenv CALDBALIAS $CALDB/software/tools/alias_config.fits

## Set-up MAXI FTOOLS
setenv MXFTOOLS $HOME/ASTRO/maxi/installed
set path = (${MXFTOOLS}/bin $path)
setenv LD_LIBRARY_PATH "${MXFTOOLS}/lib:$LD_LIBRARY_PATH"
setenv PFILES "$HOME/pfiles;${MXFTOOLS}/syspfiles:$HEADAS/syspfiles"
setenv PERL5LIB "$MXFTOOLS/lib/perl5:$PERL5LIB"

5. Uninstall

To uninstall the MAXI FTOOLS, simply delete the directory of $MXFTOOLS. The MAXI-specific parts in the configuration files of the external packages (HEAsoft and CALDB) can be left as they are. They would do no harm.

Go to Next : Data Analysis