Make your HiPS in 10 steps

(last update Jan 2017 - additions in green)

... or how to build a Hierarchical Progressive Survey from your image (or cube) collection.

  1. The observations do not fully cover the whole image arrays (ex: HST, Chandra):
    • If your image contains not exposed pixels, check if the BLANK FITS keyword is correctly set to the value of these pixels. If it is not the case, specify it via the parameter blank=xx (generally blank=0). Notice that the BLANK value is expressed as the FITS pixel coding (BSCALE and BZERO not applied), and not in physical quantity.
    • Check if there are bad pixels at the borders of the image. You can remove them via the parameter border=nn.
    • If the observation is circular or rectangular (ex: GALEX), you can specify it via the parameter shape=sss
    • If the observation is complex and can be discribed by a polygon (ex: Schmidt plate), you can specify a polygon, either associated to a specifical image (same name, .fov extension), or for a collection of images set together in a sub-directory - the file describing the polygon must have the same name of the directory, .fov extension. You must use fov=true parameter to take into account these polygon files.
      Tips:Use Aladin desktop with the tool Tag for generating the list of coordinates. Cut-&-paste the result in a clip board for keeping only the RA and DEC columns (one per line).
  2. The sky back grounds of the observations are not homogeneous:
    • If the sky back ground value is specified in the FITS header, you can indicate the corresponding FITS key to Aladin/Hipsgen via the parameter skyval=XXXX (for instance skyval=BACKGRND)
    • Otherwise, you can try with skyval=true for asking Aladin/Hipsgen to compute an estimation of each sky back ground.
  3. The original images are packaged in Multi-Extension FITS:
    Specify the good extension via the parameter hdu=nn, or all hdu=all.
  4. You would like to modify the BITPIX, notably if the original images are coded in a surprising bit range (ex: -64 for counting a few photons...) and/or you would like to minimize the HiPS size:
    • Choose a good target BITPIX (ex: 16 means 2^16 = 65536 grey levels typically enough for most of detectors)
    • Specify this target BITPIX via the parameter bitpix=nn.
    • Determine manually the physical range of original valid pixels and specify them thanks to pixelRange="min max" parameter. Otherwise Aladin/Hipsgen will do an automatical estimation on the first image of the collection, or on a specifical image specified by img="path" parameter. The min and max values are expressed in physical quantities (BSCALE, BZERO applied).
  5. Your image collection are in JPEG or PNG, and not in FITS:
    • For each image, you must provided the associated astrometrical calibration (WCS solution) either in a separated file (same name, .hhh extension) or directly inside the JPEG (resp. PNG) comment segment. The WCS solution can be expressed as a FITS header (80 character alignement), or with basic ASCII line separators.
    • Specify the resulting tile format via the parameter color=jpeg or color=png (or color=true to keep the same format that the original images).
    Tips: If your JPEG or PNG images are quite huge (>100 Megapixels), do not hesitate to split your images in sub-images of 1024x1024 pixels. The process will be really faster. Do not forget to adjust the WCS solutions according to.
  6. Your image is a HEALPix fits map:
    In this case, your original data will be not resampled, but just mapped into HiPS tiles. You have to specify the filename of the HEALPix map instead of the input directory:
    java -Xmx2000m -jar Aladin.jar in="YourHealpixMap.fits
  7. You would like to restart partially the process:
    • You can specify on which region you want to restart the process. Use the parameter moc=sss where sss is a inline ASCII MOC or a path to a MOC file.
      Have a look on mode=xxx parameter for controlling the pixels and tiles merging/replacing behaviour...
    • You can also launch Aladin/Hipsgen step by step by specifying explicitely each action. The default action list is INDEX TILES GZIP PNG DETAILS. You can restart one specifical action, for instance relaunch PNG tile computation
      java -jar AladinBeta.jar -hipsgen in=Data PNG
  8. You would like to generate a colored HiPS combining several survey filters (2 or 3):
    • For each filter, you have to create before the grey level HiPS.
      → we assume "redHips", "greenHips" and "blueHips" paths.
    • After that, launch Aladin/Hipsgen RGB action with these parameters:
      java -jar AladinBeta.jar -hipsgen inRed=redHips inGreen=greenHips inBlue=blueHips RGB
      You can adjust the color mapping scale for each color component thanks to cmRed, cmBlue and cmBlue parameters (see details below)
  9. You would like to generate your HiPS incrementally
    This facility is useful for incorporating new observations in an already existing HiPS without restarting all the process (or if you have no longer access to the original images). This option is only supported int Fits mode.
    • Your original HiPS had to be generated with the -live parameter
      java -jar AladinBeta.jar -hipsgen -live in=... out=DataHiPS ...
    • With your new observations, generate a new separated HiPS also using -live parameter:
      java -jar AladinBeta.jar -hipsgen -live in=... out=add1HiPS ...
    • Concatenate your new HiPS with the original one thanks to CONCAT action:
      java -jar AladinBeta.jar -hipsgen in=add1HiPS out=DataHips CONCAT
    • That's done. You can remove your separated HiPS (rm -Rf add1HiPS)
  10. You would like to patch (or regenerate) a HiPS on a specific region
    You can restrict the HiPS target size thanks to the region=moc parameter. This "MOC" can be described via an inline string, or via a real MOC file.
    Associated to the appropriated mode parameter, notably the value OVERWRITE (pixel replacement), or REPLACETILE (tile replacement), you will have the full control to decide how to patch an already existing HiPS.
    java -jar AladinBeta.jar -hipsgen in=Data region="3/131,132 4/565" mode=REPLACETILE ...
    Note: For using this method, you need to have access to all the original images ("in" directory must be unchanged). If you want to add new observations see the previous item)
  11. You would like to generate the progenitors links:
    This facility generates links for providing an access to the original images.
    • Launch Aladin/Hipsgen DETAILS action if required (already done by default)
    • Edit the file DataHiPS/HpxFinder/metadata.xml and follow the inside explanations, notably for re-incorporating the metadata memorized thanks to fitskey="xxx" parameter.

  1. If the images are relatively small (< 3 Megapixels) and the collection does not have a lot of overlays (<6 images on the same location), set the parameter: partitioning=false.
  2. If the images have already homogeneous sky back ground, set the parameter: mixing=false.
  3. If your have to generate JPEG or PNG tiles:
    • JPEG is 4 to 20 times faster than PNG (but does not manage transparency and introduces artefacts).
    • PNG encoder time can vary by a factor 10 depending of the sky background noise. More you remove the sky background noise by adjusting the min pixel cut, faster the PNG encoder will work.
  4. For checking the result before the end it is possible to have a quick look of the current result:
    1. Stop Aladin/Hipsgen: kill -STOP pid, or CTRL+Z
    2. Launch this command: java -jar AladinBeta.jar -hipsgen in=Data ALLSKY
    3. Look the current result: java -jar AladinBeta.jar DataHiPS
    4. Restart the process: kill -CONT pid, or fg

Usage: java -jar Aladin.jar -hipsgen in=file|dir [otherParams ... ACTIONs ...]
       java -jar Aladin.jar -hipsgen -param=configfile

The config file must contain these following options, or use them
directly on the comand line :

Required parameter:
   in=dir             Source image directory (FITS or JPEG|PNG +hhh or HiPS),
                      unique image or HEALPix map file

Basic optional parameters:
   out=dir            HiPS target directory (default $PWD+"HiPS")
   label=name         Label of the survey (by default, input directory name)
   id=identifier      HiPS identifier (syntax: AUTHORITY-ID/internalID)
   creator=name       Name of the person|institute who builds the HiPS
   status=xx          HiPS status (private|public clonable|clonableOnce|unclonable)
                      (default: public clonableOnce)  
   hdu=n1,n2-n3,...|all  List of HDU numbers (0 is the primary HDU - default is 0)
   blank=nn           Specifical BLANK value
   skyval=key|true|%info|%min %max   Fits key to use for removing a sky background,
                      true auto detection or percents of pixel histogram kept
		      (central ex 99, or min max ex 0.3 99.7)
   color=jpeg|png     The source images are colored images (jpg or png) and the tiles
                      will be produced in jpeg (resp. png)
   shape=...          Shape of the observations (ellipse|rectangle)
   border=...         Margins (in pixels) to ignore in the original observations
                      (N W S E or constant)
   fov=true|x1,y1..   Observed regions by files.fov or global polygon.
   verbose=n          Debug information from -1 (nothing) to 4 (a lot)
   -f                 clear previous computations
   -n                 Just print process information, but do not execute it.

Advanced optional parameters:
   order=nn           Specifical HEALPix order - by default, adapted to
                      the original resolution
   bitpix=nn          Specifical target bitpix (-64|-32|8|16|32|64)
   pixelCut=min max   Specifical pixel cut and/or transfert function for PNG/JPEG 8 bits
                      conversion - ex: "120 140 log")
   pixelRange=min max Specifical pixel value range (required for bitpix
                      conversion, or for removing bad pixels - ex: "-5 110")
   pixelGood=min [max] Range of pixel values kept
   img=file           Specifical reference image for default initializations 
   mode=xx            Coadd mode when restart: pixel level(OVERWRITE|KEEP|ADD|AVERAGE) 
                      or tile level (REPLACETILE|KEEPTILE) - (default OVERWRITE)
                      Or LINK|COPY for CUBE action (default COPY)
   fading=true|false  False to avoid fading effect on overlapping original images 
                      (default is true)
   mixing=true|false  False to avoid mixing (and fading) effect on overlapping
                      original images (default is true)
   partitioning=true|false True for cutting large original images in blocks
                      of 1024x1024 (default is true)
   region=moc         Specifical HEALPix region to compute (ex: 3/34-38 50 53)
                      or Moc.fits file (all sky by default)
   maxRatio=nn        Max height/width pixel ratio tolerated for original obs 
                      (default 2, 0 for removing the test)
   fitskeys=list      Fits key list (blank separator) designing metadata FITS keyword 
                      value  to memorized in the HiPS index
   minOrder=nn        Specifical HEALPix min order (only for DETAILS action)
   method=m           Method (MEDIAN|MEAN|FIRST) (default MEDIAN) for aggregating 
                      colored compressed tiles (JPEG|PNG)
   tileOrder=nn       Specifical tile order - default 9
   mocOrder=nn        Specifical HEALPix MOC order (only for MOC action) - by default 
                      auto-adapted to the HiPS
   exptime=key        Fits key to use for adjusting variation of exposition
   inRed              HiPS red path component (RGB action)
   inGreen            HiPS green path component (RGB action)
   inBlue             HiPS blue path component (RGB action)
   cmRed              Colormap parameters for HiPS red component (min [mid] max [fct])
   cmGreen            Colormap parameters for HiPS green component (min [mid] max [fct])
   cmBlue             Colormap parameters for HiPS blue component (min [mid] max [fct])
   tileTypes          List of tile format to copy (MIRROR action)
   maxThread=nn       Max number of computing threads
   target=ra +dec     Default HiPS target (ICRS deg)
   targetRadius=rad   Default HiPS radius view (deg)
   -nice              Slow download for avoiding to overload remote http server
                      (dedicated to MIRROR action)

Specifical actions (by default: "INDEX TILES PNG GZIP DETAILS"):
   INDEX      Build index (in HpxFinder directory) + MOC index
   TILES      Build all true value pixel tiles (FITS) + Allsky.fits + MOC
   JPEG       Build all preview tiles (JPEG) + Allsky.jpg
   PNG        Build all preview tiles (PNG) + Allsky.png
   RGB        Build and RGB HiPS based on 2 or 3 other HiPS
   MOC        (Re)build the MOC (MultiOrder Coverage map)
   ALLSKY     (Re)build all Allsky files + index.html
   TREE       (Re)build HiPS tree structure from already existing tiles
   MAPTILES   Build all FITS tiles from a HEALPix Fits map
   CONCAT     Concatenate one HiPS to another HiPS
   CUBE       Create a HiPS cube based on a list of HiPS
   GZIP       Compress some FITS tiles and Allsky.fits
   CLEANFITS  Delete all FITS tiles and Allsky.fits
   DETAILS    Adapt HiPS index for supporting the "detail table" facility
   MIRROR     Mirror a remote HiPS locally

(c) Unistra/CNRS 2016 - Aladin.jar -hipsgen based on Aladin v9.017 from CDS