HMI_Logo HMI Ring Diagrams
Pipeline Module Specifications

datavg

Module Version
	1.6	2 X 2017

General Description
	datavg takes as its input a dataset of records containing at
	least one data segment of structured data (FITS NAXIS > 0) and the name
	of a data series to which it writes a record containing at least one
	and up to three segments of the same structure as the input records and
	containing the per-pixel mean, variance, and count of valid values,
	respectively.

Status
	Not scheduled for inclusion in any JSOC release
	In use for ring-diagram pipelines on 2017.10.02 only
	for hmi.V_avg120[2195][240])

	This version involved a special call for pre-staging records suggested
	for improved efficiency in the database, but the efficiency actually
	declined, so the version was withdrawn.

	The source code and Makefile is in ~rick/src/drms/versions/datavg/

	This version has been superseded by Version 1.7

Usage
	datavg [-nov] in= DataDescriptor out= DataSeries
	    [param= val ...]
	
Flags
	-n	no output records produced, diagnostics only
	-o	remove effects of observer velocity
	-v	run in verbose mode

Parameters
	in		Either the name of an input data set (record selection)
			or the name of an input data series; in the latter case,
			tmid and length parameters must also be supplied.
			No default, must be provided.

	out		The name of the output data series. No default, must be
			supplied even if run with the -n flag.

	tmid		The target time at the midpoint of the averaging interval,
			in either DATE_TIME or Carrington CR:CL form.
			Default: Not Specified.

	length		The length of the averaging interval, in degrees of
			Carrington rotation or seconds, depending on the format
			of tmid. Default: Not Specified

	count		The name of the segment in the output series expected to
			contain the per-pixel valid data counts. Default:  valid

	mean		The name of the segment in the output series expected to
			contain the per-pixel means. Default: mean

	power		The name of the segment in the output series expected to
			contain the per-pixel variances. Default: power

	log		The name of the generic segment in the output series expected
			contain the run log. Default: Log

	reject		The name of a file containing a list of records in the
			input data series to be rejected regardless of their
			quality mask values. The format of this file is to be specified.
			Default: Not Specified, implying no rejection list

	qmask		A 32-bit quantity specifying acceptable values of the quality
			value of the key associated with qual_key. If the bitwise
			and of qmask and the quality value for an input record is
			non-zero, that record will be skipped (treated as if it
			contained no data).Default: 0x80000000

	copy		A comma separated list of keywords, optionally terminated with
			',+', whose values are to be propagated to the output record
			as-is from the first input record in the input data set.
			Default: + implying the following default list:
			"TELESCOP", "INSTRUME", "WCSNAME", "WCSAXES"

	average		A comma separated list of keywords, optionally terminated with
			',+', whose values are to be averaged from the accepted input
			records for propagation to the output record. If the keyword
			'+' occurs in the list, the following default set of keywords
			will be averaged in addition to any others listed: OBS_VR,
			OBS_VW, OBS_VN. N.B.: The WCS keywords CRVALn, CRPIXn, CDELTn,
			and CROTAn, along with the tobs_key are always averaged.
			For the averaged keyword XXX an additional RMS keyword
			D_XXX will also be set. Default: "+"

	mscale		The FITS BSCALE value to be applied to the output mean segment.
			Default: Segment Default, implying use of the defalut value
			for the relevant segment in the output series

	mzero		The FITS BZERO value to be applied to the output mean segment.
			Default: Segment Default, implying use of the defalut value
			for the relevant segment in the output series

	pscale		The FITS BSCALE value to be applied to the output power segment.
			Default: Segment Default, implying use of the defalut value
			for the relevant segment in the output series

	pzero		The FITS BZERO value to be applied to the output power segment.
			Default: Segment Default, implying use of the defalut value
			for the relevant segment in the output series

	pkey		The prime key name for the index over which records are selected
			for averaging. Default: T_OBS

	qual_key	The key name for a 32-bit image quality field in the input data
			series;  Default: Quality

	roll_key	The key name for the input roll-angle field;  Default: CROTA2

	pa		The centre of acceptable roll angle values of the roll_key (deg);
			Default: 180.0

	dpa		The maximum deviation of acceptable roll angle values of the
			roll_key (deg) from pa; records with roll-angles outside this
			range will be skipped. Default: 1.0

	cvkey		The key name to be used for the key value representing the
			calibration version in the input data series to be tested
			against values of cvok and cvno; Default: CalVer64

	cvok	 	The acceptable value of cvkey; Default: any

	cvno	 	The unacceptable value of cvkey; Default: none

	setkey	 	The name of a special extra key to be set to the value setval
			in the output record; Default: Not Specified

	setval	 	The value to which the keyword setkey is to be set in the
			output record; Default: Not Specified

Input Data series class:
	In principle, any series containing at least one data segment of structured data
	(a hypercube, FITS NAXIS > 0); if the series contains multiple segments (of any
	type!), the segment to be averaged must be specified as part of the input dataset
	name.

Sample input series:
	hmi.V_45s, hmi.rdVpspec_fd05, hmi.rdVpspec_fd15, hmi.rdVpspec_fd30

Input keys potentially used or inspected:
	CAR_ROT, CDELT*, CRLN_OBS, CROTA*, CRPIX*, CRVAL*, CTYPE*, CUNIT*,
	LOG_BASE, OBS_VR, T_REC_index, WCSAXES (plus those specified as
	calling parameters)

Output Data series class:
	Data statistics hypercubes: expected to contain up to three segments,
	named as per the count, mean, and power parameters, with same rank and
	size (if not vardim) as the input series, and an optional generic segment
	named as per the log parameter.

Sample output series:
	hmi.V_avg120, hmi.rdVavgpspec_fd05, hmi.rdVavgpspec_fd15, hmi.rdVavgpspec_fd30

Output keys set (if possible):
	CarrRot, CMLon, MidTime, Module, BLD_VERS, Source, Input, Created, Interval,
	CTYPEn, CUNITn, COMMENT, HISTORY, DataRecs, MissRecs, T_FIRST, T_LAST,
	CRPIXn, D_CRPIXn, CRVALn, D_CRVALn, CDELTn, D_CDELTn, CROTAn, D_CROTAn,
	CRLN_OBS, D_CRLN_OBS, LOG_BASE, CountMIN, CountMAX, MeanMIN, MeanMAX, MeanAVG,
	MeanRMS, MeanSKEW, MeanKURT, PowrMIN, PowrMAX, PowrAVG, PowrRMS, PowrSKEW,
	PowrKURT (plus any keys [and D_keys as appropriate] as specified by copy,
	average, and setkey parameter values)

Sample commands:
	datavg in= hmi.V_45s out= hmi.V_avg120 tmid= 2161:300 length= 120 \
	    qmask= 0x80004000 reject= ~jsoc/hmi/tables/LHS_reject.v \
	    mscale= 0.25 mzero= 0.0 pscale= 10.0 pzero= 32767.0 \
	    average= "T_OBS,DSUN_OBS,CRLT_OBS,RSUN_OBS,OBS_VR,OBS_VW,OBS_VN" 

	datavg out= hmi.rdVavgpspec_fd15 mean= logP pkey= CMLon \
	    mscale= 0.0005 mzero= -11.5 \
	    in= hmi.rdVpspec_fd15[2160][][][+45.0][-30.0] \
	    copy= "CarrRot,LatHG,LonCM,DELTA_K,DELTA_NU,D_OMEGA,Apode_f,APOD_MIN,APOD_MAX" 

Bugs in current version:
	The application of the optional orbital velocity correction is purely scalar;
	only the value of OBS_VR is subtracted from all image values, regardless of
	image location, and transverse components of the orbital velocity are ignored.

	No check is performed to verify that values of keywords copied as-is from
	the first record in the dataset do not vary.

	The presence of generic segments in the input data series should not require
	specification of the segment name in the input data set.

	The reported extremal values of mean and power are determined before possible
	output scaling; they may be outside the representable range

	If the input dataset is specified explicitly in the in string, the keywords
	for CarrRot and CMLon are set to garbage, and there is no checking that the
	length is correct.

	If the input record set is not explicitly specified, the records must be
	uniquely tagged by either time or Carrington rotation and longitude.
	In the former case, geocentric physical ephemerides for the Sun are assumed.

Significant changes from Previous Version
	None; added stage_records call to pre-cache SUMS info

Valid HTML 4.01 Transitional HMI Ring Diagrams 3 May 2019, 10:44-0700