15 Nov The new Weighted BatchPreprocessing v1.4.0
Click here for the Italian version.
There is a big news for BatchPreprocessing lovers, but also for those who are not and could become one!
I’ve always said, especially in dedicated tutorials, that BatchPreprocessing is a great script to automate calibration and integration processes and quickly obtain the final masterLight, but it does not allow total control over the individual phases to let our frames to express themselves at their best.
This was the old BatchPreprocessing (BPP) and, partially, that’s still the case. But there is a new version of BPP, or rather an extension signed by Roberto and Tommaso that enriches the old BPP with a set of features aiming at reducing the gap between the script and the typical manual calibration and integration methods. Since several new features are added, let’s dig into them one by one.
THE DAWN OF THE SCRIPT
It was the great Roberto Sartori who dragged me into this adventure, a person I didn’t know until he contacted me to make this proposal. 
After reading my tutorial on Subframe Selector (SFS) he had the idea to integrate it into the BPP script, asking me for help in the test phase and in the tuning for those parameters on which the weights of the SFS are based. In fact, for the uninitiated, SFS evaluates the individual Light frames by measuring SNR, eccentricity and FWHM of the stars (in addition to other parameters that we do not currently consider). With these 3 parameters we can determine which frames are the best and which are the worst, such that they can be weighted accordingly during the integration phase in order to obtain the best possible result. These weights varies when we relate these 3 measurements to each other, but for a deeper understanding I recommend reading a dedicated tutorial.
Due to the possibility of creating weights, the new script was called Weighted Batch Preprocessing, or more simply WBPP. Roberto took charge of all the dirty work, thanks to his knowledge he gave birth to the WBPP, I had the task of looking for meaningful presets to relate the above three parameters to each other. So I started to do several tests on resumed sessions from the past by varying the values of the parameters until the results started looking better, once I found some valid presets, we gave birth to version 1.0.0 of WBPP. But my images, even if numerous, did not provide a valid statistical population and for this reason we asked other friends to test it and provide feedbacks. This fundamental test phase was needed to extend the tests to a wide set of different astrophotography sessions and to bring out as many usability issues and bugs as possible. It was precisely at this stage, from the analysis of the script’s execution and its issues, that new ideas were born and implemented by Roberto, and in a few weeks we moved from version 1.0.0 to 1.4.0, a clear sign of the great work done and the numerous new features added to the script.
You can download the script directly from here (version 1.4.0) and add it to PixInsight.
WHAT’s NEW IN WBPP
The list of new features in WBPP are the following:
- possibility of weighing our Light frames thanks to the Subframe Selector
- possibility to automatically select the reference frame for Light frames registration
- possibility of grouping Light frames based on different integration times, useful for creating HDR images later
- smart naming
- a different approach to flat frames calibration
- automatic choice of the best pixel rejection algorithm in the integration phase
- extended diagnostic
- smart reporting
- session saving
Some of these functions are rather simple and intuitive, others more complex and arduous. So let’s dig into the details of every single new feature to allow everyone to use each of it them at best.
CREATE WEIGHTS AND AUTOMATICALLY CHOOSE THE REFERENCE FRAME
The most important feature consists in having automated the whole process carried out by the Subframe Selector in a few simple steps. We find everything we need in this new tab
As you can see there are 3 different options:
– Generate subframes weights: by activating this option we will activate the weighing of our Light frames, the weights will be assigned on the basis of the parameters that we will see shortly and the integration will actually be weighed. By disabling this option the integration will occur normally, as in a normal BPP.
– Compute weights after registration: by activating this option the weights will be computed after the light frames have been registered, keeping it disabled the weights are computed before the registration.
– Use best frame as reference for registration: this is one of the most interesting option. It allows you to automatically choose the reference frame for the registration, this frame is not the frame with the absolute best weight, but it is the one with the best stars, that is the one that has better FWHM and eccentricity, ignoring the SNR. To find out which frame is the most suitable as a reference for the others, a first Subframe Selector is run evaluating only these two parameters and without storing any weight into Light frames. As soon as we activate this option it will no longer be possible to choose the reference image manually.
By simply switching it off we will re-enable the manual choice of the reference frame for the registration.
The above option allows you to automatically choose the reference frame for the registration without the need to weight your images and carry out a weighted integration. To do that activate this option and deactivate “Generate subframes weights”
It’s time to finally look at how to set the parameters for the weights in the integration phase! By clicking on Weighting parameters we could set these parameters. We have 3 presets available that can be selected from the drop-down menu called Nebula, Galaxy and Cluster, designed to obtain different results based on the subject of our image. To understand how to interpret the values, let’s take the Nebula preset as an example
This preset will create a weight given by:
5% FWHM + 10% eccentricity + 20% SNR + 65
No frame can have a weight less than 65, moreover the SNR is weighed more than the other two parameters, this leads to final masterLight that depends more on the SNR. If we believe that none of the three presets is the right one for us, all we need to do is to tune the values by manually writing them directly in the relative box or simply by dragging the cursors. We will notice that the name of the preset disappears from the drop-down menu, and we must not worry if the sum of the values is not 100, at the end the weights will be internally recalculated in percentage terms.
Let’s take a closer look at these presets before moving to other features.
HOW TO CHOOSE THE BEST PRESETS
The 3 presets lead to slightly different results because they have been designed for different purposes, in particular:
– Nebula: aimes at obtaining the best SNR from our images
– Cluster: aims at improving the stars, obtaining a masterLight frame with a better FWHM and stellar eccentricity than a standard integration, at the expense of a minimal loss of SNR.
– Galaxy: intermediate between the two, leads to masterLights with good SNR and excellent FWHM and stellar eccentricity, ideal if we would keep good SNR and details at the same time.
It’s worth to highlight that we are not obliged to use the Galaxy preset only on images of galaxies, but, once understood their purpose, we can choose a preset regardless of the subject of our photo. Let’s take few examples:
Imagine a series of shots on M45 Pleiades, an open cluster of stars. The name suggest to use the “cluster” preset, but you could alternatively use the “galaxy” preset which would still lead to good star definition without sacrificing the nebulosity that surrounds the cluster.
Another example could be the one concerning a long focal imaging on nebulae, let’s imagine for example a very narrow field on the Pillars of Creation of the Eagle m16 nebula or on the area of Melotte 15, in the center of the Heart Nebula. In both cases we can slightly sacrifice the SNR, looking for a greater detail to highlight the beautiful nebular structures present here, avoiding the “Nebula” preset and choosing “Galaxy“, or even “Cluster“.
These are obviously only two examples, but what is important is to understand that the presets can be freely used. Alternatively, we can go and choose the values of each parameter independently, moving away from the 3 presets scheme, in order to weigh our images in a totally customizable and subjective manner.
FRAMES GROUPING AND SMART NAMING
The standard BPP groups our Light frames by binning and filter name in order to integrate Light fames obtained with different binning and filters at the same time. In our new WBPP we also have the possibility to group light frames by exposures. Thus we can add Light frames of different exposures to obtain different masterLights. Here is an example
In this case I acquired a first set of images on m31 at 180″ and a second set with an exposure of 240″, both with the same binning and the same luminance filter. In this way I can integrate the two different series separately, obtaining two different masterLights, one from 180″ frames and one from 240″ frames. This function is very useful if then the two masterLights are used to compose an HDR.
But it can also happen that you may want to integrate Light frames of different exposures into a single masterLight. It is still possible to do it by simply changing the “Exposure tolerance” parameter
By raising this value we’ll group Light frames if the time difference between these frames falls within the tolerance range. This value is expressed in seconds: by raising it to 60, for example, we will group the Light frames with a maximum exposure difference of 60 ″, in our case we will group the frames of 180″ with those of 240″.
And if we wanted to group all our Light frames together to get a single masterLight? nothing simpler, it will be enough to raise this value up to the maximum exposure difference between our Light frames.
But sometimes frame grouping could not appear correctly, it could happen that the added frames miss some information in their header so the script doesn’t succeed in reading filter name, binning or exposure from the file. Here is a practical example
As you can see in the image, the script does not recognize the exposure of my masterDark, thus assuming a time of 0.00 seconds even if, in reality, the masterDark has an exposure time of 1200 seconds. This could compromise the Light frame calibration result. However, we can work around this issue leveraging the new SMART NAMING function of the script, which allows us to add the missing information by renaming the files properly. In my case I simply added the string EXPTIME_1200 to the name of my masterDark
As you can see the script now recognizes the duration of the frame from the file name when it is added. We can rename our files in order to provide all the missing information to the script, namely binning, filter name and exposure time to properly perform the calibration.
Moreover, WBPP recognized the exposure also with the format “900s” or “900sec” or “900_secs”. These are the common syntaxes used by many astrophotography softwares.
If more than one of these information are missing we can chain them to the file name. For example, if we have a file with an exposure of 900 seconds, a Luminance filter and binning 1 we can simply rename it adding:
BINNING_1_FILTER_Lum_1200s
and here the script recognizes all three information correctly
Use this nomenclature to give the information you want to the script: filename_BINNING_XX_FILTER_XX_XXXs
Alternatively, you can add these information to the enclosing folder name so all included files inherit these information by default.
But be careful, smart naming does not override any information contained in the header file, if duration is properly saved as a FITs key then the file renaming does not change it. If you really want to force the script to assume different values with respect those contained in the header file you must add the file using the “add custom” option.
Finally, there is also a change in the name of the master files generated by WBPP. The standard BPP generates masters files into “master” folder with the following name conventions:
bias-BINNING_1.xisf
dark-BINNING_1-EXPTIME_0.2.xisf
flat-FILTER_Ha-BINNING_1.xisf
light-FILTER_Ha-BINNING_1.xisf
Differently, the new WBPP will generate the following master files:
masterBias-BINNING_1.xisf
masterDark-BINNING_1-EXPTIME_0.2.xisf
masterFlat-BINNING_1-FILTER_Ha.xisf
masterLight-BINNING_1-FILTER_Ha-EXPTIME_300.xisf
with the addition of the exposure time to the master light frame and the “master” prefix to clearly distinguish master files from single frames.
FLAT FRAMES CALIBRATION
Another small change concerns the calibration of the flat frames. The standard BPP implements two flows: either calibrate our flat frames by picking the masterDark frame with the closest exposure or by picking the masterFlat dark frame with exactly the same exposure in order to avoid to calibrate the flat frames with a masterDark that has a huge different exposure. This last option is enabled by activating the option “Calibrate with flat darks only“
If this option is active and no suitable masterFlatDark with the same exposure is found then Flat frames calibration is entirely skipped, ignoring the presence of any masterBias. In the new WBPP, instead, masterBias frame, if provided, is always used and the flats calibration will be performed despite the presence or not of a suitable masterFlat dark frame, an alternative not allowed until now.
AUTOMATIC CHOICE OF THE PIXEL REJECTION ALGORITHM
In the standard BPP, as in the Image Integration process, the user is responsible for the choice of the rejection algorithm to perform the important function of pixel rejection in the integration phase. In our WBPP you have the possibility to make it automatically done by simply selecting auto in the appropriate menu
The automatic selection of the rejection algorithm is simply based on the number of individual frames of each group, in particular it will be chosen:
Percentile Clipping for a number of frames up to 7
Averaged Sigma Clipping if the number ranges from 8 to 10
Winsorized Sigma Clipping from11 to 19 frames
Linear Fit Clipping for a number of frames between 20 and 24
Generalized Extreme Studentized Deviate (ESD) for 25 or more images
This choice can be really useful and even mandatory if we have multiple groups with a different size since the auto option can assign different rejection methods to different groups while any other fixed method is shared across all groups of the same frame type (bias, dark, flat and light). By selecting auto we should no longer worry about choosing the proper rejection algorithm and each group will be integrated with the proper rejection method.
EXTENDED DIAGNOSTIC
The new WBPP has the same diagnostic option already present in the standard BPP
In our WBPP, however, the diagnostic covers more potential issues you should be aware of and a wider number of new warning messages have been prepared.
For example, in the above case if I run the diagnostic before starting the script I get the following message
In a nutshell, it warns me that my light frames will be calibrated with a masterDark of different duration, using the optimize option, without reporting other errors. Before starting our script, we always recommend checking this useful tool.
SMART REPORTING
The WBPP always carries out its work up to the end, even when errors are encountered during the various phases. Now, at the end of each run, a new report is shown listing all the work phases performed by WBPP, this report highlights which phases have been carried out correctly and which have encountered issues, so the user can easily identify and solve those that have been failed. Here is an example of a report, with no errors reported.
In standard BPP some errors encountered can completely interrupt the script. We decided to eliminate this interruption and add the final report just to highlight which phases failed. But why not interrupting the script? Because it seemed counterproductive to us that the script would stop completely due to an issue that maybe involves only a single group of light frames, in this way even if calibration / registration / integration fails for one group the remaining groups can still be processed. At the end, thanks to the final report, the user will be able to quickly identify and address any issue occurred.
SAVING THE SESSION
Finally it’s worth to highlight the possibility to save the images loaded into our script by simply activating the “Save frame group on exit” option, so if we close and reopen our script we will find the images previously loaded, without the need to manually reload the individual frames again. It can be very useful if you want to do more integration tests with the same image session or for anyone, like me, to preload by default both masterBias and masterDark previously generated.
INTERESTING RESULTS
As mentioned at the beginning of the article we tested our extension on different images, but we know very well that they are still too few to consider them a valid statistical population, moreover the amount of variables in astronomical images is so high that we do not always expect positive results (but we believe that in the in most cases you can get benefits).
Among the first positive results we had, we were stunned by this image sent by Cristian Mari in which we see a comparison between a masterLight obtained using WBPP with the Nebula preset (on the left) and a masterLight obtained with the standard BPP, without weights (on the right).
This is the same series of images on the Pacman nebula with Ha filter, it seems evident the improvement of SNR in the image obtained with the new WBPP.
Another case interesting case is provided by Riccardo Sgaramella, where again we compare the image obtained with the WBPP on the left and the image with the standard BPP on the right
Probably you would already have recognized the Veil nebula in these images, in this case the image obtained with the WBPP seems to be noisier and less clean, but at the same time it appears even sharper, with sharper stars, finer details and a greater contrast.
But the result is not always so obvious, the improvement or deterioration is often a matter of decimal values and hardly distinguishable by eye, so it is spontaneous to open the Subframe Selector script and measure the masters obtained with the two scripts to see if we can objectively measure the difference.
I have personally integrated the same series of images of the NGC 7822 nebula with both the standard BPP and the new WBPP using all three presets. Here are the results
This time numbers talk. We immediately see that the image obtained with the standard BPP has the worst stars in terms of both FWHM and eccentricity. The preset cluster brings the best results on the stars by sacrificing the SNR, aiming at achieving the highest possible sharpness. The nebula preset is the one that best expresses the SNR of my image and, finally, the galaxy preset is an excellent compromise between the other two presets providing an excellent SNR and, at the same time, a good eccentricity and FWHM.
But the really amazing thing about this measurement is that overall all 3 presets have yielded better results than the standard BPP. I decided to compute the weights of the 4 images, evaluating FWHM, eccentricity and SNR and assigning the same values to these 3 parameters. The final result sees the image obtained with the standard BPP in the last place, while the best is the one generated by WBPP with the Nebula preset. Since the subject of the image is indeed a nebula, the aforementioned NGC 7822, the result obtained has fully matched the expectations.
FOR THOSE WHO WANT TO DIG MORE
With this short article you are already able to take advantage of the new features of our Weighted Batch Preprocessing, but for anyone who wants to go deeper into the theory and mechanisms behind certain processes, you can download from HERE a more technical and detailed pdf document, written directly by Roberto.
Acknowledgments
The WBPP script that you can download from this page is the work of many people, Roberto and I alone probably would never have achieved these goals in such a short time without the help of a group of astrophotographers who spent part of their time on testing, reporting issues and providing invaluable feedbacks. Therefore we would like to say a big “thank you” to
Alberto Vezzani
Alessandro Soffia
Cristian Mari
Domenico Cataldi
Emanuele Fumagalli
Federico Carollo
Giambattista Rizzo
Gianluca Spanò
Marco Formenton
Marco Prelini
Matteo Fumagalli
Maurizio Berti
Mauro Narduzzi
Mirko Tondinelli
Oscar Cattina
Riccardo Balia
Riccardo Sgaramella
Stefano Bolis
Valerio Avitabile
To report any bugs or errors please send an email to the following address weightedbpp@gmail.com