Batch processing RTKLIB solutions with RNX2RTKP and Python

The RTKPOST GUI app in RTKLIB is a great tool for interactively exploring post-processed PPK or PPP solutions. However, once you have settled on a solution configuration and simply want to run that solution on many data sets, RTKPOST is not the right tool. The RNX2RTKP command line app is a better choice but will still only run a single solution at a time. If your goal is to run the same solution configuration on many data sets, then you will need to add some kind of wrapper to call the RNX2RTKP app multiple times.

I usually use a python script to do this. Below I have included a simple script that I used to process the large number of cellphone data sets in my previous post. It is configured to run in Windows but with possible minor modifications, it should run in linux as well. It is not meant to be used as is, but is a template you can use to write your own script to match the structure of your data.

'''
 batch_rnx2rtkp - template to run multiple simultaneous solutions of rnx2rtkp in Windows.
    This example is configured to run the cellphone data sets available at 
    https://data.mendeley.com/datasets/5prmtwgph3/2
 '''

import os
import subprocess
import psutil
import time

# set location of data and rnx2rtkp executable
datapath = r'C:\gps\data\cellphone\Mi8_Julien\0521_dataset'
binpath = r'C:\gps\rtklib\bins\demo5_b34d'

# Choose datasets to process
DATA_SET = 0  # 0=open-sky, 1=partial forest, 2=forest
USE_GROUND_PLANE = True  # True or False

# set input files
cfgs = ['ppk_ar_1027_snr24']  # list of config files to run (files should have .conf ext)
rovFile = 'GEOP*.19o'  # rover files with wild cards
baseFile = 'BBYS*.19o' # base files with wild cards
navFile = r'"..\brdm*.19p"' # navigation files with wild cards
outFile = 'ppk_ar_1027'

# set maximum number of simultaneous occurences of rnx2rtkp to run
max_windows = 10 

# get list of current threads running in Windows
current_process = psutil.Process()
num_start = len(current_process.children())

# get list of date folders in data path
dates = os.listdir(datapath)

# loop through date folders
for date in dates:
    datepath = datapath + '/' + date
    print(datepath)  
      
    # Filter which folders to process
    if not os.path.isdir(datepath):  # skip if not folder
        continue
    if USE_GROUND_PLANE:
        if date[-2:] != 'gp':  #skip folders without ground plane tag in name
            continue
    else: # no ground plane
        if date[-2:] == 'gp':  #skip folders with ground plane tag in name
            continue

    # Get list of datasets in this date folder
    datasets = os.listdir(datapath + '/' + date)
    
    # Select desired folder in data set 
    dataset = datepath + '/' + datasets[DATA_SET] # 0=open-sky, 1=partial forest, 2=forest 
    os.chdir(dataset)
 
    # Run a solution for each config file in list       
    for cfg in cfgs:
        # create command to run solution
        rtkcmd=r'%s\rnx2rtkp -x 0 -y 2 -k ..\..\%s.conf -o %s.pos %s %s %s' % \
            (binpath, cfg, outFile + '_' + cfg, rovFile, baseFile, navFile)    
        
        # run command
        subprocess.Popen(rtkcmd)


    # if max windows open, wait for one to close
    while len(current_process.children())-num_start >= max_windows:
        time.sleep(1) #wait here for existing window to close

# Wait for all solutions to finish
print('Waiting for solutions to complete ...')  
while len(current_process.children())-num_start > 0:
    pass #wait here if max windows open        
print('Done')

RTKLIB is a single threaded app so will not take advantage of multiple processors on a single computer. To get around this limitation and maximize the use of all processors, the script will launch separate processes for each RTKLIB solution until a maximum number of simulatenous processes has been reached and then will wait for a prior process to complete before launching a new solution. For my typical laptop, I find that setting the maximum number of simultaneous windows to 10 works fairly well but this number can be adjusted up or down based on the processing power of your computer.

In this case I have run only one solution for each dataset but the script is set up to run as many different solutions as desired. Just add an additional config file name to the list of config files for each desired solution. Be aware that the list of config files leaves off the file extensions. The actual config files should all have a “.conf” extension.

Note that the list of input files uses wildcards in the file names in many cases, since the file names for each data set will likely vary slightly from data set to data set.

In my example, the datasets contain three different types of environment (open sky, partial forest, and forest) and two different antenna configurations (ground plane or no ground plane). I have set the script up to only run the data for one type of environment and one antenna type as specified in the input parameters at the top of the script. However, this can be modified to filter the datasets in any way desired or to just run all of them.

There are probably faster and more elegant ways to do this, but if you are just looking for something simple to allow you to quickly run a given solution or set of solutions on many data sets, then you may find this useful.

Advertisement

7 thoughts on “Batch processing RTKLIB solutions with RNX2RTKP and Python”

  1. Hei,
    First of all, thank you for your website, it has been extremely helpful to decipher errors and optimise results.
    I am very puzzled by position discrepancies when using the GUI (RTKPOST) vs the command line (rnx2rtkp). Have you ever experienced differences in solution between GUI and CLI?

    My static workflow is as follows: I start with the GUI, optimise the parameters to improve precision and then, after exporting the same configuration parameters in a .config file, I run rnx2rtkp for the same files, and do not obtain the same result. The visual comparison is done in RTKPOST. The solution points are similar but variations in positions can be one order of magnitude different and show different pattern of the solution (also a lower Q in my case). My conclusion is that the GUI performs better than the CLI, and I have no idea why. I hope to find what causes the difference as I plan to perform most of my computations with rnx2rtkp for parallelisation purposes using xargs.

    Thanks for your help!

    Like

    1. Hi PiM. RTKPOST and RNX2RTKP share all the same code except the user interface so any differences between the two must come from different inputs. I’m not aware of any code issues that would casuse this but it’s always possible. If you can email the data files, config file, and your solution files to me at rtklibexplorer@gmail.com I’ll take a look.

      Like

  2. Great script !
    On Linux I made a quite similar one in Bash. It collects base observations and ephemeris by ftp with wget and compute filename components (gps week, day of week, etc…) with the “date” program.
    I didn’t thought about multi threading. Next time I’ll try to use gnu parallel inside the script…

    Like

      1. Here it is :


        #!/bin/bash
        ls_stations="sleu sntl lepo pier"
        conf="PTR-tropo.conf" # RTKLIB config file for post-processing
        y=2020

        Functions

        Time variables

        vartemp() {
        t=$(date -d "$((y-1))-12-31 $doy days" +%s)
        gps0=$(date -d "1980-01-06" +%s)
        gpsw=$(((t-gps0)/604800))
        dow=$(date -d "@$t" +%w)
        ys=$(date -d "@$t" +%g)
        }

        Post-processing function for various elevation masks (el)

        usage ptr base_name rover_name

        ptr() {
        echo $1 "->" $2
        for el in $(seq 0 5 50); do
        fichier=$y$doy$1$2$(printf "%02d" $el).pos
        echo $fichier
        rnx2rtkp -k $conf -m $el {$2,$1}${doy}0.${ys}d $2${doy}0.${ys}{n,g} ig*$gpsw$dow.sp3 > $fichier
        done
        }

        Collect RGP stations data

        for doy in {001..366};do
        vartemp
        echo "**********"
        echo $(date -d "@$t" +%F) "#" $gpsw $dow $doy
        for station in $ls_stations;do
        wget -nv -nc ftp://rgpdata.ign.fr/pub/data/$y/$doy/data_30/$station${doy}0.${ys}{d,g,n}.Z
        done
        wget -nv -nc ftp://rgpdata.ign.fr/pub/products/ephemerides/$gpsw/ig{s,l}$gpsw$dow.sp3.Z
        done

        uncompress -f *.Z

        Post-processing

        for doy in {001..366};do
        vartemp
        echo "**********"
        echo "Post-processing" $(date -d "@$t" +%F) "#" $gpsw $dow $doy
        ptr "lepo" "sleu"
        ptr "pier" "sleu"
        ptr "sntl" "sleu"
        ptr "lepo" "sntl"
        ptr "pier" "sntl"
        echo "**********"
        done

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: