Databases
StackOverflow2013
Postsdbo
POHow to improve my performance in filling gaps in time series and data lists with Python
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to improve my performance in filling gaps in time series and data lists with Python
    primarykey
    data
    text
    <p>I'm having a time series data sets comprising of 10 Hz data over several years. For one year my data has around 3.1*10^8 rows of data (each row has a time stamp and 8 float values). My data has gaps which I need to identify and fill with 'NaN'. My python code below is capable of doing so but the performance is by far too bad for my kind of problem. I cannot get though my data set in anything even close to a reasonable time.</p> <p>Below an minimal working example. I have for example series (time-seris-data) and data as lits with same lengths:</p> <pre><code>series = [1.1, 2.1, 3.1, 7.1, 8.1, 9.1, 10.1, 14.1, 15.1, 16.1, 20.1] data_a = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] data_b = [1.2, 1.2, 1.2, 2.2, 2.2, 2.2, 2.2, 3.2, 3.2, 3.2, 4.2] </code></pre> <p>I would like series to advance in intervals of 1, hence the gaps of series are 4.1, 5.1, 6.1, 11.1, 12.1, 13.1, 17.1, 18.1, 19.1. The data_a and data_b lists shall be filled with float(nan)'s. so data_a for example should become:</p> <p>[1.2, 1.2, 1.2, nan, nan, nan, 2.2, 2.2, 2.2, 2.2, nan, nan, nan, 3.2, 3.2, 3.2, nan, nan, nan, 4.2]</p> <p>I archived this using:</p> <pre><code>d_max = 1.0 # Normal increment in series where no gaps shall be filled shift = 0 for i in range(len(series)-1): diff = series[i+1] - series[i] if diff &gt; d_max: num_fills = round(diff/d_max)-1 # Number of fills within one gap for it in range(num_fills): data_a.insert(i+1+it+shift, float(nan)) data_b.insert(i+1+it+shift, float(nan)) shift = int(shift + num_fills) # Shift the index by the number of inserts from the previous gap filling </code></pre> <p>I searched for other solutions to this problems but only came across the use of the find() function yielding the indices of the gaps. Is the function find() faster than my solution? But then how would I insert NaN's in data_a and data_b in a more efficient way?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USFred Foo
    1. USBetrieb
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to improve my performance in filling gaps in time series and data lists with Python
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to improve my performance in filling gaps in time series and data lists with Python
      1. USunutbu
      1. VTFavorite
    3. VO
      singulars
      1. POHow to improve my performance in filling gaps in time series and data lists with Python
      1. USVicente Reig
      1. VTFavorite
    1. COWhat's your access pattern? If the result doesn't need to be eagerly evaluated, you could lazily generate `[(series, a, b)]` and that might be faster depending on how you use it. Or, if you need random access, a container-like class implementing `__getitem__` could do something clever
      singulars
      1. POHow to improve my performance in filling gaps in time series and data lists with Python
      1. USUseless
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload