python - How to save data from the data stream while not blocking the stream? (PyQt5 signal emit() performance) -

i'm developing pyqt5 application. in application, has data stream, , speed 5~20 data/sec.

every time data arrives, following ondata() method of class analyzer called. (following code simplified code of app)

class analyzer():     def __init__(self):         self.cnt = 0         self.datadeque = deque(maxlength=10000)     def ondata(self, data):         self.datadeque.append({             "data": data,              "createdtime": time.time()         })         self.cnt += 1         if self.cnt % 10000 == 0:             pickle.dump(datadeque, open(file, 'wb')) 

but problem is, datadeque object large(50~150mb) dumping pickle takes 1~2 seconds.

during moment(1~2 seconds), requests calling ondata() method got queued, , after 1~2 seconds, queued requests call lots of ondata() method @ simultaneously, distorts createdtime of data.

to solve problem, edited code use thread (qthread) save pickle.

the following code edited code.

from pickledumpingthread import pickledumpingthread pickledumpingthread = pickledumpingthread() pickledumpingthread.start()  class analyzer():     def __init__(self):         self.cnt = 0         self.datadeque = deque(maxlength=10000)     def ondata(self, data):         self.datadeque.append({             "data": data,              "createdtime": time.time()         })         self.cnt += 1         if self.cnt % 10000 == 0:             pickledumpingthread.pickledumpingsignal.emit({                 "action": savepickle,                 "deque": self.datadeque             })             # pickle.dump(datadeque, open(file, 'wb')) 

the following code pickledumpingthread class.

class pickledumpingthread(qthread):    def __init__(self):        super().__init__()        self.daemon = true        self.pickledumpingsignal[dict].connect(self.savepickle)     def savepickle(self, signal_dict):        pickle.dump(signal_dict["deque"], open(file, 'wb')) 

i expected newly edited code dramatically decrease stream blocking time(1~2 seconds), code still blocks stream 0.5~2 seconds.

it seems pickledumpingthread.pickledumpingsignal.emit(somedict) takes 0.5~2 seconds.

my question 3 things.

1. is signal emit() function's performance not this?

2. is there possible alternatives of emit() function in case?

3. or there way save pickle while not blocking data stream? (any suggestion of modifying code highly appreciated)

thank reading long question!

something might work

class pickledumpingthread(qthread):    def __init__(self, data):        super().__init__()        self.data = data     def run(self):        pickle.dump(self.data["deque"], open(file, 'wb'))        self.emit(qtcore.signal('threadfinished(int)'), self.currentthreadid())  class analyzer():     def __init__(self):         self.cnt = 0         self.datadeque = deque(maxlength=10000)         self.threadhandler = {}      def ondata(self, data):         self.datadeque.append({ "data": data, "createdtime": time.time() })         self.cnt += 1         if self.cnt % 10000 == 0:             thread = pickledumpingthread(self.datadeque)             self.connect(thread, qtcore.signal("threadfinished(int)"), self.threadfinished)             thread.start()              self.threadhandler[thread.currentthreadid()] = thread      @qtcore.pyqtslot(int)     def threadfinished(id):         del self.threadhandler[id] 

self.threadhandler know how many threads still running, can rid of , threadfinished method