What's the Best Way to Schedule and Manage Multiple Processes in Python 3 -

I am working on Python 3 on a project that includes reading lines from the text file, in some way these lines By adding, and then writing out the results of that manipulation in another text file. Applying that flow in serial manner is trivial.

However, it takes a long time to run on every step (I'm working on text files that are several hundred megabytes / many gigabytes in size). I thought about breaking this process into multiple, real system processes. Based on the recommended best practices, I am going to use Python's multiprocessing library.

Ideally, one should read only another process and writing text files is part of the manipulation, however, that's where I'm running into issues.

When "Reader Process" reads a row from the initial text file, then it will be the same as line . Then drag the "manipher process" from that line to qi , then do your work, then put it in another Q , which takes the "author process" and another In the text file as it stands now, the manipulation process only checks to see if there is data in "reader line ", and if this happens, then they get ( ) Data from line and talk about them, however, those processes read The process runs may be the first iteration, the program due to the stall.

In your opinion, there will be "Best Way" to schedule the processes in this manner, so long as the reader process does not insert data in the Queue Until the manipulation procedures will not go, and vice versa with the process? I considered switching off a custom signal, but I'm not sure that this is the most appropriate way. Any help will be greatly appreciated!

If I were you, I would split your file in the indexed volume and calculate - intensive manipulation part If this is not possible (for example, if the lines are not independent of any reason), then you may have to implement serial implementation in any way.

Once you have N Chunks in different files, you start your serial manipulation script N for each stream, later, combine the output back into one file. If you do it like this, then no queue is needed and you will save yourself some work.