linux - How to execute shell command and stream output with Python and Flask upon HTTP request? -

following this post, able tail -f log file webpage:

from gevent import sleep gevent.wsgi import wsgiserver import flask import subprocess  app = flask.flask(__name__)  @app.route('/yield') def index():     def inner():         proc = subprocess.popen(                 ['tail -f ./log'],                 shell=true,                 stdout=subprocess.pipe                 )         line in iter(proc.stdout.readline,''):             sleep(0.1)             yield line.rstrip() + '<br/>\n'      return flask.response(inner(), mimetype='text/html')  http_server = wsgiserver(('', 5000), app) http_server.serve_forever() 

there 2 issues in approach.

1. the tail -f log process linger after closing webpage. there n tail process after visiting http://localhost:5000/yield n time
2. there can 1 client accessing http://localhost:5000/yield @ single time

my question(s) is, possible make flask execute shell command when visit page , terminating command when client close page? ctrl+c after tail -f log. if not, alternatives? why able have 1 client accessing page @ time?

note: looking general way of starting/stoping arbitrary shell command instead of particularly tailing file

here code should job. notes:

1. you need detect when request disconnects, , terminate proc. try/except code below that. however, after inner() reaches end, python try close socket normally, raise exception (i think it's socket.error, per how handle broken pipe (sigpipe) in python?). can't find way catch exception cleanly; e.g., doesn't work if explicitly raise stopiteration @ end of inner(), , surround try/except socket.error block. may limitation of python's exception handling. there may else can within generator function tell flask abort streaming without trying close socket normally, haven't found it.

2. your main thread blocking during proc.stdout.readline(), , gevent.sleep() comes late help. in principle gevent.monkey.patch_all() can patch standard library functions block thread yield control gevent instead (see http://www.gevent.org/gevent.monkey.html). however, doesn't seem patch proc.stdout.readline(). code below uses gevent.select.select() wait data become available on proc.stdout or proc.stderr before yielding new data. allows gevent run other greenlets (e.g., serve other web clients) while waiting.

3. the webserver seems buffer first few kb of data being sent client, may not see in web browser until number of new lines have been added ./log. after that, seems send new data immediately. not sure how first part of request sent right away, it's pretty common problem streaming servers, there should solution. isn't problem commands terminate on own, since full output sent once terminate.

you may find useful @ https://mortoray.com/2014/03/04/http-streaming-of-command-output-in-python-flask/ .

here's code:

from gevent.select import select gevent.wsgi import wsgiserver import flask import subprocess  app = flask.flask(__name__)  @app.route('/yield') def index():     def inner():         proc = subprocess.popen(                 ['tail -f ./log'],                 shell=true,                 stdout=subprocess.pipe,                 stderr=subprocess.pipe                 )         # pass data until client disconnects, terminate         # see https://stackoverflow.com/questions/18511119/stop-processing-flask-route-if-request-aborted         try:             awaiting = [proc.stdout, proc.stderr]             while awaiting:                 # wait output on 1 or more pipes, or proc close pipe                 ready, _, _ = select(awaiting, [], [])                 pipe in ready:                     line = pipe.readline()                     if line:                         # output report                         print "sending line:", line.replace('\n', '\\n')                         yield line.rstrip() + '<br/>\n'                     else:                         # eof, pipe closed proc                         awaiting.remove(pipe)             if proc.poll() none:                 print "process closed stdout , stderr didn't terminate; terminating now."                 proc.terminate()          except generatorexit:             # occurs when new output yielded disconnected client             print 'client disconnected, killing process'             proc.terminate()          # wait proc finish , return code         ret_code = proc.wait()         print "process return code:", ret_code      return flask.response(inner(), mimetype='text/html')  http_server = wsgiserver(('', 5000), app) http_server.serve_forever()