Python JQUERY like syntax to mimic Unix Shell Pipelines -

i not sure if has been asked before, wondering how can implement "set oriented" syntax in python mimic unix shell pipelines. in particular, how use "plain python functions" can produce, modify, or consume stream of records, , can glued using "." operator.

a straightforward example assuming python functions unix programs:

in unix can do:

    ls | egrep '^a' | wc -l 

to count files in current directory start 'a'. how about:

     unixtools import *     ls().egrep('^a').wc(countlines=true) 

and implementation of ls generator:

    @pipeline     def ls():        file in glob.glob("*"):           yield file 

i have omitted obvious glue code, make possible connect ls other "commands" of pipeline.

@pipeline attempt use decorators convert plain python function member of pipeline protocol.

i realise can done using traditional syntax of python, less readable forcing write in reverse, , more error prone:

     wc(countlines=true,input=egrep("^a",input=ls())) 

i guess study implementation of python pyquery figure out.

apparently, python pipe package close asking, , not aware of it.

so, answering own question since pipe package documentation not intuitive stated problem. when comes implementing full unix shell pipe semantics. @ end, there exist more unresolved questions regarding common unix tricks.

we can create producer python generator, using @pipe decorator. warned, type of producer can first member of pipeline:

from pipe import pipe @pipe def producer(arg=10):     x in range(arg):          yield x 

like in unix, can define transformer, expected transform input , generate output:

@pipe def transformer(iterable, arg=2):     x in iterable:          if x % arg == 0: yield x*2 

we can hook producer , transformer using "pipe" syntax, resembles of unix shell pipeline:

producer() | transformer() 

unlike unix shell, produces nothing output ;-( may write consumer, consumes input , prints something:

@pipe def consumer(iterable):     x in iterable:         print(x) 

and hookup work expected:

producer() | transformer() | consumer() 

interestingly, on second or more pipeline element, can omit parenthesis , use name of pipe function:

producer(10) | transformer | consumer 

another feature liked, equivalent of shell "here document":

[1,2,3,4] | transformer | consumer 

in case [1,2,3,4] interpreted sequence of values generator have been produced , "piped" next element in pipeline.

and can of course collect pipeline results:

a = [1,2,3,4] |  transformer | consumer 

---

and cases not obvious how deal "pipe":

it not clear how equivalent of stdout merging in shell:

(echo today ; date) | wc 

maybe using "cat" producer, able concatenate of iterable inputs?

cat(echo("today"),date()) | ... 

it not clear how consume few elements of pipeline , continue function, in unix shell:

cat /etc/passwd | (read line ; cat) 

in case, read consumes first line , cat sees rest of them. can overload "(" , ")" symbols?