airflow - Can a workflow with many, many subdags be performant? -

-

i have workflow involves many instances of subdagoperator, tasks generated in loop. pattern illustrated following toy dag file:

from datetime import datetime, timedelta airflow.models import dag airflow.operators.dummy_operator import dummyoperator airflow.operators.subdag_operator import subdagoperator  dag = dag(     'subdaggy-2',     schedule_interval=none,     start_date=datetime(2017,1,1) )  def make_sub_dag(parent_dag, n):     dag = dag(         '%s.task_%d' % (parent_dag.dag_id, n),         schedule_interval=parent_dag.schedule_interval,         start_date=parent_dag.start_date         )     dummyoperator(task_id='task1', dag=dag) >> dummyoperator(task_id='task2', dag=dag)     return dag  downstream_task = dummyoperator(task_id='downstream', dag=dag) n in range(20):     subdagoperator(         dag=dag,         task_id='task_%d' % n,         subdag=make_sub_dag(dag, n)         ) >> downstream_task

i find convenient way organize tasks, particularly since helps keep top-level dag uncluttered, if subdag contains more tasks (i.e. tens, not couple.)

the problem is, approach doesn't scale number of subdags (20 in example) increases. find when total number of dag objects created in overall workflow surpasses 200 (which can happen production workflow, if pattern occurs several times) things grind halt.

so question: there way organize tasks way (many similar subdags) scales hundreds or thousands of subdags? profiling suggests process spends lot of time in dag object constructor. perhaps there way avoid instantiating new dag object each of subdagoperators?

well, seems @ least significant part of problem indeed due cost of dag constructor, in turn in large part due cost of inspect.stack(). put simple patch replace cheaper method , there indeed seem improvement - flow several thousand subdags failed load me loads. we'll see if goes anywhere.


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more