data warehouse - ETL Testing Datasets / Framework -

-

i'm trying create reasonable tests our etl processes.

i'm thinking reference / testing ingestion dataset needed. don't want use client data (which other alternative here).

i run current etl on testing dataset reference transformations. way, when source code changes can test references being produced in etl , make sure no regressions created.

i'm not sure right approach. example, if transformation changed in source code, tests compare reference transformation rightly fail. we'd have create new reference transformation dataset transformation. can see getting crazy once team of developers starts making changes separate transformations.

ultimately, need way produce test dataset , test transformations. ideas?

create test data set, containing @ least 1 row every possible transformation outcome. you'll use test data set source every etl test run. new transformations or bugs come up, add additional rows test data set cover transformations.

in etl destination, create tests verify transformation of source data set. you'll need test every transformation outcome ensure complete code coverage. since test data set known , consistent source, tests should have predictable outcome.

automated etl testing isn't complex, complicated , can time consuming set up. requires disciplined development team maintain. luck.


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more