python - Google Earth Engine : Exporting MODIS images from GEE to AWS S3 bucket -

-

i working on machine learning project uses modis dataset. pc doesn't meet computational requirements of project, had taken aws server. problem earth engine exporting images google drive or google cloud storage want them exported s3 bucket.

i have come across answers suggesting download data local storage , upload them s3 bucket. given huge datasets , poor data speed, take me ages so. hence want export them s3 bucket directly using earth engine.

i have gone through documentation exporting happens (ee.batch.export.image). thinking of writing function exports geotiff images aws s3 bucket instead of google drive or cloud storage.

p.s.

  • i have verified amazon modis public datasets , datasets want (mod09a1 , few others) aren't offered amazon.
  • i have windows 10 installed on pc.

modis imagery on aws s3 (https://aws.amazon.com/public-datasets/modis/)

however, interesting question other data set , here few things consider

1) google earth engine can write onto google cloud storage buckets (gcs) free upto 5 gb or google drive has limit of 15 gb including gmail. able download these images local drive before pushing aws s3 need make sure have enough space available on either gcs or drive.

2) google earth engine not export metadata , split large geotiff if exceeds file size limits(certain things keep in mind) incase want mosaic split images single image before uploading aws. can export properties metadata csv or kml file.

3) once know have enough buffer space in gcs or google drive ideal method be

  • push ee drive/gcs
  • pull drive/gcs local , push aws (if wanted using google's network speed instead of client resource can spin small micro instance under always free model google)

  • another way without using client resource using web integration service, example, zapier can link drive aws new files come in copied aws , can use new file coming in trigger). [i have not tried know can done using zapier or ifttt.

  • periodically check cloud storage , delete files copied onto gcs can have check if files or folders synced , delete files on gcs/drive free space again , repeat process.

i sure others might have other great suggestions way of doing it. hope helps

sam


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more