A project I wish existed is an open source competitor to Google Earth's automatic generation of 3D models from satellite images. Their generator is pretty smart and even generates pretty good approximations of trees, bushes, streetlights, and so on. This is very useful when you need to figure out 360° line-of-sight from a point in space (i.e. a viewshed) for things like planning where to put RF antennas as topographic maps don't take vegetation or other obstructions into account.
Google Earth even has built-in viewshed generation which uses their 3D model data (works great), but unfortunately the 3D model layer only seems to exist for fairly populated areas (probably because it's computationally expensive) - so doing RF planning in rural areas requires more traditional legwork.
So it would be cool for a project to provide similar functionality for OpenStreetMap using this public satellite data. OpenStreetMap does have 3D modeling support but last time I looked it seemed pretty focused on buildings and people hand-editing the data.
The fact that Google has so many cities mapped for aerial imagery is very interesting to me. Do we know how they get this aerial imagery data? (i.e. Is it one or more subcontracted companies that specialize in aerial photography?)
"The US National Agriculture Imagery Program (NAIP) offers aerial image data of the US at one-meter resolution, including nearly complete coverage every several years since 2003. Earth Engine includes this data as well as sample imagery from several commercial providers."
I’d argue that we shouldn’t be so quick to say what isn’t in OSMs scope.
To me, OSM should be the open Source Of Truth for all worldwide GIS data, a way to refer to our entire cartographic world in a digital format. Perhaps now it’s restricted to mapping data, but I would love to see the steering committee bring additional products out (temporal layers, the mapping you mention, and so on).
If OSM is important to you, there’s an open executive position advertised on the last HN Who’s Hiring post.
> If OSM is important to you, there’s an open executive position advertised on the last HN Who’s Hiring post.
That position is only for the "OpenStreetMap US" local chapter, not OSM globally. OSM is quite large outside the US, so if you get that job, don't expect to be able to shape the OSM community & project, which can be like herding cats...
The best way to affect change like this in OSM, is to become active and respected in the project. It's quite a do-ocracy.
They certainly would have a lot of influence within the OSM US org, and amoung US companies who do OSM stuff. But the OSM community isn't as big in the US (per capita, or in raw numbers) compared to other countries.
The technology would be available already, what is needed is the data on a competitive scale. Google buys a lot of high resolution aerial imagery. A free project has no chance against that.
The 3D buildings are generated using photogrammetry, which needs aerial photos taken from multiple angles. Satellite imagery doesn't work for the sort of approach they're using. This is why it's mostly cities and important landmarks: it's very expensive to fly the planes to capture that imagery.
Photogrammetry is not rocket science and recent smart phones scan photos into 3D. Provide some Exabytes server space and start a movement with people taking city selfies ;)
Nice. Is there a simple explanation somewhere of what the different bands mean? E.g. I'm looking at my place now and we have this spot on our property where the vegetation is quite different and this exact spot shows up clearly on the NDVI layer. Houses etc do as well so could be temperature?
NDVI is a well established vegetation index that takes advantage of the relatively unique spectral qualities of vegetation, in that it absorbs large amounts of red light and reflects large amounts of infrared. It is the normalized difference of a sensor's near infrared band and its red band. There are numerous band combinations available to highlight certain materials within a scene.
Here is a link describing some additional indices:
ESA's Sentinel fleet offers similar images with much higher spatial but lower temporal resolution, here's a two days old nearly cloud free view of San Francisco:
This is great news for a project I'm currently working on: https://news.ycombinator.com/item?id=17237445. I'll be sure to look into this and see what's new compared to what's already out there. There is a lot of public satellite imagery.
That said, I wouldn't be surprised if this isn't actually the first surfacing of this particular dataset. My experience is that most data of this kind (satellite, land station, oceanic/marine, balloon, radar, etc...) is already available online. It's unfortunately mired in a sea of complexity, filled with shifting departments, confusing acronyms, funding depletion and musical chairs.
Here is my favorite example. The NOAA works directly with NASA for much of this kind of data, and stewards it online here: https://www.ncdc.noaa.gov/data-access. Observe that the URL indicates the department is the NCDC (National Climatic Data Center). As it turns out, the website informs us that the NCDC no longer exists, and is now known as the NCEI (National Centers for Environmental Information).
But let's put that detail aside. Now we go to the page listed for satellite datasets: https://www.ncdc.noaa.gov/data-access/satellite-data/satelli.... Lots of acronyms - acronyms for everything! Let's try the OISST (Optimum Interpolation Sea Surface Temperature) dataset: http://www.ncdc.noaa.gov/sst/. Whoops, that 404s. It also doesn't tell us what the acronym stands for. Fortunately we can find the canonical page by Googling for it.
Okay, let's try another dataset back on that first page. How about the SSM/I-SSMIS (Special Sensor Microwave/Imager and Special Sensor Microwave Imager Sounder): https://www.ncdc.noaa.gov/ssmi? Well, we're told we have to go to http://www.class.noaa.gov/ to get that data. Now the first thing we see on that page is an "urgent notice" that URLs are breaking, and anyone using the current ones (for e.g. FTP access) has to migrate to a slightly different URL with a choice of subdomains appended. Moving past that initial notice, if we actually try to find the SSMI dataset, we find out that it's actually listed as the DMSP (Defense Meteorological Satellite Program): https://www.bou.class.noaa.gov/saa/products/search?datatype_.... Now to download this data, we're presented with this wacky interface: https://www.bou.class.noaa.gov/saa/products/psearchDMSP.
This is basically why I decided to work on my current project to consolidate, document and organize all of this data in a public search engine. It feels like so much climate data is only public in the absolute minimum sense of the word, because trying to access it (or even make sense of all of it) transports me back to 2006. There's a very real sense that the right hand doesn't know what the left is doing here. Clearly these organizations are aware that accessing the data is difficult, and they must have good intentions by going through the effort to document it and host it in FTP servers. But there are so many conflicting names and acronyms for the "stewards" for overseeing this or that data, which redirect you to various "portals" for searching it, which tell you to add data to your "cart" to "order" in some sort of contrived interface. It's maddening.
Well, there ends my rant for the day. If anyone has any good ideas for working on this, please feel free to get in touch.
The GCMD holds more than 35,000 Earth science data set and service descriptions, which cover subject areas within the Earth and environmental sciences. The project mission is to assist researchers, policy makers, and the public in the discovery of and access to data, related services, and ancillary information (which includes descriptions of instruments and platforms) relevant to global change and Earth science research. Within this mission, the directory also offers online authoring tools to providers of data and services, facilitating the capability to make their products available to the Earth science community. In addition, citation information to properly credit data set contributions is offered, along with direct links to data and services. As an integral part of the project, keyword vocabularies have been developed and are constantly being refined and expanded.
I have. I think it's definitely a step in the right direction. But I have a few observations specific to EarthData Search:
1. It places a huge emphasis on visualization. The interface is extraordinarily opinionated. The first thing you're presented with a map which is, in my opinion, completely superfluous. The intention seems to be an intuitive interface, but I would not call this intuitive whatsoever.
2. Search results are not organized in a transparent way, and metadata is not presented consistently. There is no single format for the principal name, stewarding agency, associated department(s), citation information, dataset size, category, temporal resolution(s), spatial resolution(s), date range, geographic range, update frequency, access information, etc for datasets in each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly inform you of the sources used in various queries and requests.
3. The search experience is very slow. This doesn't seem to be intrinsic to how much data is being searched - rather, it seems to be an artifact of how results are being presented. It's not actually reading over the datasets themselves, it's just returning dataset metadata. But search queries for generic climate terminology can take tens of seconds to load, with no progress indicator.
4. This site still suffers from what appears to be a systemic problem of maintainability: it's easy to trigger "Error retrieving site preferences There was a problem completing the request" errors (on latest Chrome, latest macOS). An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is currently experiencing download errors and there's explicitly no estimate for when this will be resolved.
EarthData Search is one of the websites I was thinking of when I said "portals" - this is clearly a well-intentioned effort, and it's a lot better than browsing undocumented FTP servers. But for climate scientists (or cross-pollinated researchers, such as statisticians and "data scientists"), a much more static interface with a common format for searching and documenting all of this data would be much more useful.
I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself work on processing of level 2 and 3 data products. A key point that needs to follow your comments is that various data products that are often purpose made for the goals of the people that use them... and that interpreting said data requires understanding the choices made, although this can be found in the papers that document the products. What this means though, in the long term, is that there's often more than one data-set for each parameter and they're all horribly documented.
[most data of this kind] .. is unfortunately mired in a sea of complexity, filled with shifting departments
NOAA works directly with NASA for much of this kind of data, and stewards it online
You seem to have a good grasp of the current state of "climate data" in the US (and Europe). It's hard for scientists to simply get enough money to launch a satellite and that money usually doesn't include a budget for "packaging and organizing data to be published". For the most part, NASA provides "official" data products for its satellites and then enters into partnerships with other agencies like NOAA (and other scientists) to use and host these data products. Often this means refining the NASA products into something more useful. Every grant I've seen that's creating/taking its own have had one line that says "will put data up on ftp" and nothing more.
Also, as you're likely quickly finding out... the storage and processing requirements are often "enterprise class". For quick reference, the level 3 product I work on (EASE-gridded daily means) is roughly 30TB. This is one product from one satellite for one type of observation. The level 1 data that's at native time/spacial resolution is something like 3TB per day. There are several agencies who work on a similar product with different goals, from the same level 1 data.
I can say in no uncertain terms that you're unlikely to see anyone organize these data sets any time soon. I admire you for attempting to tackle it. I'm not sure what your mission is but if I were to advise you myself (having seen 30 different agency sponsored hosting sites come and go) your time would be most wisely used in two ways:
1) If you're really determined to host the data yourself... find and choose what you think are the most important products from each satellite, archive those products and deeply document the sources (i.e. papers published and up-to-date links to related products that you aren't hosting).
2)Or what I think needs to be done: Create a web portal... like... a really good web portal. This doesn't exist, which is obviously the motivation for what you're doing. Something that cleverly organizes and documents the different types of climate data and the places that you can find them and the places/papers they're documented in. In my humble opinion, this is what needs done most. NASA/NOAA/everyone else will gladly host the data but they're never going to create a "climate data portal" that documents and catalogs data from the other agencies and sources.
Well, not never... they've tried. It's just really difficult, these are the ones I know:
the right hand doesn't know what the left is doing here
It's more so that the right hand has nothing to do with the left hand. Each agency, each satellite, each group of scientists, is totally independent of the other. While sometimes there's noble leaps taken to work together... there's not much incentive financially or otherwise. Not to mention that each team is so specialized that they often know nothing about the other 98% of the data. Take me for example, I've helped work on a very popular data set for a specific part of the earth system... I don't think I've ever once looked at ocean data... I don't even know what's out there or how good it is or where you find it.
Thank you for writing the above well-informed comment.
One important observation at the top of the comment is: "data products ... are often purpose made for the goals of the people that use them" -- and this explains a lot of the atomization and overlap among these data.
You can see this effect across NASA science data (i.e., not just Earth). A group of investigators sees the potential to make an innovative new measurement, and develops the instrument or retrieval system to get it. This can be a decade-long process.
The best people to talk about the features and limitations of that data are, clearly, that team. And they are generally quite approachable for other in-discipline specialists at the usual conferences and workshops. There will be published documents as well, but they are best if you already know what you want.
But these methods do not help outsiders! It can be hard for out-of-discipline people to discover what data (among what's available) would be helpful or applicable for their purposes. It can be very tricky - "you can't use data X for purpose Y because we filter effect Z out of X, and Z is critical to doing Y."
One other general point - related to the data maturity pipeline. NASA is chartered with developing, exploiting, and archiving novel measurements. NOAA is chartered with operational analysis and use. So there is a handoff needed there, and a difference in emphasis.
This is all very useful information for me, thank you. Would you be willing to send me an email so I can ask you further questions? I'm in the process of reaching out to other climate scientists as well.
Google Earth even has built-in viewshed generation which uses their 3D model data (works great), but unfortunately the 3D model layer only seems to exist for fairly populated areas (probably because it's computationally expensive) - so doing RF planning in rural areas requires more traditional legwork.
So it would be cool for a project to provide similar functionality for OpenStreetMap using this public satellite data. OpenStreetMap does have 3D modeling support but last time I looked it seemed pretty focused on buildings and people hand-editing the data.