The Experiment

1. I will publish office hours a week in advance
2. You can go sign-up for slots, including a description and how I will contact you. I’m open to any means of communication as long as they are free.
3. At the slotted time we will begin working on your problem. If it is a problem where it is be worked on asynchronously then I will work on it during the time-slot. If it is something that more requires discussion it will occur during the reserved time.
4. If you are satisfied with the help and would like to pay for it then you can pay me through Paypal, send me a gift, trade anything that seems appropriate to you.

There is more background on my idea around this experiment on my general blog. And you can reserve time through Google’s Appointment feature in Calendar.

I could have taken more classes on remote sensing, but I didn’t. My focus was more on cartography and statistical analysis.

Anyway, recently HOT bought some imagery. I naively thought that I could just run it through gdal2tiles.py and go skipping down the road to digitizing. Well, I was wrong.

I received a lot of advice on what to do with the imagery. For example changing it from 16 bit to 8 bit. Which I tried, oddly it worked on one image and not the other. This turns out was just dumb luck that the stretch was almost right when you did that. Other suggestions involved reordering the bands and things such as that. I tried a bunch of different things, none of them worked.

I did discover that if I applied a standard deviation stretch to the imagery in QGIS then it looked right. There wasn’t a way to save this stretch though directly to the GeoTIFF. I did a bunch of searching to see if there was a manual way to do this and I came across this post on using VRTs to add the minimum and maximum of the bands.

So here is my pretty manual way of doing things. Yes I could have done some automation but I figured it was just a couple images.

First I download the image and open it in QGIS. Then I right clicked on the layer and selected properties. In the properties dialog I selected “use standard deviation” and he “apply.” The image should now look like natural color.

Next I copied the Red, Green and Blue min/max values.  Now with the correct values the key is to feed them in GDAL correctly. Here you are building a VRT or Virtual Format.  So the actual processing doesn’t take place, you are describing it though.

First step is to stretch each band individually:

gdal_translate -b 1 -scale min max -ot Byte -of VRT your_image.tif 1.vrt

gdal_translate -b 2 -scale min max -ot Byte -of VRT your_image.tif 2.vrt

gdal_translate -b 3 -scale min max -ot Byte -of VRT your_image.tif 3.vrt

Next put the bands back together (virtually):

gdalbuildvrt -separate b123.vrt b1.vrt b2.vrt b3.vrt

Now actually create your GeoTiff

gdal_translate -of gtiff b123.vrt my_beautiful_tif.tif

After that you can tile your tif or do anything else you might want to do with it. In my case I ran gdal2tiles.py

gdal2tiles.py my_beautiful_tif.tif

I suspect there may be a better way of doing things than this.  Though after a lot of researching I didn’t find one.  So this is the process that works for me.  Any suggestions?  Please, please, please leave them in the comments!

The premise of the week was to bring individuals from open-source projects together as well as others specializing in documentation in as well to help. A combination of an unconference and then intensive documentation writing ensued. The OpenStreetMap group included Ian Dees, Shaun McDonald and myself from the project. We additionally had Anne Goldenberg, Anne Gentle, Tomi Toivio and Nóirín Plunkett. Having so many people from outside OpenStreetMap was extremely valuable. They were able to help us be far more clear about the materials and see past assumptions we may have made about topics. At the end we had a published book, which can be purchased from Lulu.com. Having a completed work at the end of the sprint was a great feeling. Often after hackathons that is missing for me and it ends up feeling like we just flogged ourselves with some code for a set period of time.

In going through this process I started to think about other topics that could benefit from this type of approach. I started thinking about the topics that we go over repeatedly within the Humanitarian Open-Source, CrisisMappers and Volunteer Technology Communities (for lack of a better name). I really think the community could benefit from a generic guide to issues within the humanitarian technology realm. Topics such as standards, licensing and workflows could be documented. It is impossible to say what exactly the book would cover, because that is determined as part of the book sprint process. What I am proposing though is that it would be greatly beneficial to gather for this process.

Typically a book sprint would happen over roughly a week time with five to ten individuals with different experience on the topic. I’m proposing the same thing. Let’s plan a book sprint and figure out how to make it happen. The idea would be to gather about ten people with different experience, everything from technologists to humanitarian responders to specialists in documentation. This group would get together for an intensive week long book sprint, the goal being to create a manual of the basics of humanitarian technology. Topics that come to mind for me are data licensing, standards, security, privacy and workflows. I’m only one individual and I’m sure topics would change and mold to the importance decided on by the group. The point though would be to create a resource useful within our community.

Does this sound interesting to you? Contact me in the comments or by email and let’s make it happen!

I’m typing this email from Padang, Indonesia in Western Sumatra. Yesterday I hosted a workshop at the University of Andalas to teach them about OpenStreetMap as part of the Humanitarian OpenStreetMap Team’s project in Indonesia. This is the fifth workshop I’ve done at a University in Indonesia. Things went well, much like all of the other workshops I’ve given. Students were excited and picked up the tools very quickly. The one difference? Available imagery.

This region of Indonesia is especially vulnerable to earthquakes and tsunamis, the last major earthquake in Padang was in 2009. In preparing for this workshop series I investigated each area we were going to be working to check for available imagery. For those not familiar OpenStreetMap is allowed to use Bing’s imagery to trace information, this is one of the major sources of imagery for OpenStreetMap. In checking I looked at Jakarta, Bandung, Yogyakarta and Surabaya. All four of these cities had really night, half-meter or so imagery. In Padang? It appears to just be Landsat. I think this highlights the need for imagery to be available beforehand though. If something major happened here I think it is likely imagery would become freely available. Though creating a detailed map through crowd-sourcing? That would take a week or two and be a huge effort from people that don’t know the area as well as locals. So why not map ahead of time? Imagine the power if after an event the map just had to be updated, instead of created?

There are many different ways imagery could be made available. Providers could donate it, I’m not asking for the shot the day before, fancy brand new imagery. What about imagery that is a year old or so? People purchasing imagery are usually looking for something more recent, the older stuff isn’t as valuable. An imagery fund could be created for preparedness. What about doing a highschool style fundraiser where everyone purchases and adopts a square kilometer? When purchasing imagery those doing the purchasing could negotiate for better terms on it. For example sometimes you can add additional licensees without increasing the cost. What if everyone purchasing imagery at least asked if they could license the imagery also for OpenStreetMap to use to trace vectors? Or an even more liberal license if possible.

I’m interested to hear what others think about this and if there are other ideas. Also don’t worry specifically about HOT’s project in Padang, we were fortunate to be able to purchase imagery. As I suggested about licensing above, we also made sure to extend the license to the government and our partners.

Anyway I ended my full-time employment at GeoIQ on May 16th and began transitioning over to working for HOT. I loved working for GeoIQ, my coworkers were wonderful and I really believe in the technology that was being built. However last year after going to Haiti and working remotely on other projects HOT was helping on, I really found a calling. Working on things for HOT is yet to have felt like work and I hope it never does. After the transitioning my real first day of only working for HOT was about June 15th. That day I attended a meeting in New York City with other members from the Volunteer Technical Communities and the United Nations (OCHA to be specific). I thought this was a good kick-off to my new position, because it is one of my goals for myself. To be able to help coordinate with large organizations and other tech response groups so we can all be more effective. I also hope to do some fundraising for HOT so we can coordinate better when disasters do happen. There is a need to have infrastructure, software to better assist with mapping as well as individuals to help coordinate. This items don’t always translate into the need for funding, but I think there are going to be situations where it will.

A couple days later that week I switched to the other role I want to play as well. Coordinate projects and doing field work when possible. On June 17th I left the United States to return to Indonesia. Here I’m working on a pilot project to use OpenStreetMap to help collect information to be fed into risk models. Myself and Jeff Haack are here and mainly focused on giving workshops on using OpenStreetMap, setting up simple tools to allow people to use the data and coordinate between the interested organizations. At the end of the three months we’ll look at how things went and determine with our partners if the pilot can turn into a project for all of Indonesia. Saying all of Indonesia sounds scary, since it is 17,000 islands and many diverse people. Though so far everyone we’ve worked with has been enthusiastic and quick to pick-up the tools as well, so I’m excited to see if it can work. I think if things keep going the way they have so far it will.

So I will be in Indonesia until mid-August, but after that I’ll be a bit all over the place. If I happen to coincide with where you are maybe we can meet-up. As always the best way to keep up with me is on Twitter and hopefully I’ll keep up the HOT Twitter Account better as well.

The combination of the strong geo community in DC plus a great crowd traveling in for the event made the talks really exceptional.  I was really excited to see Sophia Parafina’s talk “WMS is Dead.” Just the title stirs up controversy among the geonerds, controversy is always a good aspect to an ignite talk.

No matter what realm of the geospatial world you come from I think there is something to be learned from her talk.

Javier de la Torre took a different approach to his talk and just showed a really great project.  I’m always surprised when someone hasn’t heard of Old Weather, it is my favorite crowd-sourcing project (well other than OpenStreetMap;)).  The only good source of weather data from 100 years or more ago is ships’ logs.  Old Weather makes it fun to transcribe that information and yes you can become the captain of a ship!

All the talks are listed on the WhereCampDC website, I suggest if you missed the event like I did you check them out.

For anyone interested in open data or new methods of crisis response I recommend you read the whole report.  For others, well a seventy-two page report might seem like a bit much so what might be key is the recommendations.  Key is the idea that information communities (called V&TC in the report) came together after the earthquake in Haiti to help.  Much of this revolved around the sharing and collection of data.  The value of this type of activity to make a difference was clearly shown.  The real question though is how do you make this happen again?  And how can this fit into a framework that makes sense from the perspective of the large organizations as well as the informal communities.

The recommendations suggest a method so that the V&TCs and the traditional humanitarian system can work together. This includes 5 parts:

“1. A neutral forum to surface areas of agreement and conflict between international humanitarian system and the V&TCs.
2. An innovation space where new tools and practices can be explored as experiments, allowing for the failures that are a necessary component of learning new ways of working.
3. A deployable field team with a mandate to deploy the best available tools and practices from the V&TCs to the field.
4. A research and training consortium to evaluate the work in the field and to train humanitarians and V&TCs alike in the best practices for information management in a humanitarian context.
5. A clear operational interface that outlines ways of collaborating before and during emergencies, with agreed procedures for communication, shared standards for data exchange and an understanding of roles, priorities and capabilities”

One of the key differences between technologists experimenting within the humanitarian response space versus a traditional technology start-up is the cost of failure.  Remember in the early days of  Twitter when things went down all the time?  Or other technologies that didn’t work quite right?  Well, usually the risk is to shareholders, in crisis response this failure can translate to lives.    Hence the recommendation of a neutral space, an experimental zone, a deployment team and a way to communicate successes and failures back.  In some ways this is taking what people were already doing in informal ways and giving it formal pathways.

I think much as new technologies replace old in the start-up space there will still be innovation from outside within the crisis response space.  If the innovation space is working in an optimum setting it will be able to operationalize these techniques coming up outside the formal structures.  Though I suspect some will still slip through, since when there is an emergency people will use what is simple, they know and what works.  No matter what I think if these recommendations are adopted it will be a step forward in bringing the formal and informal groups together.

218 different individuals have editing in Japan in OpenStreetMap.  Those editors have edited 389,431 nodes (this includes adding/removing/changing) .  This has all happened through 1,409 editing sessions.

Keep it up everyone!

(Update) Some more details.

bbox = ((42.13, 133.45), (32.81, 142.57))

Starting at 00:00 2011-03-10 going to 03:00 2011-03-16.

I think part of the reason I’ve never really thought about this is that to me data is either authoritative or it is not, but that isn’t what determines how I use it.  One of the previous jobs I had was doing data quality assurance, during that time I found many errors in that is considered authoritative data.  Because of this previous job I have a lot of experience evaluating data.  Everyone should do some evaluation of data before they use it, but it really depends on the type and the source how much they actually need to do.  In thinking about this evaluation I began to think about the process of when we “qa’d” data before it was delivered to customers.  Essentially that QA process was making data authoritative.  What we were actually doing to create the data was not much different from the crowd-source efforts of many groups today though.

There was usually an expert in the field we were creating datasets for who would figure out what to be collected.  The majority of the actual research though was done by interns or recent college grads.  These people scoured the internet mostly, but would occasionally call to confirm information.  Once a new dataset was created or an old one updated it would go through a quality assurance process.  Much of this was running automated tools to look for data that did not conform to what was expected.  For example geocodes in the wrong state/zip, phone numbers of the wrong length and comparing the new data to old when possible.  Then we would manually look for errors in the data.  Once we decided information was “good enough” it would be pushed out to customers.  This is the point where the data became “authoritative.”  In the whole data creation process I estimate each dataset was touched between two and one hundred sets of hands.  Two if the expert created the dataset by themselves and then one person tested it.  One hundred if the dataset had been around through multiple updates that large groups of people worked on.  If you compare this to the number of hands datasets from the CrisisMappers list, such as the recent mapping of polling stations in Sudan or the road data for OpenStreetMap the authoritative data hasn’t even been gone through as thoroughly as the non-authoritative data source.   Also if a mistake was found in the authoritative data source it would have to be updated and go through the whole quality assurance process again.  What really is the difference between the two types of data in this case?

To me, really just that a company put their name on the authoritative data.  I suppose if you are a buyer of the data you would have some recourse if there was a problem with the information.  You could probably ask for all or some portion of your money back, if things went really wrong you could sue.  Though in my examples I believe the crowd-sourced information had been more reviewed, so perhaps the chance of having issue with it is less. I realize this comparison does not work in the case of all datasets, though it is something to think about when discrediting crowd-sourced data. From where did your authoritative data come?

One of the things I made sure to look at was the maps of Haiti there. We looked through the original paper copies of the old U.S. Defense Mapping Agency maps which were used to add the original road names to OpenStreetMap immediately after the earthquake. At the time we used the scanned maps which are still available here, but within all the rows of drawers in the map room were the originals. Another map in the drawer I found to be interesting was a Texaco road map from the 1970s of Haiti which marked all the major roads as well as the Texaco stations there. The library can’t scan copyrighted maps which is why you don’t see this map online (I did snap a picture of it).

70s Map of Haiti from Texaco

Unfortunately the library was closing so we were a bit rushed (one of the reason I neglected to get a photo of the actual map room). Though if you are a geonerd and find yourself in Austin this is a stop I highly recommend.