This page has become my GIS therapy. Every time I encounter something stupid about ArcMap (every 40 seconds?) I come here and write it down. My rant that used to be mid-page is now at the top so I can access it easier. If you enjoy a good ESRI rant – read on! Scroll past the bullets for the original Android/GIS article.

ArcMap is excellent at organizing data and allowing it to be manipulated, but nobody in the foreseeable future is going to ask ESRI for directions to the nearest coffee shop. ESRI should, in my opinion, spend the money they sink into superfluous garbage on refining the interface and usability of ArcMap. Some suggestions:

• What about the “Reshape Feature” tool and how it doesn’t work at all? “The task could not be completed?” No kidding. Users have to double-click, pick a point feature and move it a tiny bit, then click outside the feature and select the “reshape feature” tool again. This is how we have to use a tool? By fake using a different tool first? This is a huge bug that really stinks.
• Why when users attempt to save a multi-page .pdf for the second time does the program crash? It does this EVERY. SINGLE. TIME. Save a 9 page set of maps with only topo features displayed – works fine. Turn on geologic layers and attempt to save again with a different name? CRASH. “Would you like to send an error report?” NO. Damn you. I’d like to throw Jack Dangermond in a very cold and shallow lake. I’d like to shoot video of the incident and then edit it in Adobe software that actually lets me change a name and save it again.
• Add drop shadow capability.
• Let me create editable layers.
• Make custom colors easier – make it so I can save a color palette
• Opacity slider for graphic elements
• Make it so I can add a field during an editing session already.
• Why don’t my fields retain the order I put them in? If I remove a layer and add it back I have to reorder the fields? Ridiculous.
• Remember my .pdf export settings so I don’t have to select “all pages” EVERY.SINGLE.TIME.
• You know how in the “Attribute Table” users can select an attribute column and drag it over a few spaces to reorganize? Why can’t I Ctrl + Click multiple columns and drag them?
• Have you devs ever done a spatial join? No. You haven’t. Know how I know that? Because after a spatial join there’s like 3.4 million fields that no user ever asked for. Sum of the Northing?! Are you kidding me!? Maybe you could add an “Easting divided by Pi” field? That’s not the worst though – the worst is that I can’t ctrl+click on multiple columns to delete more than one at a time! Seriously? You give me 19 columns I never asked for and then force me to delete them individually! Is there a character lower and worse than Satan? That must be the guy that came up with this. Every time I have to delete individual columns he chuckles to himself and forces servants to smother kittens and massage his cloven hooves.
• On every tool dialog the “import” field is at the top and the “output” field is under it, but the “Grid Index” tool is reversed? It has the “Output” field at the top! Aren’t there companies that are paid to “proofread” software? Can you hire one?
• Make it crash only 4 times a session instead of 11. This would better serve core users than offering some half-baked “Crowdsource Polling” – that’s what Twitter is for. Or Facebook, or any one of a hundred better standalone apps. ESRI should worry less about diversifying and more about its core business.

 

I’ve been following the development of GIS solutions for Android for a while now, and change comes slow. TL/DR – Android is getting better at data collection with more and more options, but the “old” way of Pathfinder Office, TerraSync, and ArcMap is still a better solution for professional applications.

 

Geographic Information Systems (GIS) offer myriad options for collecting, symbolizing and presenting data. Global Positioning System (GPS) data logging units are ubiquitous and range from simple, tiny tracking units, to wearbles like watches and fitness trackers to small handheld units used for outdoor recreation to professional grade units with sub-meter accuracy, steep learning curves, and astronomical prices. While consumers used to have to seek out and purchase a GPS unit as a separate piece of (expensive) electronic equipment, most consumers now have a fairly accurate GPS data logger with them at all times in the form of the modern smartphone.

Smartphones are capable of much more than just logging positions though. With an included camera and access to the internet smartphones are becoming less about the way we interact with our friends and family, and more about the way we interact with our surroundings. Apps like TripAdvisor, Foursquare, and Yelp! use GPS to log where we eat and photos of our appetizers are available instantly to others. Users add comments and videos about tourist attractions, hotels, and outfitters for others to see and refer to. Multiple millions of people have cell phones and are intimately and instantly familiar with touchscreen interfaces and the simple design language of a maturing internet. Young children can download new applications and figure them out in just a few clicks.

Environmental and resource management agencies and contractors rely heavily on GPS data too, but they’re stuck in the past. The exact opposite of intuitive and simple – professional data collection and mapping techniques require weeks and months of training to become familiar with. Environmental surveys rely on hardware that was outdated when it premiered in 2008 and software that hasn’t been relevant for even longer. Touchscreens on supposedly “high-end” units are resistive instead of capacitive (requiring a stylus and some pressure), they lack modern colors, are not adaptive to light levels, and have such poor contrast as to make viewing in sunlight almost impossible. Trimble data loggers that cost $6000.00 – $8000.00 offer less features than a smartphone that retails for less than $100.00. Not only is the hardware prohibitively priced, the software is too. Licensing fees to Pathfinder Office, Terra Sync, and ESRI can run into the multiple thousands of dollars on the first day of use. To be fair, the GPS receiver in high-end hardware is more accurate, often achieving sub-meter accuracy without post processing. But when 3-5 meter accuracy is sufficient for a survey, why don’t we use something more familiar and more user-friendly to collect and categorize data?

The answer is simple. We don’t use smartphones instead of Trimbles (to single out the big guy) because smartphone programs or apps stink at data collection. Writing a custom app is difficult (next to impossible in my experience) and existing apps don’t do what TerraSync does. It’s tough to compete with a technology that’s been around as long as Tribmle has, and that introduces another factor – stubbornness. We cling to the idea that as an agency or company we’ve been using Trimbles and they work well enough, so why should we change? Add to that the complexity of learning and acclimatizing to a new set of software or app, and most are happy to stick with what they know rather than waste time testing a new system.

So came about my idea write an app for Android (my preferred smartphone operating system). We’ve all read about the young kid that created an Android app and he’ll never have to work again (or at least he won’t have to pay for his own college) and thought “Hey, that sounds great – I bet I can do that.”  Google has strong resources and seems genuinely interested in helping people in general learn to write and be comfortable with code. It’s where I landed when I wanted to create and app.

I started simple. Google (Android really) has an entire page devoted to developers where people can start learning how to build apps. I had to first download and install Android Studio.

Android Studio is what’s called an IDE or Integrated Development Environment. Integrated Development Environment sounds fancy and complete, right? Like it’s “integrated”. Things that are “integrated” a part of a larger whole, right? Turns out, what’s “integrated” in Android Studio are things I’d never heard of – it includes a compiler, an interpreter, a class browser, an object browser and other things. Those things are integrated into Android Studio, sure, but just installing Android Studio is only half the battle. More like .25 of the battle. Users have to install tools from the Software Development Kit (SDK) and then, if users want to modify existing apps to suit their own needs they must install the Java Development Kit (JDK) and create a GitHub account and find and download a GUI (graphic user interface) for GitHub and then link GitHub to Android studio and figure out how to pull remote versions to local machines for modification. But that comes later. Just building the most basic app proved a fair challenge for me. After installing everything all that’s left to do is dive in. I started out at the most basic level.

I followed the steps to the walkthough to create the most basic app.

The instructions take users from naming to selecting devices the app will run on, and users can select the oldest SDK users want the app to run on.

According to the software if you choose API (Application Program Interface) 4.0.3 (named Ice Cream Sandwich) then the app should run on about 94% of active Android devices. Next we add an activity.

Then we customize the activity:

And click “Finish” to complete the project set up. After the project is set up, we’re presented with the Android Studio interface.

And our app is created and “finished”. At least it’s finished in the sense that it says “Hello World.” That’s all it does. It’s a little anticlimactic. It took me 15 minutes to figure out where to type to change the “Hello World” message. I finally found it.

I made it say something a little more in keeping with the way I felt at this point. I made it say “I don’t have time for this nonsense.” Which, I guess, is only partly true. To be honest, I like computers. I’ve had almost every version of Photoshop and I’m comfortable with a vast amount of it. I actually enjoy altering and adding functionality to the data dictionaries on our Trimble units. I’ve got my own website (obviously) that I created from WordPress using only online resources. I feel like I’m pretty good with computers but the interface and possibilities in Android Studio overwhelmed me almost instantly. Just switching windows leads from one incomprehensible window of code to another.

This isn’t to say I couldn’t figure it out – just to say I realized almost immediately this was going to take far longer than a few Saturday evenings watching tutorials on YouTube to figure out.

I decided to tackle it anyway.

I created a GitHub account and installed a GUI and connected my Android Studio account to GitHub and figured out how to pull a version of a GPS tracking program to my computer for me to alter. I’m embarrassed to admit this took more than a few hours, several tries and many YouTube tutorials. This served only to reinforce the notion that “I don’t have time for this nonsense.” As soon as I viewed the code for an app called GPSLogger I gave up.

The code was complex and foreign. Altering this code to suit my needs was going to be like learning calculus. In Latin. I’d need a much more comprehensive introduction and time to be able to figure this out. While I’m sure it’s possible to create a completely custom app from scratch for a specific application, it’s far beyond my current skill/patience level.

Being in this deep though made me think. People create stuff like this all day, every day. Granted, not people like me, but people do it. Economies thrive on it. There has to be an app out there that will be able to do what I need it to. So I set out to find that app and alter it to suit my purposes.

My first stop was the obvious one: ESRI. ESRI’s website claims users “can be up and running in minutes with ready-to-use maps, apps and templates.”

The ESRI walkthrough seems easy enough; in fact it lists just a few steps:

1. Create new app
2. Pick style
3. Select map
4. Add widgets
5. Configure attributes
6. Preview and launch

It sounds promising. So much so I figured I could do it and have an app ready to go almost immediately. I gave it a shot.

I selected “Create” and “App” then selected “Using a Template” and began the process. I was presented with my template choices:

I was initially concerned by the terminology ESRI uses on this page – The title of the popup is “Create a New Web App” – what’s a “Web App?” How is that different than a regular app? Available options are:

• Build a Story Map
• Collect/Edit Data
• Compare Map Layers
• Explore/Summarize Data
• Make a Gallery
• Map Social Media
• Provide Local Information
• Route/Get Directions
• Showcase a Map

Sidebar/Rant Warning: This is where we get to what I think is one of the problems associated with ESRI and every other company grown to the “too-big-for-its-own-good” stage – they become myopic. They think people will want to use their software for everything. Just like Tarantino didn’t use Windows Movie Maker for “Kill Bill” trust me ESRI, I can “Provide Local Information” to more users with a dozen better apps. We don’t need another (poorly-written) one. If I want to “Route/Get Directions” I’ll ask Google, not ESRI. ESRI products are really good for planning professionals and management agencies and they excel at conquering mountains of information. ArcMap is excellent at organizing data and allowing it to be manipulated, but nobody in the foreseeable future is going to ask ESRI for directions to the nearest coffee shop. ESRI should, in my opinion, spend the money they sink into this garbage on refining the interface of ArcMap: Add drop shadow capability, let me create editable layers, make custom colors easier, make it so I can save a color palette,  make it so I can add a field during an editing session already, remember my .pdf export settings so I don’t have to select “all pages” every. single. time. You know how in the “Attribute Table” users can select an attribute column and drag it over a few spaces to reorganize? Why can’t I Ctrl + Click multiple columns and drag them? Have you devs ever done a spatial join? No. You haven’t. Know how I know that? Becase after a spatial join there’s like 3.4 million fields that no user ever asked for. Sum of the Northing?! Are you kidding me!? That’s not the worst though – the worst is that I can’t ctrl+click on multiple columns to delete more than one at a time! Seriously? You give me 16 columns I never asked for and then force me to delete them individually! Is there a character lower and worse than Satan? That must be the guy up with this. Every time I have to delete individual columns he chuckles to himself and forces his servants to rub his cloven hooves. On every tool dialog the “import” field is at the top and the “output” field is under it, but the “Grid Index” tool is reversed? It has the “Output” field at the top! Aren’t there companies that are paid to “proofread” software? Can you hire one? Make it crash only 4 times a session instead of 11. This would better serve core users than offering some half-baked “Crowdsource Polling” – that’s what Twitter is for. Or Facebook, or any one of a hundred better standalone apps. ESRI should worry less about diversifying and more about its core business. </rant>.

I saw the “Collect and Edit Data” template option, so I plowed forward.

The template options available under “Collect and Edit Data” are:

• Basic Viewer
• Crowdsource Manager
• Crowdsource Polling
• Crowdsource Reporter
• Edit
• Find, Edit and Filter
• Geo Form
• Information Lookup

Geo Form sounds most promising and seems to be the only one that will, perhaps, accomplish what we want it to.

Select “Geo Form” and fill in the blank fields, adding some “Tags” or it won’t let users proceed, and select “Next.”

We get a little more information on what the GeoForm is going to do on the next screen:

We learn “GeoForm is a configurable template for form-based data editing of a Feature Service. This application allows users to enter data through a form instead of a map’s pop-up while leveraging the power of the Web Map and editable Feature Services. Use the following steps to customize and deploy your GeoForm.”

Users are encouraged to “Select A Webmap” and we’ll select the default for our test app – it’s called “Rate the GeoForm”. Select “Next.”

On the next screen we’ll opt to leave “All Layers” editable:

Next we’ll fill out some form details:

Next we get to choose the fields that will appear on the form:

Notice there are 5. 5!? My current data dictionary has about 40 fillable attribute fields.

If users play around a little, it becomes obvious very quickly that there isn’t going to be a lot of customization going on here.

On the next screen we get customize the look of the interface, and here ESRI offers quite a few options. I scrolled through them all and selected “Lumen.”

Next we “Configure the Viewer”

I selected “Comment” so I could view user comments.

The next page offers a few more “Options” to configure:

I opted to change the pin color to red and to change collection coordinates to UTM.

On the next screen we finally get to “Preview” the app we created in all its limited glory:

Functionality is… limited? But we made it through. Move forward through the “Save” and “Publish” options and we’re mostly finished.

Now I have an option to share the form with others… Notice the usage of the word “form” we didn’t actually create an “app” in that it’s a piece of software that users can install on a device – we made a form. We added a tiny bit of functionality to an existing website by allowing people we share this form with to comment briefly on the form itself and then we collect a position and store it with form’s comments.

This is useless.

It’s not configurable to meet my needs and it’s not easy to use. It collects a very limited amount of data about a single source and the data it does collect, while slightly alterable, is worthless without the ability to add more fields and different types of data. It requires a data connection to collect data and can’t create a line or a polygon.

I tried to open it on my phone, it requires a data connection and a login to ESRI’s site, again demonstrating it’s not a real app – but actually just an interactive website.

Once logged in we’re presented with the “app” or data collection area of the website. It asks us to fill in the fields we slightly modified earlier:

It’s actually well-laid out and I like the “Lumen” theme I picked, but again, it really only allows me to collect limited data about a single point – and to do that it requires a data connection.

Scrolling down users are able to see their current location, and even in UTM. Many fitness oriented sites don’t offer the ability to see users’ actual coordinates. Scrolling down further allows users to see a map and a button for submitting the form.

This brings us back to ESRI’s sneaky use of the term “Web App.” It’s not an application that works on its own or allows expanded functionality of a device – in basic terms it’s not an app – it’s a slightly interactive website.

Having struck out on the ESRI side of things, I decided to turn to other apps for Android to see if any were configurable on the level I wanted them to be. Is there an app that allows users to collect points, lines, and polygons easily with lots of associated data, or attributes?

For better or worse, there are dozens and dozens of apps for collecting data. Some of the GIS forum suggested ones are:

EpiCollect
OpenDataKit
AntiMap
GISCloud
Geopaparzzi
Mappt
CyberTracker
GPSLogger for Android
Basic Air Data
PathAwayPro
Collector for ArcGIS
Trimble Navigator
MapIt
Survey123

I’ve downloaded and played with almost all of these, and while each excels at a certain thing, no single app could (sadly) replace the Trimble and TerraSync. While none are perfect, there are some that are better than others. Three that stand out are GIS Cloud, Mappt, and Trimble Navigator. Sadly Mappt isn’t compatible with my phone – only my tablet.

I literally took these apps around the block to see how usable they are in the real world. The results were… disappointing.

GIS Cloud is a pretty good example of where GIS for Android is probably headed. It’s well-thought out and has a 200+ page pdf user manual. The GIS Cloud app allows a tremendous amount of form customization – in fact users familiar with setting up a data dictionary in Pathfinder Office should have no trouble flying through the form creation portion of the website.

GIS Cloud allows for many different types of fields too – radio buttons, drop-down boxes and fillable text (see below). Users can decide if fillable fields are mandatory or optional and can set default options as well. GIS Cloud offers abundant customization options.

GIS Cloud requires an account and a sign-in, but there  is an offline option and information gathered offline will be synced once users are back within service range – cellular data or wifi – it’s up to the user.

In the field it’s relatively easy to use once you acclimate to all the different names for the buttons. “Send” instead of “store” or “save” or “done” confused me for a minute.

But once I got over the confusion, It was relatively simple to fill out the custom form I’d created.

I was able to attach a photo to the point easily and quickly. When users hit “send” the feature is stored or synced depending on user settings.

I hate that time doesn’t auto-fill. The app should pull the time from the phone and fill it without user input. As it stands now, I found filling out the time field very annoying – even the popup clock didn’t have the correct time and a simple “confirm” button, but I had to select the time for each point I created.

In the end though, the most annoying thing was the lack of ability to create a line. The points were easy to create and the form was easy to use and customize. If users could create a line and fill a similar form – then this app would be much more complete.

It would be easy enough to create a series of points with attributes and then use the “point to line” feature in ArcMap. In this sense, GIS Cloud comes close to a viable alternative to a Trimble.

I liked the good interface from GISCloud. Easy to use and free for single person private use, which makes it $590.00 cheaper than Mappt.

Mappt is a seemingly great program that actually appears as if it could almost give ArcMap a run for its money on a very small scale. It has editable layers and you can create lines and polygons and edit and measure them as well. It fell short for me for three reasons;  1. It’s not compatible with my phone – only with my Nexus 7 tablet. 2. It froze up halfway around the block so it only recorded the second half of the trip.

And 3. in order to save my polyline Mappt wants me to “upgrade”.

For $590.00! It’s way to soon to be asking for this much money. I can’t even pull a file off and look at it without paying the fees? Mappt should go to a real trial period – 30-60 days and then ask for payment once users are roped in and comfortable. At least give me a  chance to play around with the file.

This tragic fee cripples Mappt far to early for users to decide whether or not to pay. It’s a shame because Mappt seems to have strong support – they even have a Youtube channel with tutorials and everything.

Last, and with an unfair advantage, comes my reluctant favorite at the moment – Trimble Outdoors Navigator. It has an unfair advantage because I’ve used it a few times before and even relied on it once when the big Trimble died on me unexpectedly. Reluctant because I hate to recommend something that essentially comes from a manufacturer I was hoping to escape.

Trimble Outdoors Navigator has an easy-to-use interface and and even the “pro” version costs only $5.00. There’s another, higher, license level as well, but the “pro” has always allowed me to do what I needed to without upgrading.

Users can select a map view or “layer” over which tracks and points will be symbolized.

I chose Aerial and the images loaded quickly and it was easy to see where I was.

Users simply need to press “Track” and it will switch from “OFF” to “ON” and turn green to let you know it’s recording your movement.

The camera is easily accessible and users can assign minimal data to the photos.

Users can find a surprising amount customization in the menus.

And, gratefully, the app allows users to select UTM as an option.

Users have to go to the “Waypoint” screen to see current coordinates – then you could select to “Cancel” or “Save” depending on your mood. I wish you could see coordinates on the front or map screen – ideally with the option to turn them on or off by simply tapping.

 

At the end I had a fairly accurate and easy to see route I’d mapped. Sure, it doesn’t have 40 fields of data associated with it, but it at least proves I was out there and demonstrates where I went. Sometimes that’s all users need.

Saving and accessing the stored files is straightforward.

Files are saved in a folder and can be emailed or “shared” according to user preference.

For comparison I used a Trimble Geo XH sub-meter unit alongside my phone and tablet to collect some data as I walked around the block. I also used a Garmin Fenix 3 watch. I then pulled the data off of the respective devices and wrangled it into ArcMap.

It was interesting to see what a pain the data conversion was. In the end, familiarity won out over almost anything else, and I thought the Trimble Geo XH was easiest. The Garmin watch took a surprising amount of time to coerce into ArcMap because recently (and apparently without my knowledge) Garmin dropped the .gpx format and began using a new .fit format for files. I had to upload it to Garmin’s Connect website, then download it as a .kmz then import into ArcMap. Too many steps.

The GIS Cloud App, although only logging a single point, took an enormous amount of time. I had to upload it the GIS Cloud site, download a .csv, open it in Excel and save it and create a feature from a table. It took me several tries to get the projection right.

The Trimble Outdoor Navigator allowed me to download to .gpx to my phone storage and I simply shared it with myself and used ArcMap’s “GPX to Feature” tool. It was relatively easy.

Below are the three lines I created. Based on having walked the route myself, I know the Trimble Geo XH is the most accurate – as I would expect it to be. I was surprised by the shift to the east of my Garmin watch, but I’ve never compared it to anything else before.

My phone, (the blue line below) was right in the middle. And of course without paying $590.00 there won’t be a line from the Mappt app to compare.

Above we can see the lines are relatively close together, although something caused the blue line (Trimble Outdoor Navigator app on my phone) to create a section of two lines for some reason.

Again, based on having walked this myself, I was on the west side of the road fairly near the sidewalk. The Trimble Geo XH wins for accuracy hands down.

In short terms:

GIS Cloud is super cool and easily customized, but can’t collect a line.

Mappt looks great and seems powerful but it crashed and is too expensive.

Trimble Outdoors Navigator is easy to use and creates good data, but lacks the ability to customize it.

The Garmin watch is easy to use, super small and light (it’s a watch, right?) and it syncs with my phone and offers a lot of data for fitness, but for an environmental survey all it would serve is to prove you were there. It’s also a pain to retrieve the data.

The Trimble Geo XH, with it’s crappy screen and ridiculous price is incredibly capable. It’s accurate, reasonably fast, and can store so much data it makes the others look like toys.

While many apps perform their functions and are able to collect data after a fashion, none of them are as powerful as the Trimble unit. They’re also nowhere near as accurate. Many times it’s the little quirks that take some getting used to – the way we populate attributes for example, or choosing to collect lines or polygons or points. The insane amount of customization available in Pathfinder Office is certainly a point in its favor over Android apps. Another monumental point in favor of the Trimble is, surprisingly, lack of connectivity.

If there’s one area where GIS applications for Android fall truly short – it’s security. Most of these apps read your phone state, carrier, identity, and some access your call log and messages, and even have access to your photos. When dealing with any sort of sensitive resource, such as archaeological or paleontological finds, I’d be hesitant to log them on my Android device for fear of a data breach/leak. If the creators of the app have access to your photos and messages, it stands to reason they have access to the GPS data you collected too. In the case of GIS Cloud, collected data is synced online and accessible as soon as you reach a reliable signal again. While great for many applications, I can’t imagine the BLM being excited that coordinates to sensitive paleontological or archaeological  finds are available or floating around out  there.

In my opinion, the best solution would be a program like Pathfinder Office that allows users to create a data dictionary and export it to Android. It would run locally, only on the device it’s installed on, and would export lines and points and polygons only when told to by the user. It would connect to a button-sized sub-meter receiver that’s clipped to a surveyor’s hat.  I believe we’re getting closer and closer to that future, but sadly, at the end of 2015, it’s not here yet. Android GIS collecting options are getting better but if I were conducting a sensitive survey with confidential data, based on security alone, I’d choose a Trimble over a smartphone any day.