Windows Keyboard Shortcuts every GIS Professional Needs to Know

These keyboard shortcuts in particular make my life as a GIS Professional so much easier.  I'm talking every single day I'm at work (and even some days when I'm not at work).

Understanding Semantics and Ontologies in Spatial Database Applications

To understand the concept of semantics in spatial database applications picture this scenario: Joan has been tasked with mapping points of interest in her neighborhood.  Before mapping, she creates categories for these points of interests.  She chooses restaurants, schools, clothing stores, gas stations, parks, dry cleaners, etc.  For each of those entities that exist in her neighborhood she drops a point on the map and assigns the appropriate category.  Across town, John is sitting in his office and is working on creating a major point file for the whole county.  John has a large task and creates broader categories like commercial, recreation, industrial, education, etc.  Joan and John both put a point on the High School but Joan categorizes it as a school and John marks it as education.  This scenario provides a very general understanding to the concept of semantics, meaning “there is no right or wrong description, just different meanings for different purposes for different people” (Hunter, 2002, p.85).  Let’s say that Joan and John were tasked to merge their point of interest and major point file to create a searchable database to be published on the web and accessible to everyone.  This would give an opportunity to introduce the concept of ontology.  Instead of breaking down one of their databases to conform to the other, Joan and John “could integrate several ontologies so that [they] in fact create a new formal description of how separate databases compare to each other (without actually changing the nature of the databases) (Hunter, 2002, p.86).  Integrating ontologies would allow them to keep both of their databases as is and provide a seamless searchable database to the end users.

In the example above Joan is a citizen who wants to map out points of interest in her community for informational purposes while John in a mapping professional who needs to map major points in the county he works in for an upcoming project.  Citizens and professionals represent two different types of “information communities”.  A citizen of the community, visitor to the area, or a potential citizen wants to see places they can eat supper tonight or go shopping.  Planners, Realtors, Administrators and other professionals may only need to identify and analyze commercial corridors in the area.  Every information community is looking at the same spatial entity but interpreting and analyzing it in very different ways.  When performing a search a hungry student may enter restaurants while a city planner may enter commercial buildings and they would both get the same results.  This reinforces the concept that “ontologies can be used to establish agreements about diverse views of the world and consequently carry the meaning of the original ideas that are embedded in the representation of geographic phenomena in the human mind” (Fonseca, 2002, p.146).  

“To adequately represent the geographic world, we must have computer representations capable of not only capturing descriptive attributes about its concepts, but also capable of describing the geometrical and positional components of these concepts” (Fonseca, 2003, p.356). Many times the nature in which points of interests in a community are used determines how they should be represented on a map.  A realtor or potential citizen may only need a single point to determine how many schools are located within a mile of a house.  A student or visitor to the school would like to see the building footprints on the school campus to determine where they would need to go.  The school district administrator would like to see the campus boundaries in order to determine what the acreage is.  These three examples are of the same spatial entity (a school) but represented and understood in three very different ways.  Identifying and understanding semantics is a good place to start in order to determine and integrate ontologies into a spatial database.  This is easier said than done, but as our world becomes more connected we find the need to do this more important.    

References

Hunter, G.J. (2002, March).  Understanding Semantics and Ontologies: They’re Quite Simple Really – If You Know What I Mean!  Transactions in GIS 6(2), 83-87.

doi: http://dx.doi.org/10.1111/1467-9671.0009

Foncesca, Frederico, Davis, Clodoveu, & Câmara, Gilbert (2003, December).  Bridging Ontologies and Conceptual Schemas in Geographic Information Integration.  Geoinformatica 7(4), 355-378.

Retrieved from http://link.springer.com.libproxy.usc.edu/article/10.1023%2FA%3A1025573406389

Foncesca, Frederico, Egenhofer, Max, Davis, Clodoveu, & Câmara, Gilberto (2002, September).  Semantic Granularity in Ontology-Driven Geographic Information Systems.   Annals of Mathematics and Artificial Intelligence 36(1-2), 121-151.  Retrieved from

               http://link.springer.com.libproxy.usc.edu/article/10.1023%2FA%3A1015808104769

Extracting Census Data from American Factfinder to Use in ArcMap

Well, that escalated quickly.  Introductions to American Factfinder. 

I'm ready to get down to business because I sit a computer for 7.5 hours a day and I learn a lot of cool tricks and methods for getting stuff done.  Most of the time I celebrate these accomplishments with a silent fist pump at my desk.  With a blog, I can celebrate and share with the world.

Up until a year ago, The US Census American Factfinder website was a mystery to me.  I am here today to show you how to easily extract data from American Factfinder, join it to your GIS data, and view it within ArcMap.

Step 1:   Find Census TIGER/Line Shapefiles (if you don't have them already)

I'm talking tracts, block groups, blocks, etc.  These shapefiles have no meaningful attributes when downloaded, but they do have GEOIDs in their attributes which are super important to have in order to join them to the data later on.  You need these shapefiles to successfully view any kind of American Factfinder Data in your GIS.  If you already have some census GIS layers available to you, just make sure there is a GEOID column in the attribute table.  Otherwise, download from the Census site.

Do this by going to the Census website --> Click the 'Geography' tab -->
Choose 'Maps & Data' in the dropdown --> Click on the 'Tiger Products' Link

-or-

Just click here to get to the Tiger Products Page

There are lots of options when downloading TIGER files.  We are going to keep this tutorial simple and focus on the first option you see on the webpage; the TIGER/Line Shapefiles.  Go ahead and click on the link.  You will find that you can download the current year’s TIGER/Line Shapefiles or you can go back in time and download a vintage year.  Many will find that the current year’s shapefiles work for them, but if you are working with older Census data, be sure to download the correct vintage for that data.  The Census website provides some good documentation for choosing here.

Step 2: Download Census TIGER/Line Shapefiles

When you click the download button you will have two options for downloading.  Choose the ‘Web interface’ option.  From the web interface page select your area of interest.  If you’re new to Census data, ‘tracts’ are your safest bet.  However, if your geographic area of interest is a smaller area, you may want to consider ‘blocks’.  Or, you may want to look into the numerous other geographic area types the Census offers.  Once you select your geographic area type, the website will allow you to pick your area of interest. 

Choose the Web interface option when downloading TIGER/Line Shapefiles

 For this tutorial, I have chosen ‘tracts’ for ‘South Carolina’.  Download and save the zipfile.  Go to the location you saved the zip file and unzip it.

Now open up ArcGIS or whatever GIS software you use and add the shapefile in to make sure you got what you wanted.  Open the attribute data and you will see all the codes and GEOID that will be used in the next step when you join the data.

This is what the TIGER/Line Shapefile attribute table should look like when downloaded from the Census website.  Notice the GEOID column which will be used to join AFF data.

Step 3: Finding Data in American FactFinder (AFF)

Go to the U.S CensusBureau American FactFinder website.  Click on ‘Advanced Search’ and then click the ‘Show Me All’ option.

The best way to think of all the data in AFF is to think of a huge pile of unorganized papers and your job is to find the exact papers you need from that pile.  AFF is the tool you use to narrow down your options by process of elimination.  

You can eliminate the data in any order you’d like depending on what you are looking for.  I usually start by narrowing the search down the specific geographic region I am interested in.  This is done by clicking the blue ‘Geographies’ tab on the left menu.  From the ‘Select Geographies’ menu I would choose ‘Census Tract – 140’ as the geographic type (this would be different if I was using another geographic type).  Then I would select my state ‘SC’ and my county ‘Florence’ and then I can choose between all the tracts in Florence County or select just a few tracts.  Then I click the ‘Add To Your Selections’ button, this takes the query and adds it to ‘You Selections’ menu in the left menu.  Take note of the blue circle with the white ‘x’ that is your selection and it only goes away if you click on that blue circle with the white ‘x’.  So, if you wanted to get data for another geographic location you would need to first delete the other geographic area you currently have selected, otherwise you will get no results.  

You can add more than one selection, so if you are interested in a certain dataset or just want to get incomes or ages, you can narrow it down however you like.  For this tutorial I have selected ‘all Census Tracts within Florence County, South Carolina’ and narrowed it down to just the ‘2010 Redistricting Data SF (PL 94-171)’ Dataset.  

Screenshot of AFF in action.  Notice my selections in the top left corner, if I need to remove a selection I just click the blue circle with the white 'x'.  Beneath that are the blue boxes which I can drill down my selections even more.  The 7 results are tables based on my specific selections and can be downloaded to my computer.  If I need to get more information or see the data in a table before I download it I can click on the blue hyperlink or the 'i' in the about column.

I’ve got seven tables in my results to choose from.  If you click on the information button in the ‘about’ column you can get a pretty good idea of what kind of data is in that particular table.  I’m going to choose the first table, 'QT-PL' ‘Race, Hispanic or Latino, Age, and Housing Occupancy:2010'.  If I click on the blue hyperlinked table name I can actually see the attributes of the data by each tract in Florence County.  This allows me to determine if this data is going to be useful to me before I even download it.  This particular data is going to be useful to me so I can click on the download button from the ‘Actions:” menu above the table.  Clicking the download button here give me a dialogue box with options (leave them as they are).  If I were to navigate back one page, check the box beside the table I wanted, and click the ‘download’ button above it, it would not give me these extra options.  It would just automatically download (which is fine).  Make sure you save the zip file to your computer.  Navigate to that zipfile location and unzip it.  If you want, you can examine files in the folder which are in a Comma delimited (.csv) format.

Screenshot of the files included when I downloaded a table from AFF.  Notice they are in .CSV format.

 Step 4: Viewing the .CSV Data in Excel

So you have your downloaded data, but now you need to pull it into Microsoft Excel and clean it up before you can join it to you TIGER/Line Shapefiles.  

Open Excel (2007 or higher).

From the Top menu click ‘Data’ and then choose the ‘From Text’ option.  This will allow you to import the .csv file into the spreadsheet.  Navigate to the location where you saved your download.  There will most likely be two tables to select from, you need to choose the table that is the name of your dataset followed by ‘_with_ann.csv’.  For example, I would choose ‘DEC_10_PL_QTPL_with_ann.csv’ the other file is metadata (which you may need to use later on, but not now).  

Excel will now give you a dialogue box for importing this data.  Choose ‘Delimited’ and then click next.  For delimiters check only ‘Comma’.  Press next again.  Now be sure that the column data format is set to ‘General’ and then click finish.

Your table should look something like this in the image below.  Please note that you will need to remove extra header rows that include descriptions before you bring into ArcGIS, but you may want to name the headers accordingly so you can easily identify them.  Just be careful not to use any special characters so Arc will accept it and you won't spend half of your day beating your face in your desk.

Screenshot of what your data should look like after you import is from text to your excel spreadsheet.

Step 5: Reformatting the Excel table in order to Join Data to TIGER/Shapefile Lines

Notice the column titled ‘GEO.id’, this is the GEOID number you need, but it needs to be split and reformatted to a 'Text' data type in order to be able to be properly joined to your data in ArcGIS.

Insert a column to the immediate right of ‘GEO.id’ and label is GEOID.  Now highlight the ‘GEO.id’ column (click the top of the column, A), then in the top menu click the ‘Data’ tab and then choose ‘Text To Columns’.  A ‘convert text to columns wizard’ will pop up.  Choose ‘Delimited’ and click next.  Choose only the ‘Other’ option and type “S” in the box for delimiters.  Choose next and under the column data format choose ‘Text’ while being sure to click on the column with no name (the second column).

Make sure the column data format is set to 'Text'.

Click Finish.  Excel may ask if you want to replace all of the values in the column, choose OK.

Now your GEOID column is ready to be joined to your TIGER/Line Shapefiles.

Before going any further, save your work as an excel spreadsheet.  Also, please be sure that there are no invalid characters anywhere in your excel spreadsheet.  ArcGIS will not accept hyphens, periods, parentheses, brackets, or symbols (`!@#$%^&*()+=-|\{}[];:’”><,.?/) you get the point, right?  Clean it up!     

Screenshot of the cleaned up table.  Notice that the description header was deleted, and the headers have no special characters.  Be sure to save this as an excel spreadsheet.

Step 6: Joining the Data to the TIGER/Line Shapefiles in ArcMap   

Once you have everything cleaned, give your excel a final save, close it, and open up your ArcMap.  If you don’t have your TIGER/Line Shapefiles added to the map, go ahead and add them now.  Navigate to your saved excel and add the table to your ArcMap.

Open both the attribute tables, and make sure your newly created GEOID field looks similar to the GEOID field in your TIGER/Line Shapefile.  It does?  Great!  You are ready to join.

In the TOC, right click your TIGER/Line Shapefile layer.  Choose ‘joins and relates’ --> ‘join’.  In the dialogue box choose the GEOID field as the join field for both features.

Join Data dialogue box.  Choose GEOID as the joining field and choose to Validate the Join to make sure everything will join correctly.  Then press OK.

Click the ‘Validate Join’ button to make sure everything is going to work properly when you join.  Once validated, click ‘OK’ and open the attribute table of the TIGER/Line Shapefiles, you will notice that all the attributes are now attached.  This is a temporary join, if you want to make it permanent just right click your TIGER/Line Shapefile Layer in the Table of Contents and choose ‘Data” --> ‘Export Data’.  

Now you are ready to do whatever it is you wanted to do with the Census Data.  That’s another blog entry for me!

How Did You Get into GIS?

I'm pushing 28 years old.  I know for many that may seem young, and I realize that I am still pretty young and I try to enjoy that feeling everyday.  I'll tell you what though, I'm starting to feel old.  I've gotten to that point in one's life where you can sit back and remember the person you were ten years ago and not just hear your mom tell you funny baby stories.

If someone would have told me when I was 17 that I was going to be a GIS Analyst when I grow up I probably would give them that smile that you give people when you don't really understand what they are saying but you want them to think you are engaged, you know the one I'm talking about.  Ten years ago, I was planning a career as an artist and wait...I'm supposed to living in Italy right now and painting beautiful pictures to sell to all my rich friends all over the world.  Damnit, I missed that career goal by a longshot.  Darlington, South Carolina, the city i currently reside in, has that old world feel sometimes and I work in the city of Florence...hello?  That is totally a city in Italy!

Anyway, many times I get asked, "how did you get into GIS?"  Well, I stumbled upon geography in college as a studio art major when I realized I had a strong dislike for my professors and all of the students in the art department.  I was also beginning to understand that once out of college I would need to secure a job.  My love for nature attracted to me the course offerings in the Geography Department and once I had a visit with my advisor he “advised me” to get into the GIS concentration.  I started the GIS program immediately and haven’t looked back yet. 

The thing I admire most about GIS is the problem solving.  GIS involves many types of problem solving.  There is the traditional problem solving you perform with spatial data.  Then there is the problem solving with the actual GIS software when something doesn’t run quite like it should.  Once you have all your problems solved, you are faced with the problem of getting your information out to the masses.  The problem solving never ends with GIS.

I find GIS data easier to handle than other types of data because it has a location that can be related to.  You can put that data exactly where it belongs in the real world and begin to see patterns and answer questions to figure out why something is the way it is.  I have also found that folks who dabble in GIS are some of the nicest, friendliest, and most genuine people I have ever known.  These characteristics of GIS professionals really make a career in GIS that much more enjoyable and rewarding.

I have been working as a GIS Analyst since graduating from Appalachian State and in retrospect I think GIS and I were destined to get paired together.  I don't think it just happened by accident, I think I was bound to get hooked sooner or later, it just happened sooner than later and I am thankful for that.