Changing encoding on a dbf file/column

I know, not very exciting, but I thought I'd capture this before I forget.  Having some character encoding issues with a particular dbf file.  Apparently, it was encoded as ISO-8859-15 but everything seems to try to read it as UTF-8...so with Nick's help, came up with a quick way to convert DBF columns from one encoding to another.


How to convert a dbf file from one encoding to another. In this example, only convert the first column (NAME) from ISO-8859-15 to UTF-8 while keeping all other columns the same.

Step 1: Convert DBF file to csv

    ogr2ogr -F "CSV" NEWFILE.csv OLDFILE.dbf

Step 2: run the attached perl script convert.pl sending the new csv file in as stdin:

    ./convert.pl < NEWFILE.csv > NEWFILECONVERT.csv  


Step 3: Convert the newly encoded csv file back to your DBF file

    ogr2ogr -F "ESRI Shapefile" NEWFILECONVERT.csv OLDFILE.dbf


====== convert.pl ==========
#!/usr/bin/perl

while (<>)
{

    chomp; @_ = split /,/;
    my $name = $_[0];
    if ($name =~ /\w/)
   {
        $_[0] = `echo -n "$name" | /usr/bin/iconv -f ISO-8859-1 -t UTF-8`;
    }

    print join(',', @_) ."\n";
}

====== convert.pl ==========

Location & Proximity

Yes, I'm finally back...and I mean it this time!

I've been thinking a lot about the difference and similarity of Location and Proximity. Most of what I've been doing in the last couple of years has focused on Location...latitude, longitude, altitude, place -- a named, specific space. Something that creates a binding with the world.

Pondering the difference between Location and Proximity makes me really want to illustrate the distinction. It is true that you can be in proximity of a Place, but you can also be in proximity of something that is less permanent, something ephemeral - temporary in time or space such as another person or a vehicle or another device.

Sometimes they run hand in hand -- for example when I drive on Mass Pike (and have that angry look in my eye) I come within proximity of the toll booth. For me to use my infamously misnamed FastLane transponder, I need to be in proximity of the toll booth...but we also now know that I was at that Location.

Sometimes location isn't necessary to have interactions. For example while driving around, I notice somebody sharing a song over their (still to come) auto-share audio system, I only really care that I am within proximity. Now, it might be interesting to add location to that so we can create some more social verve and track what songs are shared at what location and what times are most interesting for fellow listeners.

In fact we use proximity information every day to infer location on our system -- by knowing what signals are proximal to your device, we can infer that you are located near to the signal source -- whether it be a cell tower or a WiFi access point. By knowing that your device is proximal to multiple beacons, we can further refine the device's possible locations. We also use the proximal information to understand when and where beacons themselves are. By understanding proximity, location can be improved, refined, and corrected over time.

But how and when is proximity useful in and of itself? Certainly the NFC activity is very deeply tied with proximity. But location is nearly always an important part of this as well. Proximity is often used when talking about things -- where is the closest X (where X could be pen, defibrillator, beer, etc.). It's also interesting to think about proximity in activities like shopping. I always think of a geiger counter or the 'hotter-colder' game -- I want my phone to start beeping faster as I'm getting closer to my target. When hunting for my favorite toothpaste at the store, I want to know my proximity to my brand for example. Now, perhaps this is just a matter of computing the distance between two locations, or it might be using the signal strength to measure proximity, nonetheless, I don't really care that I'm in aisle 7 and looking at shelf 3...I care that I am near my goal.

But it seems in social activities, proximity could be quite useful. When checking in someplace, I don't necessarily only want to see all of the people in Starbucks (especially since I never go there), but perhaps to see friends or possible friends that are within some proximity of me.

So, while these two concepts of Location and Proximity do have a strong affinity to each other, they are not one and the same and should be treated as distinct entities.

Cheers,

Kipp

2010 And all is well

I seem to have fallen off the writing wagon and would love to get back on it. Tweets just don't seem to satisfy my need and while Facebook is good for many things, I miss being able to actually write a full paragraph and have it somehow seem semi-permanent.

So, with that, I have renewed my intent to write...we shall see how well i do, but I feel confident that it's going to happen!

I'd like to close with a link to one of the authors I respect on both a technological philosophical level as well as from a science fiction perspective --

Cory Doctorow: Writing in the Age of Distraction

20 minutes each day is hereby dedicated to writing.

Cheers all,

Kipp

2009

Okay, so it has been 1 year since I posted...I stink at this.  I'm still at Skyhook in Boston and loving it (although if we could shrink winter, I'd be happier!).

A lot has happened in that one year (other than the global economic disaster, the historic presidential election, etc.), some location highlights:

• Wifi positioning has become an accepted (expected?) method for enhancing location determination
• Millions of devices are running our software
• We grew to over 100M APs and over 400K Cells in our system
• Major GPS vendors have licensed our tech to include XPS with their chips

2009 will:

• see location becoming a first-class citizen at the OS/driver level
  
• see a growth in privacy concerns as the number and types of apps grow
  
• see a huge growth in web based location integration
• see location based context become a focus
• push the capabilities of indoor location to an unprecedented level

I'll be continuing to work with the team to push on these and other fun technology fronts (scalability being another interesting one) as well as working on some more futuristic researchy items related to the above.

On the personal front, probably best to track me on facebook if interested in that sort of thing.

I promise to push myself to keep this one active on the resarchy side of things.

Kipp

Apple and Skyhook

My goodness but it's been a long time since i made it out here. I'll blame it on the work...and with good reason.

We've been pretty much heads down for the last several months preparing for what came out last week. As most of you who read this already know (since I've been crowing about it for the last week) --

Apple announced the inclusion of Skyhook's Wireless Positioning Service in the upgrade of the iPhone and the iPod Touch. Steve Jobs announced this during his keynote at Macworld in San Francisco last week.

USA Today
Video podcast
Boston Globe
Links to pr and keynote
Some photos


among others....the best is to see Steve on stage explaining it -- "Isn't that cool? ... That's really cool!"

So, we're happy to be included in the iPhone, a pretty fantastic phone/computing device. It really does change the way you think about the device -- or at least it has for me. Phone seems like such a limiting term...

Well, that's the big news. A number of other things have been going on that are worth mentioning:

- patent 7,305,245 "Location-based services that choose location algorithms based on number of detected access points within range of user device " was issued and includes me as an inventor
- we continue to hire some excellent people into the team (and we're looking for more!)
- the prospects look good for adding WPS into other devices

Personally, I'd like to be able to spend some time focusing on continuing to improve the system for indoor use. There is a lot of research that has been done on indoor positioning systems that we can leverage in considering a large scale indoor positioning system. It has some great challenges that make it hard, but extremely interesting. Not to say that outdoor positioning doesn't also have some great problems as well, but the game changes a bit when you move indoor.

Not a bad start to the year!

Kipp

Moving and Austria

So we have a new house in Medfield! All of our PODS showed up on Saturday and it looks like most of our stuff made it in it's requisite number of parts, but we'll see when it's all unloaded.

Meanwhile, I took off Saturday to Munich. Arrived on Sunday and had a road trip to Innsbruck, Austria. Lovely town and beautiful weather. I'll try to get some photos online today if possible.

Attending the UbiComp conference for the next couple of days. I've found at least one person that I know from GT (Gregory Abowd) and likely will find a couple others along the way.

Opening keynote is coming up Antonio Calvosa from Ferrari...should go fast :-)

Home

Nearly there, we close on the house in mere hours (http://www.flickr.com/photos/22868073@N00/sets/72157601442689857/). Looking forward to getting out of our little tiny one room hotel suite!

We should be able to take possession immediately and start evacuating from the hotel. The PODS are scheduled to show up tomorrow, so I'm hoping to be able to get some of the basics (mattresses) out before I fly off to Europe. Yup, me being me, I fly off to Munich tomorrow afternoon to leave Pam to get us moved in. Aren't I sweet?!?

I'll be attending UbiComp which is held in Innsbruck, and then off to Loca2007 which is in Oberpfaffenhofen, near Munich. Back in the States Saturday evening.

Hopefully once back, I'll be able to find all of my things (like the card reader for my camera) and get back into the swing of things.

Work is going great, the kids are liking their school, but I think it'll get even better once we are in our new place and can call it home!

Kipp