Changing encoding on a dbf file/column

I know, not very exciting, but I thought I'd capture this before I forget.  Having some character encoding issues with a particular dbf file.  Apparently, it was encoded as ISO-8859-15 but everything seems to try to read it as UTF-8...so with Nick's help, came up with a quick way to convert DBF columns from one encoding to another.


How to convert a dbf file from one encoding to another. In this example, only convert the first column (NAME) from ISO-8859-15 to UTF-8 while keeping all other columns the same.

Step 1: Convert DBF file to csv

    ogr2ogr -F "CSV" NEWFILE.csv OLDFILE.dbf

Step 2: run the attached perl script convert.pl sending the new csv file in as stdin:

    ./convert.pl < NEWFILE.csv > NEWFILECONVERT.csv  


Step 3: Convert the newly encoded csv file back to your DBF file

    ogr2ogr -F "ESRI Shapefile" NEWFILECONVERT.csv OLDFILE.dbf


====== convert.pl ==========
#!/usr/bin/perl

while (<>)
{

    chomp; @_ = split /,/;
    my $name = $_[0];
    if ($name =~ /\w/)
   {
        $_[0] = `echo -n "$name" | /usr/bin/iconv -f ISO-8859-1 -t UTF-8`;
    }

    print join(',', @_) ."\n";
}

====== convert.pl ==========

Location & Proximity

Yes, I'm finally back...and I mean it this time!

I've been thinking a lot about the difference and similarity of Location and Proximity. Most of what I've been doing in the last couple of years has focused on Location...latitude, longitude, altitude, place -- a named, specific space. Something that creates a binding with the world.

Pondering the difference between Location and Proximity makes me really want to illustrate the distinction. It is true that you can be in proximity of a Place, but you can also be in proximity of something that is less permanent, something ephemeral - temporary in time or space such as another person or a vehicle or another device.

Sometimes they run hand in hand -- for example when I drive on Mass Pike (and have that angry look in my eye) I come within proximity of the toll booth. For me to use my infamously misnamed FastLane transponder, I need to be in proximity of the toll booth...but we also now know that I was at that Location.

Sometimes location isn't necessary to have interactions. For example while driving around, I notice somebody sharing a song over their (still to come) auto-share audio system, I only really care that I am within proximity. Now, it might be interesting to add location to that so we can create some more social verve and track what songs are shared at what location and what times are most interesting for fellow listeners.

In fact we use proximity information every day to infer location on our system -- by knowing what signals are proximal to your device, we can infer that you are located near to the signal source -- whether it be a cell tower or a WiFi access point. By knowing that your device is proximal to multiple beacons, we can further refine the device's possible locations. We also use the proximal information to understand when and where beacons themselves are. By understanding proximity, location can be improved, refined, and corrected over time.

But how and when is proximity useful in and of itself? Certainly the NFC activity is very deeply tied with proximity. But location is nearly always an important part of this as well. Proximity is often used when talking about things -- where is the closest X (where X could be pen, defibrillator, beer, etc.). It's also interesting to think about proximity in activities like shopping. I always think of a geiger counter or the 'hotter-colder' game -- I want my phone to start beeping faster as I'm getting closer to my target. When hunting for my favorite toothpaste at the store, I want to know my proximity to my brand for example. Now, perhaps this is just a matter of computing the distance between two locations, or it might be using the signal strength to measure proximity, nonetheless, I don't really care that I'm in aisle 7 and looking at shelf 3...I care that I am near my goal.

But it seems in social activities, proximity could be quite useful. When checking in someplace, I don't necessarily only want to see all of the people in Starbucks (especially since I never go there), but perhaps to see friends or possible friends that are within some proximity of me.

So, while these two concepts of Location and Proximity do have a strong affinity to each other, they are not one and the same and should be treated as distinct entities.

Cheers,

Kipp

2010 And all is well

I seem to have fallen off the writing wagon and would love to get back on it. Tweets just don't seem to satisfy my need and while Facebook is good for many things, I miss being able to actually write a full paragraph and have it somehow seem semi-permanent.

So, with that, I have renewed my intent to write...we shall see how well i do, but I feel confident that it's going to happen!

I'd like to close with a link to one of the authors I respect on both a technological philosophical level as well as from a science fiction perspective --

Cory Doctorow: Writing in the Age of Distraction

20 minutes each day is hereby dedicated to writing.

Cheers all,

Kipp

2009

Okay, so it has been 1 year since I posted...I stink at this.  I'm still at Skyhook in Boston and loving it (although if we could shrink winter, I'd be happier!).

A lot has happened in that one year (other than the global economic disaster, the historic presidential election, etc.), some location highlights:

• Wifi positioning has become an accepted (expected?) method for enhancing location determination
• Millions of devices are running our software
• We grew to over 100M APs and over 400K Cells in our system
• Major GPS vendors have licensed our tech to include XPS with their chips

2009 will:

• see location becoming a first-class citizen at the OS/driver level
  
• see a growth in privacy concerns as the number and types of apps grow
  
• see a huge growth in web based location integration
• see location based context become a focus
• push the capabilities of indoor location to an unprecedented level

I'll be continuing to work with the team to push on these and other fun technology fronts (scalability being another interesting one) as well as working on some more futuristic researchy items related to the above.

On the personal front, probably best to track me on facebook if interested in that sort of thing.

I promise to push myself to keep this one active on the resarchy side of things.

Kipp

Apple and Skyhook

My goodness but it's been a long time since i made it out here. I'll blame it on the work...and with good reason.

We've been pretty much heads down for the last several months preparing for what came out last week. As most of you who read this already know (since I've been crowing about it for the last week) --

Apple announced the inclusion of Skyhook's Wireless Positioning Service in the upgrade of the iPhone and the iPod Touch. Steve Jobs announced this during his keynote at Macworld in San Francisco last week.

USA Today
Video podcast
Boston Globe
Links to pr and keynote
Some photos


among others....the best is to see Steve on stage explaining it -- "Isn't that cool? ... That's really cool!"

So, we're happy to be included in the iPhone, a pretty fantastic phone/computing device. It really does change the way you think about the device -- or at least it has for me. Phone seems like such a limiting term...

Well, that's the big news. A number of other things have been going on that are worth mentioning:

- patent 7,305,245 "Location-based services that choose location algorithms based on number of detected access points within range of user device " was issued and includes me as an inventor
- we continue to hire some excellent people into the team (and we're looking for more!)
- the prospects look good for adding WPS into other devices

Personally, I'd like to be able to spend some time focusing on continuing to improve the system for indoor use. There is a lot of research that has been done on indoor positioning systems that we can leverage in considering a large scale indoor positioning system. It has some great challenges that make it hard, but extremely interesting. Not to say that outdoor positioning doesn't also have some great problems as well, but the game changes a bit when you move indoor.

Not a bad start to the year!

Kipp

Moving and Austria

So we have a new house in Medfield! All of our PODS showed up on Saturday and it looks like most of our stuff made it in it's requisite number of parts, but we'll see when it's all unloaded.

Meanwhile, I took off Saturday to Munich. Arrived on Sunday and had a road trip to Innsbruck, Austria. Lovely town and beautiful weather. I'll try to get some photos online today if possible.

Attending the UbiComp conference for the next couple of days. I've found at least one person that I know from GT (Gregory Abowd) and likely will find a couple others along the way.

Opening keynote is coming up Antonio Calvosa from Ferrari...should go fast :-)

Home

Nearly there, we close on the house in mere hours (http://www.flickr.com/photos/22868073@N00/sets/72157601442689857/). Looking forward to getting out of our little tiny one room hotel suite!

We should be able to take possession immediately and start evacuating from the hotel. The PODS are scheduled to show up tomorrow, so I'm hoping to be able to get some of the basics (mattresses) out before I fly off to Europe. Yup, me being me, I fly off to Munich tomorrow afternoon to leave Pam to get us moved in. Aren't I sweet?!?

I'll be attending UbiComp which is held in Innsbruck, and then off to Loca2007 which is in Oberpfaffenhofen, near Munich. Back in the States Saturday evening.

Hopefully once back, I'll be able to find all of my things (like the card reader for my camera) and get back into the swing of things.

Work is going great, the kids are liking their school, but I think it'll get even better once we are in our new place and can call it home!

Kipp

Hello Boston!

Well, the summer has wrought quite the change. Whist pursuing my degree at Georgia Tech, I've been working with a company in Boston, Skyhook Wireless as I've used a lot of their data for my research in wireless positioning and mobile location services.

Well, they closed another round of funding in July and subsequently invited me to join the team. After due consideration, I accepted and the family has thus been thrust into the midst of a move to the Boston area. We're excited by the opportunity, but still trying to get our northeastern legs under us.

The company is located in downtown Boston and we've located a house in Medfield...a 'burb that puts me about 35 minutes of train ride away. The house is rather old -- built in 1751! But it's in good shape and came with 1.5 acres of land, so we're stoked.

The girls started school today and so far so good!

I'll post more info and pictures soon.

Kipp

Visited countries

I was following a trail though flickr and found this little nugget



create your own visited country map

It's sad...I've only visited 11 countries (4%)...wow I have a lot to do!

At least I've done a bit more in the States, but still only 2/3 of the way there:




Kipp

Network Effect

So, Metcalfe's Network Effect is kinda cool. I just experienced it recently as fellow from the Scottish Borders. John Connell works for Cisco as their Education Business Development Manager for the Emerging Markets.

John recently spent a brief time in Liberia, hence the connection. Check out blog and his photos. I definitely agree with his thoughts on open source and the role it will play in helping build a platform for education, especially in developing countries. He discusses the OLTP approach, but notes some of the shortcomings especially in countries like Liberia that have effectively zero connectivity outside of the capitol city.

The other critical issue (and one that is even more basic) is access to power. As Russell Southwood of Balancing Act discusses in a recent article on Power, the lack of power makes nearly everything else harder and more expensive. While we (in the US anyway) complain about the cost of gas, we have the luxury of worrying about the cost -- very few people in Liberia have a chance to even experience electric power (other than that produced by the various diesel generators --which mostly produce a lovely odor for people to experience).

Another recent connection thanks to the connected world was an introduction to a member of the parliament in Uganda. I'm hoping to get another perspective on ICT policy development from Edward Baliddawa thanks to an introduction from my friend, Josh Jackson at the University of Alabama, Birmingham.

I hope to extend these connections in the near future!

Kipp

Reflections on Atlanta Motor Speedway

Kinda cool... one of my photos from a trip to the AMS was included on schmap.com.

The photo:

History of the Internet - 6 minutes

I was wandering around a great telecom policy blog (CyberTelcom) and stumbled on a video of Ethan Zuckerman ( -- who, btw I spent some time with at a technology for reconciliation conference event last year). The video has Ethan going through the history of the Internet in a very entertaining fashion and getting through some 40 years in just 6 minutes and covering such things as the emoticon and MMORPGs in the process.

Oh, and just to tie it in with an event I'm planning on Net Neutrality, I got to CyberTelecom via the Google Policy Blog talking about the topic...





Kipp

The Left and the Right War

So, last spring (2006) I wrote a little paper on what I called the 'Left War', looking at the primarily Latin American swing towards the left and it's impact on the US and our international policies.

After pondering it some more, I would say that we are embattled in at least two simultaneous non-traditional war fronts. On the one hand, we have the Left War, battling what we perceive as a socialist revolution that threatens some of our democratic principles. It's clear that the socialist reform is appealing to the public at large -- at least at first. It is unclear how long the legs are on this movement, but as long as it is propped up by petrodollars, I imagine it can have a pretty good run. And Chavez seems to be doing a heck of a job rallying the troops right now, supported by grandpa Fidel and mini-me Evo Morales in Bolivia.

On the other hand we are fighting what I call the 'Right War'. In this case, it's seems to be about who is Right -- in the righteous way. But really it is a battle of the the Extreme Right in which the religious fervor has short circuited common sense and declared that there is only one Right. The Middle East represents the nexus of the battle for us, but you can see the spread around the world and locally.

Africa seems to be off on it's own doing its darndest to itself (well, after the rest of the world mucked with it and then ejected), however, there appears to be a growing strength inside of Africa, but it will take a long time to mature and recover, but there are indications that it can happen. There are many continuing issues, not the least of which is Dakar followed by Somalia and the horn in general. I see the Africa conflicts as related but largely independent of the Left and Right fronts. Certainly you can find connections to both fronts in action, but they appear to be offshoots rather than primary centers.

What's interesting is when these new fronts get together with our old Cold War center -- and you can see this with Chavez meeting with Iran and then going over to the Russia to stock up on weapons, including the latest expectation that he'll buy a few submarines to patrol his oil shipping lanes. So now we have an interesting confluence of the new Left and Right supported by the old guard (Cuba, Russia).

Of course there are a lot of other countries wanting to be in the mix in case we get bored...

Kipp

Minutes as Money

Since my travels to West Africa, I've definitely been pondering the impact of mobility on not just communications but life in general. With mobile phones as the only means of two-way remote communication (unless you can afford a VSAT installation) in Liberia and the lack of financial infrastructure, I've been looking at how to leverage the mobile infrastructure to make remote banking and remote microfinance a possibility.

This blog from Om Malik discusses how minutes have become currency in some areas -- and how it has become a business for many. The article references an iAfrica.com entitled "Africa's cyber currency".

The idea of using mobiles to reach the unbanked is taking root, but my sense is we are very much at the cusp of understanding the impact, capabilities, and issues -- which is why I find it an intriguing topic for further research.

More Net Neutrality

I'm working on putting together an event to discuss/debate/argue the merits and issues surrounding Net Neutrality. As such, I'm always interested in well considered arguments as well as those that come from an emotional or political angle. It's a very touchy subject for some, and covers everything from government intervention, to oligopoly market manipulation, to business models, to competition, to politics -- from wired to wireless -- from state to global. It has many facets, so having a good understanding of all of the different elements is helpful.

Here are some recent articles I've read (and I'll inject my editorial note here):

- Political (and largely baseless): Should there be neutrality in the Net? "Because Google, Yahoo and Microsoft have promised special treatment for leftist websites and blogs if net neutrality passes."

- Wireless Net Neutrality: From Tim Wu, pushing the idea of net neutrality and open access towards mobile devices (read: cell providers). Wireless Net Neutrality: Cellular Carterfone on Mobile Networks

- Academic articles and definition: Tim Wu's work

I tend to give weight to the well researched and documented arguments rather than specious non-referenced, emotional/political arguments, but that's just me...

Invisible Web

Been pondering what the 'Web' is today. Over the last 15 years, it has morphed, grown, changed, evolved into something that is really beyond the web.

As defined in Wikipedia, [http://en.wikipedia.org/wiki/World_Wide_Web WWW]: The World Wide Web (or simply the "Web") is a system of interlinked, hypertext documents that runs over the Internet. With a Web browser, a user views Web pages that may contain text, images, and other multimedia and navigates between them using hyperlinks.

But to see the 'web' of today, it's much more than that. It has enabled/spawned things like Second Life, blogs, facebook, youtube, mash-ups, etc. etc. etc.

But what are the common themes? Where is it going and where will it be in 10 years? One overriding current in the present/future web (and technology in general) seems to be the consumer aspect. Most innovations are being driven by these massive social and end-user experiments and they gradually seep back into the corporate arena (there are many examples of this from IM to mash-ups to blogs to open source to eBay to wikis to rss to Second Life).

It seems these massive dynamic social networks that are enabled by the web are what's driving a lot of the new technology and behaviors. Perhaps this 'Social Web' is a theme. I think it is something that needs to be understood, and especially in context of enterprise computing in which the slow uptake by 'stodgy' businesses can certainly impact their ability to compete in tomorrow's landscape.

Beyond the social aspects, it seems that we are starting to see the 'structure' of the web (links, pages, protocol, device) disappear with the content and the interactions becoming the dominant visible feature. This is a good thing, when the technology starts to disappear and the functionality becomes the dominant feature. I'd refer to this as the Invisible Web -- it's the infrastructure and all of the various technical underpinnings that enable this new, emergent types of interactions to take place. Note that I use invisible web different from 'dark web'.

So perhaps an evolution of Webs could be:

- World Wide Web
- Data Web
- Transaction Web
- Semantic Web
- Social Web
- Web 2.0
- Invisible Web

With the invisible web, it's not the pages, the links, the data, the network, the device, that are the driving visible features. It's the interconnected, free association, anywhere, anytime type of interaction that is enabled that is the dominant feature.

Kipp

Click FraudWall

I mentioned Jim Pitkow's Attributor launch recently...well, he's also involved with another company that looks to be addressing something like a $900 million/year problem [1], that of click fraud. The new company, FraudWall Technologies "combines cutting edge science with the aggregation of data and characteristics from networks, search engines, and advertisers into one complete scalable solution". Very interested to learn more about how they are trying to solve the problem. Clearly there are some existing measures in place to deal with this, but they are insufficient to deal with the sophisticated attacks that have been devised (and the low-tech methods like using extremely low labor to thwart automated analysis). I imagine there are a number of heuristics that can be devised to thwart these attacks, hope they have hit on some good ones!

[1] One estimate puts the amount of click fraud at 14%. Since they also estimate that about 40% of the online advertising is click-based and they expect over $16 billion to be spent on online advertising this year, click fraud would account for some $900 million.

GDP of US states versus other countries

RandomRoger caught this map which equates the GDP of various countries with the GDP of the states in the US. Interesting if not useful...but farily amazing if you think about it much...

Kipp

Web rights

A buddy of mine, Jim Pitkow and his latest company Attributor was featured in a Wall Street Journal article today talking about the technology these folk have created to scan the web for content as a means for tracking digital rights use/abuse on the web.

This could be huge, with the explosion of images, audio, and video not to mention the massive amounts of other intellectual property published on the web -- being able to track this could be an important piece to the puzzle on how to keep the web semi-organized and 'safe' for all publishers of content.

Having spent some time with DRM systems, I've always been suspect of how such systems can scale and survive on the Net. Some of them have been closed systems that require all things to be under one tent. All of them have suffered from the ability of smart people to 'break' their system, thus rendering them ineffective and setting up an arms race between the DRM and the hacker community.

On the other hand, tracking content on the web has been a challenge due to the massive scale that must be dealt with. A different kind of search engine, in this case dealing with finding and tracking specific pieces of information through, one can imagine, some form of fingerprinting technology that they've developed. A very interesting problem, I hope they've found a way to crack it.

Very cool stuff, and good luck to the Attributor team!

Kipp

Weather anyone?

From the MIT Technology Review, I caught the personalized weather forecast that IBM's supercomputer is spitting out. The article talks about why this resolution can be beneficial and what some of the downsides are.

Personalized Weather Forecasts by Duncan Graham-Rowe
An IBM supercomputer forecasts weather down to a one-kilometer resolution.

Pretty nifty animations (well, time-lapse jpegs) of tempature, cloud and precipitation, wind and surface conditions.

Now, can you link this to an investment strategy?

Kipp