1. Analyzing Customer Feedback in the Airline Environment
   • The example was fragments of text from customers that provided feedback about their experiences with an airline.
   • One analysis that can be done is the “tone” of the customer (“I think your airline is the best that I have flown on”   “Plane was late. Messed up my schedule. Arline sucks“
   • You can extract cities (to and from) as well as names of other airlines.
   • He had examples in other languages – specifically Spanish and French.  NOTE: The “stop” words concept applies to many latin-based languages.
2. Automating Extraction of Data from Raw Reports
   • The first step that Bill Inmon recommends is to strip the metadata from the report
   • This is often labels on the report – which also exist in a hierarchy that needs to be retained
   • Based on that he creates a “Metadata Template” – so that the report can be broken down more into a data aspect
   • Then that “Metadata Template” can be applied to the report itself to generate a list of the data for a given piece of metadata (which can repeat).

Summary

Overall I found the concepts that Bill Inmon presented to be of great value and insight.  He has spent years thinking of this – as he realized that the next focus would be on the “unstructured” data within our enterprises.  I’m not sure that I agree with his assertion that most vendors aren’t thinking about this – but I do believe he has a well thought out process.   The key aspect he continued to communicate was that context is needed in order to perform detailed analysis of free-form text.

His techniques are clearly focused on taking free-form text and resolving it to a traditional relational format.  He does this by adding context to the meaningful elements of the text so that they can fit into that structure. By doing so traditional query logic can be applied and the understanding of the data is much richer (Bob Jones, the pilot, is mentioned 6 times in the reports,  How many times is an animal mentioned, etc.).  I agree with this assertion that there is a lot of potential business value to be found in this unstructured data – so that focus needs to be spent on it.

He does correctly point out that many of the tools in the Big Data space do not address this challenge – as they either assume a structure (Hive) or ignore structure (Pig, Map Reduce).  Re-creating structure from that free-form text is a different exercise that requires careful planning and effort.

Other approaches

I don’t agree with his assertion that other vendors are not supplying any solutions to free-form text.  IBM has a whole software solution built around text analytics.  They have created a new API language called AQL (Annotated Query Language) to drive their text analytics engine.  While I agree that most vendors (Greenplum, Oracle, IBM, etc.) are focused more on infrastructure than software solutions they are addressing it.  What may confuse the issue is that many Big Data use cases actually use structured/semi-structured data – not unstructured data.

The other aspect to be considered is that you don’t necessarily need to deeply analyze and structure free-form text in order to gain valuable insight from it.  A simple case could be sentiment analysis of Tweets in terms of your company.  If I scan through the text of tweets looking for my company name and for each tweet found look for sentiment words (great, terrible, always, never..) that can provide a rough gauge of how a company is doing.  You can do similar things to understand how often a topic is mentioned (even using known taxonomies to increase the quality of that analysis).

For geeks like me we often want a near perfect solution – one that address all the edge cases.  Many people in business simply don’t care – they focus on the core cases.  This is an area where we can ask ourselves – is it good enough?  Can I, without the deep structure that Textual Disambiguation provides, get my answers without the same level of effort.  I think in many cases we are learning that we can – that the rough is good enough for our purposes.

Conclusion

I think any company wanting to derive value from “unstructured data” should carefully consider Bill Inmon’s approach.  I think many IT professionals (especially “Data” people) need to understand this – as it is a paradigm shift from what we’re used to dealing with.   He does identify a hole in our tendencies – as we often assume that our data will be structured in some way (even when it isn’t).  He has some very intelligent ways of addressing these needs – including working software.  Overall I was very glad that I attended this talk as it gave me lots to think about and process through.

Finally I dare you to say “Textual Disambiguation” 5 times while rubbing your stomach and patting your head.

Contracts

Below I have an example of the template of a mortgage – a very common contract.  A completed contract would have the “….” filled in with actual data – such that it blends with the text around it.  Obviously if you knew ahead of time the format of the contract (such as the template below) you could parse out the terms using the words around it. For example between “this” and “day” would have the day.  Ideally we would store the text of the contract along with the details that are entered in – for easy usage.

But let’s take an example of a large volume of varying contracts – such that we can’t rely on a specific structure (even for the same lender/agent).  Therefore we need to apply some techniques to break down the text into meaningful data.  One of the first steps we can do is to remove the “Stop” Words from the documents. For example:

This mortgage is made this 14th day of October 1998, between the Mortgagor, John William Smith (herin known as the "Borrower") , and the Mortgagee, Countrywide Financial a corporation organized and existing under the laws of California, whose address is 128 W. Absalom Rd, San Diego, California (herein "Lender")

The next thing we need to do is to begin to find “delimiters” of the different terms in the document.  If we had a CSV file we would use the commas to know when a field begins and ends. Similarly we need to identify beginning and ending words that indicate where to find an item of meaning.  If we look at the example above we could say “between Mortgagor,” and “(herein known” to know the name of the Borrower.  I suspect this may be an iterative process as the documents can vary significantly (so you may either have many different algorithms or very smart ones).

Mortgage Temple Example

THIS MORTGAGE is made this ........................ day of .............................

19 ....., between the Mortgagor, ................................................................ (herein

"Borrower"), and the Mortgagee, ...................................................... a corporation

organized and existing under the laws of ......................................

......................................................................................................, whose address is

..................................................................................

.......................................................................................................(herein "Lender").

WHEREAS, Borrower is indebted to Lender in the principal sum of ...................

Dollars, which indebtedness is evidenced by

Borrower's note dated ..................... (herein

"Note"), providing for monthly installments of principal and interest, with the balance of

the indebtedness, if not sooner paid, due and payable on .......................

TO SECURE to Lender (a) the repayment of the indebtedness evidenced by the Note,

with interest thereon, the payment of all other sums, with interest thereon, advanced in

accordance herewith to protect the security of this Mortgage, and the performance of the

covenants and agreements of Borrower herein contained, and (b) the repayment of any

future advances, with interest thereon, ma

de to Borrower by Lender pursuant to

paragraph 21 hereof (herein "Future Advances"), Borrower does hereby mortgage, grant

and convey to Lender, with power of sale, the following described property located in the

County of .................. ......................................................................... ......... , State of

Massachusetts:

which has the address of .........................................................................

[Street] [City]

.................................................................... (herein "Property Address");

[State and Zip Code]

E-mail Filtering

E-mail is a good example where a filtering process may need to be applied first before more advanced Textual Disambiguation (still a hard word to type) process is applied. This is due to the large volume of e-mails – many of which you don’t need to be looked at.  Bill Inmon described 3 categories of e-mail:

1. SPAM – this is externally generated content from outside the organization with little to no value (extraneous information).
2. Blather – this is internally generated content (from your users) with little to no business value. Examples would be jokes, personal e-mails, broadcast e-mails, etc.
3. Business Meaningful – the content you actually want to look at.

One example scenario he had was to apply analysis to current e-mails to detect and prevent potential new liabilities for a company (so they don’t end up like Enron).  He quoted that in 2012 approximately $65 billion was spent defending lawsuits.  For this to be effective the “SPAM” and “Blather” needs to be filtered out so that time and energy is spent on e-mails with potential meaning.

The principal process he applied to this problem was building a “relevance” taxonomy.  Basically a list of words (using known taxonomies as a guide) used to filter out e-mails that aren’t of interest (i.e. if they don’t have one of the words they are filtered out).

The next principle is to use “words of concern” to identify more significant e-mails.  These are human generated words that if a human was looking for e-mail they would want to see.    Some examples are:

• apologize
• attorney
• risk
• ashamed
• scandal

Additionally the relevance of an e-mail can be further filtered by looking how close these words of concern are to each other (proximity).  Typically a “proximity boundary” is selected (in bytes) to determine how close words need to be to increase the relevance of an e-mail.   Once the list of e-mails is generated there are some other techniques to narrow down the e-mails that should be looked at by a person:

• Use header information – such as From and To, Dates, etc.
• Use the # of words of concern found (i.e. more words means more likely needs to be looked at)
• “Hot” words – words that attract the attention of a human when scanning e-mails.  Use these words to increase the relevance of those e-mails

In reality this would likely be a very iterative process as the users provide feedback on why an e-mail is more or less relevant to their concerns.  Additionally changes in the business and/or legal environment could shift what needs to be looked for.

This post was about applying techniques to either filter out documents or break them apart in a systematic way.  In my next post I will summarize what I’ve learned and contrast with other techniques in the industry.

That word – unstructured – is one that bothers me – as much of the data we’re talking about does have a structure. Often what we’re describing with the term “unstructured” is the free-form text within that data. Tweets, Facebook messages, e-mails, texts, etc. – do have a structure. At a minimum they have some metadata wrapped around them (such as e-mail headers). In other cases, such as a Twitter message, there is a lot of potential data beyond just the “Tweet” itself. If you look at the format of a Twitter message there is information about the user, a time-stamp, reply-to information, and possibly location information.

I think often what we really are saying with the term “unstructured” is the free-form text within these data sources. It’s the text of a document, the body of the e-mail, the “tweet” itself, notes and comments. We can’t use traditional techniques on this data for a variety of reasons – the most prominent being that most tool rely on a structure.

One of the main points Bill Inmon made was that when processing this “text” data is that we lack context. If you think about how we often process data we have some form of context:

• A relational database relies on a defined schema – so you know the column name and it’s characteristics
• XML data is self describing (it carries it’s own context) [<myfield>abc</myfield>]
• Flat files are often either delimited or are position based – which implies a context by an external definition
• JSON Data – like the tweet – is also self-describing ["coordinates": null,"created_at": "Thu Oct 21 16:02:46 +0000 2010",]

Context is vitally important to drawing useful conclusions – as language is often imprecise. Much of what Google does today is to try to guess what your intent is – using the context of your previous behavior (try searching for something in Incognito mode in Google Chrome vs normally searching). Amazon, Facebook, Netflix, etc. depend on delivering a solution based on the context of your past behavior in conjunction with other user’s behavior.

Let’s take an example right from Bill [he has a website - http://forestrimtech.com/- where you can learn more - including downloading white papers]: “She’s hot”. OK – so what does this mean? Does it mean she is very attractive? Does it mean she is running a fever? Does it mean she’s sweating heavily due to the heat?

So how could we know what “She’s hot” actually means? What if this was mentioned in San Antonio during the heat of the summer? What if it was said by a young male around other young males? What if it was a spoken by a concerned parent at a doctor’s office?

What we are doing is adding a context around “She’s hot” in order to understand what it means. Apart from that context “She’s hot” doesn’t have clear meaning and could lead to wrong conclusions easily. [I personally remember an example years ago on a bus, probably in Junior High, where a young girl said "I'm hot". Shortly after she said it she clarified that it was temperature hot - as many people we're thinking of it as she was complementing herself.]

So in one sense we have this jumbled mess of text – without any organization. But in another sense there is structure to derive from this text – which provides us the missing context. Documents have some type of structure (especially legal documents like contracts) – even if it wasn’t planned that way. Sentences have structure – anyone remember diagramming a sentence?

Why do we care?

One simple reason we should care is we have this 80% of corporate data that is unstructured – vs. 20% that is.   Of that 80% unstructured only 1-2% of corporate decisions are based on it (vs 98% on that 20% of structured data).   So this is potentially an untapped reservoir of information to make better decisions.   There are business opportunities – both internal and external – lurking in this untapped data resource.  We will have to learn and implement new techniques to utilize this data – but the clear trend is in that direction.

In my next post we will look in more detail about Bill Inmon’s Textual Disambiguation concept itself (which he has commercialized as Textual ETL).  In a later post we’ll look at applying this at a document level – instead of just at a sentence/paragraph level.

As I previously mentioned I attended a seminar where Bill Inmon taught about “Textual Disambiguation” (quite a mouthful). His concept is that we need to apply context to that text in order to analyze it effectively. He has some specific principles on how to draw that context out – which he believes are still unique within the industry (he believes that many vendors are ignoring this issue).

So the heart of his process is to process through text in order to discover context – and therefore derive meaning. His company’s software (which he didn’t dwell on in his talk as it was NOT a sales presentation) will start with free-form text and end up with a structured analysis of the words. He was very clear that this is different than NLP (Natural Language Processing), many Text mining techniques, Map Reduce, HIVE, Pig, etc.

I don’t have a copy of his presentation of this – but I will share what I learned using my own examples. Here is the list of techniques (probably not a complete list) to add context back into unstructured text:

• Remove Stop Words
• Correct Misspellings
• Word Stemming
• Standardize Dates
• Document metadata
• Taxonomy / Ontology
• Proximity Analysis
• Number Patterns (SSN, Phone Number)
• Numeric Value Tagging
• Date Naming
• Acronyms

To make things interesting I’m going to use some Enron e-mails fragments to demonstrate some of these techniques.  You can download the whole set at https://www.cs.cmu.edu/~enron/

Stop Words

• Stop Words are basically the words that connect the words of meaning.  These are typically words like “the”, “is”, “at”, “which”, etc.  
• While they are part of the language they really don’t have any value outside of the sentence.
• Below is a paragraph from an Enron e-mail where I striked through the stop words based on this list I found.
• This technique is also used for Search Engine Indexing – as these words wouldn’t help you locate the page you want.

Mr Lay - If you really think that this sale creates a "great opportunity" for shareholders then you are more out of touch with reality than I previously thought (unless you were referring to Dynegy shareholders). Under your "leadership" the shareholders have been devastated, employees have lost their retirements, college funds have been desiminated and reputations have been ruined, including your own.  

Correct Misspellings

• A simple technique is to correct misspellings of words in the text
• By doing so we can then match words together that otherwise we couldn’t (how many times is x mentioned?)
• I suspect this requires some sophisticated logic to perform this correctly – but I did find this list from the Oxford Dictionary
• I took a snippet of an e-mail and deliberately misspelled some words as an example

Jeff,

We spent quite a bit of time over the past several months discussing a possible minority investiment of about $5MM in Silicon Energy. We have broken off those discussions because (1) their proposed pre-money valuation of $150MM was, in our opinion, excesive, and (2) our people at EES, who would be the primary users of Silicon Energy, were not happy with Silicon Energy's functionalty.

Word Stemming

• Word Stemming is a concept where we take a word and break it down into it’s root.
• For example - move, mover, moving – stem of “mov”
• I found this page that suggests that there are algorithms that can be used to accomplish this.

Maureen Smith and Ruth Concannon have raised some issues regarding how Brooklyn Union Gas is in the books, specifically, the final contract year (11/1/03 - 10/31/04) is not in the books at all (which will produce a gain when booked), and the current deal structure is telescoped incorrectly at Transco Zones 1,3, & 4 when it should be telescoped 17% at Zone 1, 25% at Zone 2, and 58% at Zone 3.  This will produce a slight loss when rebooked, but the gain from booking the final year is more than enough to offset the loss.

Standardize Dates

• Dates come in many formats (05/29/2013, 29/05/2013, May 05, 2013…)
• For ease of comparison it is best to standardize them
• Below are some sentence fragments out of e-mails with different formats

Date: Thursday, January 24, 2002

at the time of their next board meeting on February 12, 2001.

Attached is a revised Credit Watch listing as of 4/09/01. Please note that there are 12 counterparty additions/revisions to the Watchlist for this week.

Document Metadata

• Many documents have metadata associated with them – including word documents, e-mails, tweets, etc.
• This includes dates, metadata about the document, in the case of e-mail the from/to addresses
• Here is an example of an e-mail header:

Date: Tue, 16 Oct 2001 14:41:10 -0700 (PDT)
From: administration.enron@enron.com
To: owa.notification@enron.com
Subject: Outlook Web Access for Calgary
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-From: Enron Messaging Administration </O=ENRON/OU=NA/CN=RECIPIENTS/CN=NOTESADDR/CN=ENRON MESSAGING ADMINISTRATION>
X-To: OWA.Notification@enron.com
X-cc: 
X-bcc:

Taxonomy / Ontology

• A method to bring meaning between words is to build a taxonomy or ontology
• This is typically a type of, kind of, etc.   For example, a dogs and cats are types of animals
• Therefore these words can be grouped together (even though they are in different parts of a document)
• If you look at the below e-mail snippet the words “jet skis”, “boats”, and “cataraman” are all types of boats, which are a type of vehicle

This is the info.-  Kim Hillis is making the reservations (I hope there are rooms available this late)    A couple guys here have stayed there and said it's awesome.  With a great beach, close to town and golf. There's jet skis and boats right there.  I'm up for chartering a a catamaran for snorkeling and cruisin' all day.

Proximity Analysis

• Another technique is to look at how close in proximity words are (i.e. are they within a few words or paragraphs down)
• This is known process – used in search engines to some degree
• There is no guarantee this will add any value-  but could be useful in some cases
• Below is a pretty weak example as I struggled finding a good example (I don’t have good notes on Bill Inmon’s example).  Because snow is near forecast it can create a connection to quality forecasts:

 Cooper-- I have been looking at the chassis for the model we need to develop. it looks like we can expand (piggy back) the model AE uses for the Snow & water supply forecasts. They only have reports for March thru to August i.e. the period of the bulk of the runoff from snow and rain. We would have to add the remainder of the year as well as the power plants. Will fax ou a sketch of my proposed basic layout.

Number Patterns (SSN, Phone Number)

• There are many common number patterns that can be identified – given how we typically format them.
• Phone Number and SSN are probably some of the most common examples – but each use case may have it’s own format(s)

Pursuant to your e-mail to Jeff Skilling dated February 20, 2001, Mr. Skilling suggests that you contact Mr. Jim Fallon regarding CAIS Internet Inc. Mr. Fallon is managing director of trading at Enron Broadband Services. He can be reached at 713.853.3354.

EEI member utilities wishing to have access should contact Lynn Hailes at: lhailes@eei.org or 202/508-5624.

Numeric Value Tagging

• Numeric values in a document without context don’t provide a lot of value
• Often times though there is text adjacent to the number that indicates what type of number it is.

3. Book administrator rolls showing the following new deal P&L for the following new deals rolling up to the Executive DPR:

Deal #559092.1 on 3/23/01
Deal #514509.1 on 2/6/01
Deal #568025.1 on 4/2/01

Date Naming

• While we can standardize dates easily without any context they still don’t mean anything.
• We want to look for nearby words to provide that context – the name – of a date
• Below we have a date of Feburary 12, 2001.  There are two nearby sets of words that can provide context:
  • effective ==> Effective Date of February 12, 2001
  • board meeting ==> Board Meeting Date of February 12, 2001

It is my great pleasure to announce that the Board has accepted my recommendation to appoint Jeff Skilling as chief executive officer, effective at the time of their next board meeting on February 12, 2001. Jeff will also retain his duties as president and chief operating officer. I will continue as chairman of the Board and will remain at Enron, working with Jeff on the strategic direction of the company and our day-to-day global operations.

Acronyms

• We often use a cryptic set of letters in place of longer phrases – for ease of use.
• However, these acronyms themselves don’t have any real meaning – but what they stand for

If you are interested in the Live or Archived meetings of the Federal Communications Commission (FCC) or National Transportation Safety Board (NTSB), please contact us.

If you have any questions concerning the FERC Video Archives, please contact us at capcon@gmu.edu or at 703-993-3100.

Summary

I feel like I just gave a very brief summary of these techniques – where there is a lot more knowledge lurking out there.  By no means is this an exhaustive or complete list – but just a beginning.  Again – Bill Inmon’s company website – forestrimtech.com – has a lot more information.  I think the important part is to realize that you can discover context to wrap around what looks like otherwise meaningless text.

Next we’ll address this at the document level – instead of at the paragraph/sentence level.

For a kindle book we really have 2 types of data -the text/pictures from the publisher and the data we create. We create 3 types of data:

- where we are in the book (whispersync)

- our highlights

- our notes

These items (provided we have an Internet connection.) are available on different devices. It’s only over time that I’ve realized just how useful this is. I’ve blogged before about how I highlight and take notes for myself in ways that I never did with a physical book. I find myself taking advantage of these features more often now.

I wonder now how much data I’m providing to Amazon now with those highlights and notes. I know they look at the highlights – as they show the most powerful highlights. How much profiling of me can they derive from what I highlight in a given book (in comparison to others). Maybe that’s why there are so many free books…

So is it an eBook reader or something else? Am I giving up some privacy for convenience?  One advantage of this is that you can view all your notes and highlights (at kindle.amazon.com).  I still sometimes wonder though what would happen in the future if Amazon stopped selling these books – if I could somehow offline integrate the book content with my notes again (this is where I place my hope in hackers).

Maybe this is the reality to be (if it isn’t already is) – that we’re not purchasing a product but a service.  We don’t “own” something anymore – but have a right to use it.  I’m not sure if I like that – but to get the rich service we want we may have to deal with that.  Hopefully for ebooks in the future there will be a way to backup your books and notes yourself – so that you’re not beholden to a single company (like I am to Amazon).  Or maybe the NSA will offer that…

Overview

I recently finished reading a book by Sunil Soares:  Big Data Governance.  The whole “Big Data” topic has been exploding – so I’ve done a lot of research into the area.  With my background in data architecture (which inherently recognizes the value of data) the concept of applying Data Governance principles to Big Data was interesting.  So I broke down and spent the $30+ dollars on the Kindle book so I could better absorb this concept for possible use in my professional career.

Sunil Soares used to work for IBM and was one of the authors of a free eBook: the IBM Data Governance Unified Process.   I read that eBook previously and was able to absorb some of the material (I need to re-read to get more out of it).  It felt like some of the information in this book was very similar to what was written in that free eBook (so if you’re cheap like me you may want to start there).

As usual I’m going to talk about what themes I saw/learned from reading this book (which could have come from other literature by him and others).   The book itself has a pretty organized outline of the steps he would recommend for various data governance principles relating to Big Data.  You can get most of that by simply reading the Table of Contents of the book so I’m not going to repeat it here.

I believe that the “Big Data” phenomenon itself has clearly demonstrated the value of data itself -of what value it can provide.  That said – it’s the results of insight where the real business value is.  Sunil Soares says: “its value must be truly understood and unlocked by deriving insights that are revealed through analysis and then translating those insights into information, knowledge, and ultimately action.”    Companies are finding profound ways to take data – from many sources – and derive insight they never could before (either it was impossible or too expensive).

What is Big Data Governance?

Sunil Soares defines Big Data Governance as follows: “Big data governance is part of a broader information governance program that formulates policy relating to the optimization, privacy, and monetization of big data by aligning the objectives.”  This definition implies that Big Data Governance sits in the context of an already existing Data Governance program.  Therefore it seems that the author is saying that most of the principles for Data Governance in general would apply to Big Data Governance.  I’m not sure that I agree with that – but we shouldn’t ignore what has been learned in traditional Data Governance programs.

Why Big Data Governance?

I think the first question to answer is why you would even want Data Governance as part of a Big Data program.  Some of the promise of Big Data analytics is that you don’t have to do all the traditional Data Warehouse work to get results.  The concept is that you can load the raw data into your Hadoop environment and perform some advanced analysis and wala –  out comes meaningful results.  You don’t need to perform rationalization, cleaning, summarizing, etc. – you just work with the data as it is.

While some of that is true – it’s somewhat of a misleading picture.  It’s true that one of the advantages of putting all the data into the Hadoop environment is that you don’t have to rely on a sample of the data (which may not represent well the whole).  For example: if you we’re provisioning bandwidth for a set of servers and used an averaged sample of the currently used bandwidth you may miss the occasional spikes in bandwidth that really drive what’s needed [Scott Kahler explains this better than I did in this keynote video: http://www.kcitp.com/2012/09/03/big-data-kansas-city-technology-events/].

The other major difference is including far more variety of sources in your analysis – including segments of unstructured text.  The technology now allows us to efficiently process through far more volumes and variety of data than we could before.  We’ve also advanced in how we can process text and other varieties of data – in terms of our algorithms and other advanced processing. The ability to combine so much data together to get a picture is fascinating.

That said – it’s not quite that easy or simple.  When we say unstructured data what we often mean is that there is some unstructured text within an otherwise structured container.  That structure may not be as rigid – in that maybe not everything is present or it’s more variable -but it still has a structure.  Therefore an effort needs to be made to understand that data – especially in terms of it’s reliability.

Here are a few examples of data that may not be what it seems:

1. User names in Social Media – they’re not always a real name.  In some social media sites there is no guarantee that the user name is a person’s real name (or any real name).  This is significant as one of the goals is often to tie a master customer record to their social media data.
2. Sunil Soares mentioned the term “unique visitors” (in the context of clickstream data).  One site/source may measure the # of unique visitors a week vs. another measures it within a month.  If you directly compared this data without addressing this you would get skewed results.
3. Let’s say we have a measurement that represents the average temperature for the last hour.  If one measurement was a rolling average (taking into a large # of previous values) vs. another is only for that hour.
4. Location data – does each data source assign the same meaning to the same value?  If you matched data solely on the values would you really be matching the same location?
5. Another of Sunil Soare’s examples was sensor and part terminology in railroads.  If we can determine that sensor event #282 typically occurs before part #339 fails does that part have the same # in different cars/engines?  Do the different sensors produce the same code for the same event?  Would we need some type of cross-reference table to map these together?
6. At a higher level consider whether the same data is being pulled into your Hadoop environment multiple times?  Is Data Source Q really the same as Data Source A?   Did we end up wasting storage space, transmission and possibly licensing cost on duplicate data?

There are another dimensions of concern that are not technical – but a function of the complex and inter-related environment we all live in:

1. Privacy – despite fact some think that privacy is dead there are serious concerns around privacy and Big Data.
   • Consider who really owns the data?  Is it yours or the customer?  Most social media sites will tell you that the data is the customer’s – you can’t own it (you may even have to delete it if they ask you).
   • Are you, by combining data, create new types of sensitive data that didn’t exist before
   • Have you built safeguards into your Big Data platform to control who has access to what (security is not part of the native Hadoop platform)
2. Regulatory – regulatory agencies don’t care how the sensitive data is stored (i.e. Hadoop) – they will hold you accountable regardless.
   • Are you in a highly regulated industry such as HealthCare?
   • Are you dealing with sensitive corporate data governed by regulation?
   • Do you have industry constraints – such as PCI (credit cards)?
   • Do you know what the regulations are in each country you operate in (they are often different)?
3. Reputation
   • Even if something is legal – it may not look very good in the eyes of your customers or partners
   • You must weight the risk of the impact to your reputation vs. the revenue potential

What is different about Big Data Governance?

So the next question is whether “Big Data Governance” is really any different than traditional governance for operational or enterprise reporting systems.   I believe it can and should be different – as it’s often for a different purpose.  Sunil Soares puts this well:

Big data needs to be “good enough” because poor data quality does not necessarily impede the analytics that are required to derive business insights.

You may have heard of ETL (Extract, Transform, Load) but now there is a new term:  ELT (Extract, Load, Transform).  At it’s simplest the concept is that the data is loaded in its raw form and then transformed – not the other way around.  This is possible due to the fact we can both afford to load the raw data and have the computing power to transform it in place.  Therefore data quality may be enforced on the fly – instead of before the data is at rest.  So the focus is on doing a reasonable effort on the data that’s imported instead of making it pristine before it’s loaded.

How do I implement Big Data Governance?

The next question is then what’s a framework for implementing Big Data Governance.  Here are some of my thoughts (hopefully organized enough to be useful):

1. Know your Data.
   • Catalog your internal and external data.  Other than for a sandbox don’t let data into your Big Data platform unless it’s cataloged
   • Understand your data – not in complete detail but the overall quality, time scale, etc.
   • Document some of the key fields within your data – ones that aren’t intuitive and that are key to using the data effectively.
   • Develop a method to document and share this metadata
2. Know your organization and your platform.
   • Understand who can be involved in data quality – both at it’s source and while it’s in your Big Data platform
   • Understand what your platform can do – good and bad.
3. Understand constraints, regulations, etc. – especially by region.
   • Understand your legal, ethical and internal constraints.
   • Evaluate these by region – as they can differ greatly
   • Understand what your organization’s commitment level is regarding platform and people resources
4. Determine what data needs to be cleaned up and what needs to be protected.
   • Flag data that needs to be cleaned and why it needs to be cleaned
   • Flag data that is sensitive and needs to be protected.
   • Develop a method to document and share this metadata
5. Determine how and when to clean and protect the data.
   • Will you clean your data before it hits your Big Data Platform, after it hits it or in real time?
   • Determine strategies for cleaning that data
   • Determine strategies for protecting sensitive data and overall security schemes
6. Evaluate how you are doing on a regular basis.
   • Establish routine meetings (quarterly, yearly, etc.) to evaluate how things are going
   • Create the expectation that this is a process and that changes will be common

In conclusion I believe introducing Data Governance into a Big Data environment is a worthwhile choice.

Reading this book for me was a bit of journey into my past by remembering the technology.  I remember playing Oregon Trail on an Apple II (e I think) in Elementary School [Sidebar - I got distracted from writing this post by playing that game in an emulator.  I'm not sure I ever won at school either..]

I was born in 1973 so I remember a lot of the technology of early PCs – having experienced many of them. Despite the fact I grew up with this technology I still find it hard to believe how limited it was compared to what we have today.  In this day and age computers have become more of a commodity and not a special thing.  But back then, any computer was a work of art and full of amazing technology to accomplish what it did.

It is the character of Steve Jobs that is the most interesting part of this book.  He was a very unique individual – almost a force of nature.  He had such an impact on the technology we use today – not just our computers but on so much consumer technology.  He had a vision for what we needed before we knew it – an instinct that defied logic.  He made possible what many thought was impossible – and we all benefited from it.

That said – would I want to be him?  No – he’s not my role model as a person.  I don’t think I would want to work for him either – given how he treated people.  In fairness he was able to get people to do more than what they thought they could do – to perform at a high level.  I’m sure there are many people who still remember him fondly as a boss and as a person – but many others do not.  Personally I couldn’t treat people the way he did – it’s just not who I am.

I think “innovative” people like him are often over focused on their vision for the future to the point they forget there are real people are around them.  In Steve’s case it was clear that he had what they call a “Reality Distortion Field” – his own view of reality.  In some cases it worked for him – making things happen that seem impossible.  In other cases reality caught up with him and with others around him – often with tragic consequences.

Maybe one of the things he did well was take technology and make it usable for “normal” people.  What often happens with technology is that it’s made by “geeks” – who don’t think like normal people.  Therefore what comes out is designed to work they way a “geek” would want it – which often is not what a normal person would want.  A “geek” designs for edge cases (rare but difficult cases) without putting full effort into the common cases.

Steve made technology elegant and usable – attractive to “normal” people.  He made the concept of a Graphical Interface possible on computers.  He made a music player that transformed the music industry  - one that made others look weak (oh – and being able to buy music online).  He ushered in a mobile computing era – with the iPhone taking the world by storm.  He even finally designed a computing tablet that really worked – after so many others failed.

The world will miss his innovation – his focus on design and usability – not just capability.

I realized though they could learn a lot more from the data they are gathering than how much money should be in the machine.  They could tell if an item needed to be restocked and predict what items would sell well in the machine.  But that’s just the beginning – as they can learn about us from our purchases.

Imagine what they could do if they aggregate that data across many machines.  What could they predict – what could they learn? Would they be able to see trends across a metro area – or possibly the country?  Would they be able to tell about the health direction of the employees of a company based on what type of items are purchased more frequently?

What value would that data have – would anyone want to purchase it?  Could it assist with the supply chain – would that data be valuable to their suppliers?  What other data could it be correlated with to add additional insight?

These days there is so much data – those magical ones and zeroes – floating through the air. These streams of data about us and others are constantly surrounding us – flying through the air around us.  Our phones track our location, cable boxes track what we watch, websites gather and share information (ever seen ads about things you’ve searched on follow you?), who knows what the government now can track…

The world is changing – in subtle and profound ways – by what data we and others have. Do you get your bills electronically or by the mail?  When was the last time you “developed” a picture (not sure my kids even understand that concept).  Meters are going electronic – parking, electric, gas, water – with real-time feedback to the companies that manage those resources.

Sometimes when I stop and think it feels like there should be the streams of 1s and 0s flying through the air around me.  Maybe I’ve just watched too many movies like the Matrix and Tron or maybe it’s because through my life I’ve seen so much change in technology.  When it comes to data a simple example comes to mind: how much bandwidth is now available.  In college I remember a 56K modem being fast – now I can get 10 meg on my phone.  When my first daughter was born I had to write back to tape some of the edited video as there wasn’t room on my hard drive.  Now I’m going through and extracting the video off of all the raw DV tapes and the DVDs I created as I have the hard drive space.

So maybe it’s just me – but then again maybe it’s all of us.  What do you think?  Are you swimming in data?

I have used Lookout for some time – as it’s a free app that seemed to help with security.  I actually used the locate feature once on my previous phone when I thought I had lost it. I “located” my phone from my house and saw that it was near where I work in Downtown Kansas City.  That night I drove back to work and found my phone right next to where I park – a happy outcome.

This time around the outcome wasn’t quite so happy – as my phone wasn’t nearby.  After I clicked on locate in the website it showed me where my phone was – at around 31st and Linwood – which is over 5 miles away from where I work.  That’s when I knew this didn’t look good – as I definitely didn’t drop it around there – someone or something was moving it.  I tried both making the phone “scream” (quite annoying) and calling the phone multiple times.   Lookout has some advanced options – which require upgrading to the premium version ($).  After upgrading I decided to “lock” the phone to safeguard the data I have on the phone (it was setup to require a password to unlock the phone normally).

Strangely enough I located the phone again and it had moved – to the public library near Prospect.  I made the “phone” scream again to try to get someone’s attention. I even called the library to see if anyone had turned in a lost phone.  At this point I decided to “wipe” the phone to be on the safe side – as I was afraid whoever had it might turn it off.  After wiping it I could no longer track it so it was essentially “lost” at that point.

I had signed up for the insurance through Sprint as I knew this was still a very expensive phone. After starting that process I found out there was a $150 deductible I had to pay out to get it replaced. The irony is that I only paid $50 for the thing – due to a Black Friday deal.  The process of replacing the phone was fairly quick and painless -part of which had be printed off.  During the day I was able to complete part of the process – and fax/upload the documents in the evening.

The phone arrived at my house the next day before I went home.  After I got home from AWANA with my kids I started setting up the phone.  It seems like it gets easier each time to get the phone back up and running – especially with Google as the key player for the phone.  I love having my contacts centralized through Gmail – as it’s the same list everywhere.  I also had Dropbox this time as a repository of the photos from the phone – which was great.

Since I had to buy a new case I decided to get something with a more secure holster – as I think my phone fell out of the one I had.  I bought a Seido case as it had a locking holster.  I still dislike the extra bulk of the rugged holsters – but they do have their place.

Today my daughter tried to get into my phone and I was notified about it.  Apparently Lookout premium will take a picture with the front-facing camera if someone enters the wrong password 3 times (she thought I was kidding when I showed her the e-mail).  The technology today to know what’s going on with the phone even when you are away from it is fascinating.  It may not help recover the phone – but it does make for an interesting story.  Hopefully you will never lose your phone – but today it can be resolved without too much difficulty.

Today we’re starting to measure data not in megabytes or gigabytes – but in petabytes and zetabytes (and soon yottabytes).  For understanding here are some basic definitions:

• kilobyte  - 1024 bytes
• megabyte - 1000 kilobytes (1,024,000 bytes)
• gigabyte – 1000 megabytes (1,024,000,000 bytes)
• petabyte - 1000 gigabytes (1,024,000,000,000 bytes)
• zetabyte - 1000 petabytes (1,024,000,000,000,000 bytes)

Another way to put this in perspective is to compare these numbers to the storage objects that I’ve encountered in my lifetime.  For example,

• Remember those 3 1/2 inch disks (see picture above)?  They held 1.4  megabytes – which was a good amount in the early to mid 1990′s when I was in high school and college
• For backup I had an Iomega Zip Drive – essentially a removable hard drive.  These had a capacity of 100 megabytes - which meant a few of these disks could back up an entire hard drive at the time (which we’re probably around 500 megabytes)
• CDs typically held about 700 megabytes and we’re in most cases read only.
• DVDs can hold up to 4.7 gigabytes of data (which at the time seemed a lot).
• I have an inexpensive USB flash stick – which holds about 4 gigabytes itself (which are now less than $10)
• You can buy a 2 terabyte hard drive now at MicroCenter for $99.

•  500 2 terabyte hard drives (1000 terabytes / 2 = 500)
• 212, 766 DVDs (1000 terabytes x 1000 gigabytes / 4.7 gigabytes = 212766)
• 250,000 4 gigabyte USB sticks (1000 terabytes x 1000 gigabytes / 4 gigabytes = 250,000)
• 1,428,571 CDs (1000 terabytes x 1000 gigabytes x 1000 megabytes / 700 megabytes = 1428571)
• 10,000,000 Iomega 100 megabyte ZIP disks (1000 terabytes x 1000 gigabytes x 1000 megabytes / 100 megabytes = 10000000)
• 694,444,444 3 1/2 inch disk drives (1000 terabytes x 1000 gigabytes x 1000 megabytes / 1.44 megabytes = 694444444)