Which Super Bowl XLVI QB is better “in the clutch”?

Posted on February 4, 2012 by pvillanueva

With the Super Bowl XLVI coming up, there has been much debate over the two starting quarterbacks, Tom Brady and Eli Manning and whether or not both are considered “elite”. Tom Brady, without a doubt has the career stats to back up the elite label. His touchdowns, passing yards, quarterback ratings, and almost all other stats easily eclipse that of Eli Manning. Brady ranks right up there with other quarterback greats such as Aaron Rodgers, Drew Brees, and Brett Favre. But regardless of career stats, when questioned about his place in NFL history, Eli Manning himself said he was “elite”.

This argument gave me the idea of comparing stats of the two quarterbacks for this weekend’s Super Bowl. But rather than look at overall career totals/averages (because we all know Brady’s overall stats will reign supreme), I decided to try to compare their QB stats for only “in the clutch”. When I say “in the clutch”, I mean which quarterback delivered the most when it counted, that is, when the game was on the line. For my definition of “clutch”, I will look at who passed for more touchdowns in the all important and closing 4th quarter. And also who passed for more touchdowns in the 4th quarter with less then 5 minutes remaining in the game. With the pressure on, game clock is dwindling, which quarterback reigns supreme?

To perform this analysis, I needed stats that broke down at the play-by-play level. Most NFL stat sites only give stats at an end of game level (i.e. final box score). I found such play-by-play data at  www.advancednflstats.com.

As you can see from the shot above, the data is stored at the individual play level, tracking what quarter, what time, and a description of the play. The website had data going back to 2002. Tom Brady’s rookie season was in 2000 and Eli Manning’s was 2007. So there was enough data to cover the majority of both quarterback’s careers. In fact there was well over 384,000 plays performed in the entire regular season NFL games, dating back to 2002.

Using the trial version of Datameer, I loaded this data and performed some aggregations. Didn’t need to write any code, simply used the wizards to guide me through my analysis.

My first step was to import or ingest the data. Data from the site came in multiple .csv files, one for each year dating back to 2002.

After specifying the data details, Datameer was able to parse through the data and recognize all the column headers and data types.

Once the data had been imported, I opened up a workbook and linked my play-by-play data into a worksheet.

Since this data represented all plays in the NFL for all teams, I used the filter wizard to get only the records for plays by Tom Brady and Eli Manning.

Since the play-by-play data contained descriptions of the play, I applied another filter to find all the touchdown passing play descriptions, but also weeded out interceptions, reversed calls, and non plays. And since we’re looking for stats “in the clutch”, I’m only concerned with the 4th Quarter.

Next I created new columns to flag if the record was a play for Brady or Manning. Since all the details are in the play’s description field, I needed to use a “contains” function to check which quarterback the play was for. By simply double clicking into an empty column I was able to launch the wizard to help me configure the “contains” clause on the description field. I created two new columns, one for Brady and one for Manning.

Now that I have a record for every play and flag for both Brady and Manning, I could now create some analysis. By creating a new sheet and using the GROUPBY function, I grouped my data by the yearly football season.


I then performed a group count on my Brady and Manning boolean flags. One column for Brady and one for Manning.

I saved this workbook and ran it against the entire dataset.

You can now see the results, number of passing touchdowns for Brady and Manning, only in the 4th quarter. A quick plot onto Datameer’s dashboard shows me the following graphs: red lines for Brady’s stats and blue lines for Manning’s.

Then by following the same process above, but now filtering for touchdowns in the 4th quarter and less than 5 minutes remaining in the game, I get the following graphs:

This tells us that Eli Manning is actually better “in the clutch” compared to Tom Brady. Going back a couple years you will see that Manning has, for the most part, matched Brady’s stats. But this is all about “now” and coming into 2011, Manning scored 15 touchdowns in the 4th quarter and 10 with only 5 minutes remaining in the game. Brady’s numbers for the same year are 12 TD’s in the 4th quarter and 7 TD’s with only 5 minutes remaining in the game. So when the game is on the line and time is running out… Eli Manning is your elite QB!

While this data set is small, it shows how easily one can analyze data using Datameer. And since Datameer runs on Hadoop, we could easily scale up to billions of records.

So go ahead, download our trial version of Datameer and see what interesting stats you can come up for the Super Bowl.  And don’t hesitate to send us your results, who knows, it might be our next blog post!

http://datameer.com/products/download-trial.html

Posted in Big Data Analytics Perspectives | Tagged , | Leave a comment | RSS

How I hacked California’s largest healthcare provider – by mistake.

Posted on January 25, 2012 by sgroschupf

BTW, names have been omitted on advice of our lawyers.

It all started when I tried to sign up my then fiancé for a health care plan. After creating the account, the “request a quote” form was long, more than ten pages of questions about pre-existing health conditions. I ended up not having all of the information at my fingertips, luckily there was a save and continue later button. A few days later I went back and started typing in the domain name and my browser recommended the last URL of the domain. To my surprise, the form page with all my information opened. No redirect to a login page.

Interesting.

 

I copy/pasted the full URL into another browser and the page opened, no login required. Wow, personal info, credit card, health history – all there without login. The URL had an id=55237 parameter. Now I got curious and concerned at the same time. I changed the id=55236? OMG, it opened the page of a different person without any login. 55235 same thing. 44567, same thing. How could be such sensitive data not be secured?

I went on their homepage looking for a number to call. When I finally got through their voice menu, the receptionist asked me for my customer number. “I don’t have customer ID, but I would like to report a security problem on your website.” “Sir – without a customer number I can’t help you.” “Ahmm – seriously? Let me speak to your manager.” “Sir – without a customer ID I can’t put you through to my manager”. “No, you don’t understand – I managed to get access to all your customer data on your website and want to report this.” After endless minutes on hold (and me sweating that my personal and financial data was at risk), I hung up. There were other numbers listed but all ended up in the call center. I tried sending messages to their technical management folks on Linkedin, 30 minutes later – nothing.

I found the number for the PR department. A woman picked up and I explained that someone technical needed to call me back immediately. Finally a young man from the privacy department called me. At first, he thought I was trying to pull a prank. “We are the largest healthcare provider in California, we take security seriously…” I managed to convince him to open a browser and gave him the full URL. Long silence as I had him change the ID to a few different numbers. He became very nervous and promised to call me back in a few minutes. The next call came from a VP. “I want to inform you that we took our complete website offline investigating the security problem, all senior management is informed including the CEO.” Nice. Finally they took me seriously. To their credit, they kept me in the loop over the next few weeks as they crunched through the log files analyzing if IP’s might have accessed more than one customer page.

Interestingly enough, more than a year after this event, Datameer has become one of the leading big data, cyber security solution providers. Today a significant number of our customers are using Datameer to analyze log files to identify abnormal server access patterns and perform security forensics. For example, one of the biggest worldwide logistic companies analyzes access to their public website, tracking pages and APIs to identify attacks early. A major private investment service provider is analyzing server logs from hundred of servers in their service oriented architecture to understand system interaction and behavior. Two of the leading anti-virus companies are using Datameer to identify threats early and understand pattern spread by analyzing honey pot log files and virus scanner signals.

BTW, my healthcare insurance application was rejected and the healthcare company is not yet a Datameer user.

Posted in Cyber Security | 3 Comments | RSS

Calendar heatmap – “Growing a company”

Posted on January 19, 2012 by sgroschupf

 

Data visualization is worth a thousands words.

1.) 2008, 10+ engineers in a small development company doing Hadoop projects.

2.) 2009, November we founded Datameer and raised series A.

3.) 2010, developing the product, private beta, launching Datameer version 1.0 in winter.

4.) 2011, selling the product and doubling the company in less than 6 months.
≈

5.) 2012, focus on execution and growth of the company with regular department meetings.

Posted in Company Culture | Leave a comment | RSS

Welcome, Oracle. Seriously.

Posted on January 12, 2012 by sgroschupf

.

Welcome to the most exciting and important enterprise software market in the last three decades.

And congratulations on your first Hadoop appliance. Despite your original skepticism, putting real computing power in the hands of everybody is already improving the way people store, process and analyze data.  This will have a major impact on our society.

When we built the first Hadoop systems, we estimated that hundred of thousands of business would justify the investment in Hadoop once they understood the benefits.

Next year alone, we project that over 10000 new businesses will come to that understanding. Over the next decade, the growth of big data and Hadoop will continue in logarithmic leaps.  Hadoop literacy is already becoming as fundamental a skill for computer engineers and data analysts as Java or SQL.

We look forward to responsible competition in efforts to distribute this a global open source technology to the world. And we appreciate the magnitude of your commitment.

Because what we doing is increasing social capital by enhancing organizational and business capabilities to gain new insights and better understand the world.

Welcome to the task.
The Hadoop Community.

:)

P.S. Here is the original.

Posted in Big Data Analytics Perspectives | Leave a comment | RSS

Will you beat Santa to Christmas morning?

Posted on December 22, 2011 by Avillamil

It’s that time of the year again when travelers drag themselves to the airport to get on a plane and hopefully meet their loved ones at their destination in a reasonable amount of time.

Thanks to publicly available FAA data, Datameer ran some analytics and found these interesting insights from past Christmas holidays. The data comes to us from the Research and Innovative Technology Administration (RITA) and has detail on everything from individual airports to the specific reasons why delays might happen.

Using Datameer, we generated a spline graph of the average number of incidents over an 8-year period. If you’re traveling on the 21st or 22nd (and to no one’s surprise), anticipate a sea of people at your airport, and possibly delays – these are the peak travel days. Surprisingly, if you’re traveling on the 24th or 25th, most people will have already completed their travels and your journey might be a little less stressful.

So what does this look like when we look at individual airports and destinations?  Using a circular graph chart, we can see the most delayed routes by the frequency of delays below. We set this up in Datameer by first mapping out the travel routes in the worksheet, grouping them and sorting by the highest count in descending order.

We can see from the diagram and illustrated by the thickness of the line above, routes between Chicago O’Hare, Newark, and LaGuardia were most heavily delayed. We know, of course, that Chicago is the largest hub in the US so there’s no surprise there. But what IS surprising is that if we look at the top number of incidents due to weather, Dallas took the top spot in number of incidents in December beating out Chicago and Atlanta, as shown in the diagram below.  So if you’re flying from or through Dallas, you might want to pack some extra reading materials.

And if you’re curious as to which travelers have spent the most time waiting in December, San Francisco travelers get that unfortunate title. Below we’ve taken the total number of minutes spent waiting to arrive at a particular airport by summing up the delayed-minutes field by destination.

Comparing this graph to the one above, we could deduce that SFO has lengthier delays and DFW has shorter but more frequent ones and the data confirms that this is the case.

So, while we don’t expect you can alter your travel plans, we hope you find these insights informative.  There is a wealth of publicly available data like this FAA dataset that can provide valuable insights.  And, Datameer offers a free trial of our solution here that you can use to analyze and explore this and virtually any other data.

Safe travels everyone. Happy Holidays from Datameer.

Posted in Big Data Analytics Perspectives | Tagged , , , | Leave a comment | RSS

Well hydrated

Posted on December 13, 2011 by sgroschupf

 

We at Datameer strongly believe that some of the most important answers to mankind’s challenges are hidden in data. We hope we can contribute to the research efforts of science and academia by building and making available an amazingly simple, yet powerful data analytics platform that enables analysts and scientists to focus on their research without having technology get in the way.

We also believe that lessening our own human impact is key to our future and requires many small steps. So here at Datameer, along with the obvious recycling, we have embraced a culture of low impact, sustainability and going green whenever possible.  To start, we installed energy saving light bulbs in our new offices and banned all Styrofoam containers from our daily lunch deliveries.

One more impactful change we recently made is to stop buying soda and water in plastic bottles. Instead, we provided all staff with beautiful stainless steel water bottles and added filters to our tap water. We used to generate hundreds of empty soda cans and plastic bottles each month, but since we made this minor change, we have been able to cut this waste down dramatically.

 
We’ve also purchased a great soda stream machine to enable us to invent our own organic house made sodas. Turns out, this not only had a green impact but also reduced our spending. On average, we used to spend more than $2 a day per person to supply bottled water and sodas. But after less than a month, the money saved already paid for the stainless steel water bottles and the soda stream machine.

With these simple changes, we will save more than $10,000 a year just in our San Mateo office alone.  More importantly, we are not creating trash in the environment and filling up our landfills with thousands of plastic bottles.

So, go and buy some stainless steel water bottles for your company!

Posted in Company Culture | Tagged , , | Leave a comment | RSS

The Changing Data Landscape, Ready or Not, I’m in for the Ride

Posted on November 3, 2011 by ddoan

 

Post from Dianna Doan, Senior Manager, Marketing Programs

The world is changing, whether we’re ready for it or not.  I spent a number of years in the world of traditional BI with Actuate so I understand the BI space. But few could and would predict the changes that have taken place in the last few years.

What’s causing this change? Its really driven from the likes of Facebook, Twitter and smart phones, the rise unstructured data. This is redefining the world in which we live in. As Gen Y’s begin to take over the social blogosphere, we will see the changes multiply. Retail chains, banks, government, all are plugging into the world we know as ‘big data’.  I joined Datameer recently to dive into this new big data world and the gold mine created for researchers, business analysts and marketeers wanting to make sense of it all.

As I begin to understand the intricacies of what this big data ecosystem is really all about, from Hadoop, to the Hadoop distributors, to Datameer, I realize big data has opened us up to a world of very interesting and often mind blowing analysis and insight.

For example, I started reading my friend Jonathan Chang’s thesis “Understanding, Uncovering and Predicting Links” to understand more about machine learning to prepare me for the job here at Datameer.  He’s a Data Scientist at Facebook, so naturally, his thesis seemed like an interesting extension to what Datameer does, (BI platform on top of Hadoop for big data).

While reading, I stumbled upon some really cool insights that got me excited about what the future holds for the players in this space. Big data is a beast.  It’s wide, it’s expansive and will intersect all of us at some point in our lives over the next few months, if not days.
Allow me to share snippets of Jonathan’s thesis.  Below is a simple, but insightful pictogram of how people are related, made possible by Hadoop.

Figure 1.1 shows what a subset of an online social network might look like. The nodes in the graph represent people and the edges represent self-reported friendship between members. Even in this simple example, a rich structure emerges with some individuals belonging to tightly connected clusters while others exist on the periphery. Characterizing this structure has been one major thrust of network research (Newmanet al. 2006b).

Going beyond the ‘social networks’, and without getting too technical, companies can now see how one person is related to another by building predictive and probabilistic models leveraging software companies like Zementis and their predictive modeling plugins to Datameer to understand and predict consumer buying behavior.

I then started reading earlier posts from my colleagues here at Datameer like  “Predicting the stock market with Datameer or “Fishing the Clickstream…” and my mind started racing.  What other insights can companies gain?  What are companies doing if they aren’t already tapping into big data?  It seems that the way companies look at customer and prospect information and interaction has changed forever.  Without giving away any secrets, I’d love to hear about what insights companies are now capturing in their big data analytics….

 

Posted in Big Data Analytics Perspectives | Tagged , , , , | Leave a comment | RSS

Why I am at Datameer

Posted on October 21, 2011 by bsmith

 

Post from Brian Smith, Regional Director of Sales

I ran the commercial sales division at Vertica Systems for several years prior to joining Datameer. At the risk of going on (and on), I wanted to share my enthusiasm around my professional experience selling big data solutions, and why I am so excited to be at Datameer.

Circa 2011 – We’re all producers and consumers of data in almost every aspect of our professional and personal lives. Analysts anticipate a compounded 40% per year growth rate in corporate data volume, with the lion’s share of the growth in unstructured data. In 2011, the name of the game in big data is Hadoop, and it’s become the Gold Rush of interest. Why?

The economics are compelling – Hadoop is moving out costly analytic databases and warehouses, driving IT to re-look at ADBMS sales cycles, shifting IT dollars and vendor roadmaps, and generally wreaking havoc in the traditional vendor community. We’ve gone from one or two distributions to nine in the last year! And, literally every vendor in the BI/DBMS space has a Hadoop connector, the latest being the recent Oracle announcement. Everybody is on board this train – All this based upon the premise of unlimited scale and data variety at a fraction of traditional costs.  Technical challenges exist, but its clear that there’s a sea change.

Prevailing winds – In light of large corporate BI and data warehouse investments, companies are typically using Hadoop as a staging and storage area to start… This involves parallel loading high volumes of raw enterprise data into Hadoop for subsequent scrubbing, (ETL), in route to passing smaller subsets for analysis to the existing BI stack through connectors.

There are several reasons why this approach will be short-lived:

  • Business analysts in this process are by definition separated from the data that they need to do their job – The end product that they receive is a limited subset of available data for analysis, simply because traditional BI cannot handle the volumes of data.
  • Time value of data – the development/IT resource required to ETL the data and move/process the subsets introduces unacceptable delays. People have jobs to do and analyses to rely on…
  • Cost – at each stage of this process, costs are created and duplicated across developer, vendor license, support and training , only to yield a partial answer based on a fraction of the data.  It’s incredibly inefficient and complex.

Conclusion?  Where such obvious inefficiency exists in a business segment where literally billions of dollars are spent every year to achieve business efficiency (!), alternative solutions will rapidly emerge…That’s why I am at Datameer.  (“Data Ocean” in German)

“Datameer is the first BI/Analytics platform built natively on Hadoop.”

On the surface it sounds interesting, but in practice the solution is game-changing.  The Datameer Analytic Solution (DAS) connects business users directly with the entire volume and variety of their raw Hadoop data and makes it available for comprehensive analysis.

DAS does this in a way that any business user will find to be simple, efficient, and completely intuitive. IT will love it as well, as Datameer frees up the IT folk to focus on more value add work rather than writing code to get data into Hadoop.

Practice makes perfect…Datameer gives business users an iterative “prototyping” capability for the data pipeline against a small sample of the data prior to running production analytics against the Hadoop cluster. (IT guy smiling…) It’s simple, practical, and directly in line with how the business needs to access, analyze and consume data – all without stepping on IT’s toes.

Under the covers, DAS generates Java/MapReduce code that runs natively on the Hadoop cluster. All current Hadoop distros are supported – we’re Switzerland when it comes to platform support for Apache, Cloudera, MapR, IBM and the rest, we run all of it in a browser on Windows, Mac and Linux.

DAS is an open book at every stage of the data pipeline, with plug and play support at each phase – integration, analysis and visualization. So you can pick and choose,  plug in your own custom analytic functions, use your visualization tool of choice, or simply use DAS end-to-end as an integrated stack.  What’s not to like?

I attended an Hadoop user meeting in San Francisco a few weeks back when I first joined Datameer. One of the moderators in the small group discussion was a local IT/Development manager responsible for supporting a 200 node Hadoop cluster. He made a very telling comment:

“The other day, one of my business guys asked me why I couldn’t just ship him a half a petabyte of .csv files?”

Game on!

Posted in Uncategorized | Tagged , , , , , | 1 Comment | RSS

Whose Hadoop is Bigger? Really…

Posted on October 12, 2011 by sgroschupf

 

Guest post from CEO Stefan.

Actually, Datameer contributed the most code to the Hadoop ecosystem.

After weeks of hard work we made a surprising discovery, actually 99% of the Hadoop ecosystem code was written by Datameer engineers.

We hired an external, independent, respected analyst firm that had 5 PhD’s working on a new generation of algorithms that analyze the Hadoop ecosystem source code, jira posts, emails and ideas contributed in verbal conversations.

The breakthrough was that we analyzed the object inheritance and the call stack to weight the importance of each line of code. We also took the mental stability of contributors into account. BTW, if you still wondering what was in my coffee this morning, you don’t get my German sense of sarcasm.

Well….

When I joined the Nutch project in the early 2000’s, I was known to communicate my strong points of views very loudly in the community. I guess I lost some steam over the years, I have not even published a blog post in last few years and the Hadoop & Co mailing lists are on read only subscription.

But I felt I had to speak up about all this commotion around “my Hadoop is bigger than yours” currently lighting up the community.

I tried to take some wind out of this conversation over the last few months by using our product to analyze the Hadoop source code and present, in a very fun way, some Hadoop source code insights here. These analytics discovered the longest email conversation for the smallest code change or longest commit comment for the shortest change, etc etc.

So now we find our partners and friends sparring over whose contribution is bigger than the others. Frankly, this is all surprising to me since we have so much more work to do to move Hadoop forward. Don’t get me wrong, we love Hadoop for what it is but we all can agree that the code is still a work in progress, monolithic, difficult to test and concepts like inversion of control do not exist… I could go on for a while.

So actually I’m happy to announce that our own awesome engineering team is not responsible for this but instead focused on working on a great analytics product on Hadoop that brings great value to our customers.

Here at Datameer we work hard but also make sure we have a good time including sharing a laugh over the most stressful situations.

In that spirit we would love to contribute a laugh to the ongoing “civil war” in the Hadoop ecosystem.  To commemorate this epic discussion, we have designed a special t-shirt that we would love to share free with the community.

Just fill out the form and we send you your own free shirt (one shirt per household, while supplies last).

Send me one of these cool “My Hadoop is Bigger Than Yours” t-shirts.

Ok, people, now back to work – lets build some great technology instead of arguing about lines of code.

P.S. We have some customers using DAS, their Hadoop is for sure bigger than yours. :)

Stefan Groschupf

Posted in Uncategorized | Tagged , , , , , | 3 Comments | RSS

Predicting the stock market with Datameer

Posted on August 2, 2011 by mlieber

I recently read an interesting research paper by Johan Bollen, Huina Mao, Xiao-Jun Zeng, from Indiana University entitled “Twitter mood predicts the stock market,” that investigated whether “collective mood states derived from large-scale Twitter feeds” correlated with the value of the Dow Jones Industrial Average. What they found was that their algorithm not only paralleled market changes, it predicted them, with startling 87.6 percent accuracy!

As a provider of Big Data analytics software, we see this type and scale of problem all the time at our customer sites, particularly the correlation of structured and unstructured data.  For this particular study, let’s see how easy it is to reproduce this analysis with Datameer Analytics Solution (DAS).

First, let’s download the Dow Jones stock values data. You can get this freely, from Yahoo for example (DJIA). This is a simple CSV file format showing daily prices. You can also download other data, such as the NYSE Composite index, to experiment with.

Second, let’s get some Twitter data from their API, known as the “fire hose”.  For this test, we’ll use raw data (i.e. unfiltered tweets) for the entire month of March 2010.

Let’s load all of this data into DAS.  In our new 1.3.x version, you can simply upload a file from your local computer, so let’s load our Dow Jones data this way:

Upload file

Then let’s load the tweets, via an Import Job, which understands Twitter’s format natively:

This amounts to about 30 GBs of compressed data for the month.

Let’s first try and get a more accurate data set, by filtering the tweets to US users. This is something that our researchers apparently did not do: “we note that our analysis is not designed to be limited to any particular geographical location”, but this is easy to do with DAS.

We did not have OpinionFinder nor Google-Profile of Mood States at our disposal to perform sentiment analysis (these could make great new functions some day that could be added via our API!), so let’s use instead a simplified version by taking a list of positive terms (Bag of words model), and find the tweets that contain these terms.

To do this in DAS, let’s import a list of such terms (this can be easily found on different web sites), and create an outer join with our tweets, and then filter to find the tweets that contain these positive words.

In DAS 1.3.x you can filter with a complex expression directly in the ‘Advanced’ tab:

Now let’s count the positive tweets per day. This is just an aggregation sheet using GROUPBY and counting (this is the sheet preview result below, not the actual count on the full data set yet):

This represents the amount of “happiness” mood by day.

Next, let’s create a new workbook to join the resulting worksheet of “happiness” mood per day with our Dow Jones Industrial Average (ticker ^DJIA) data:

A very helpful feature in DAS is the fact that we can seamlessly exchange the DJIA history with, say, NYSE Composite index history via the ‘Exchange Datasource’ button and rerun the workbook to test the correlation with data from other exchanges. This requires no further changes or additional work (more details on this later).

Here is the resulting sheet of our join:

As you may know, building analyses in DAS works on a sample of the entire data set, which enables users to easily interact with the data until they’re satisfied with the analysis.   Now that we are happy with our analysis, let’s run the workbook on the entire data set.

Now let’s graph the tweet “happiness” mood and the DJIA market closing value over the same days and compare:

As the researchers pointed out, we can note a correlation between our Twitter “happiness” index, and how the Dow Jones Industrial Average went up or down between two and six days later; first see the progressive parallel mood upswing (1), then the drop on (2) (drop in Twitter mood followed by drop in the DJIA value), an upswing again on March 19 at (3) – Twitter mood goes up quickly followed by the DJIA value -, then a parallel drop on March 22 followed by the same drop in DJIA value a few days after  (see (4)). A similar correlation can be found by using NYSE data instead.

Disclaimer note: this analysis was done on only a month’s worth of data, but could be expanded to more data very easily with no further changes in the analysis. We also used a very simple technique of sentiment tracking, which could be further improved. Finally, due to the small amount of data, we did not have outliers in the data like our researchers did (“significant socio-cultural events such as the Presidential election and Thanksgiving, short-lived uptick in positive sentiment specific to those days”), but we could easily filter had we worked with more data.

Pretty easy, wasn’t it?  That simplicity: combining all kinds of data ad hoc, while harnessing the power and scalability of Hadoop to extract insights, is what Datameer is all about.  I hope you’ve enjoyed this post, and you can learn more about Datameer at www.datameer.com/products.

Posted in How-to, Uncategorized | Leave a comment | RSS