At the recent Gartner BI conference, I was struck by the focus on “big data”. It wasn’t the level of attention that was surprising, big data is everywhere.  It was how deeply it has now penetrated into the traditional BI space.  Everyone was talking about it in every vendor’s booth (large and small) and in most of the conference sessions, whether by a Gartner analyst or in a paid vendor session.

Why all of the attention?  It’s probably not what you think. This isn’t a case of big data analytics replacing traditional BI; BI is here to stay because of how well it works for reporting on transaction data. But the BI vendors realize there’s a huge market opportunity when it comes to big data analytics as a complimentary technology. Big data analytics is what allows for insights across all of the data including BOTH transaction and interaction data, which is a combination of traditional structured data along with semi-structured and unstructured data.

Big Data, Big Use Cases

While “big data” is not a great term to describe what is going on in analytics today, Gartner’s original framing of the issues around the volume, variety, velocity and volume of data is a good way understand the new big data use cases.

For volume, it’s pretty simple. If you can analyze more data, then your insights will be better. The ability to analyze more data has wide-ranging applications in financial services, retail, telecommunications, medicine, pharma and other areas.  For example, in financial services, if you can analyze a longer time span of transactions, say over 5 years, you can detect fraud patterns that simply are not apparent when looking at 3 months of data. And while some traditional BI solutions can handle large data sets, the cost of hardware to do that has become prohibitive.

But big data gets way more compelling when you get to data variety and velocity.  Data variety is driven by new devices and user behaviors. Between smart phones, social media, websites, games and device-to-device communications, we have transitioned from a transaction society to an interaction society. Inherently, the data that results from interactions is semi-structured or unstructured, and the fact of the matter is that traditional BI solutions simply cannot analyze this data. By analyzing interaction data by itself or with correlations to transaction data, we can now ask and answer questions that were simply not possible before, allowing for new insights into customer behavior, business operations, and company performance.

And finally, the velocity of data, both throughput and new sources, continues to skyrocket. The amounts of new data each minute, hour and day overwhelms traditional BI as does the need to quickly connect, integrate and analyze new data sources with what data you already have. Traditional BI, constrained by the need to pre-model the data due to limitations of storage and compute, simply can’t respond quickly enough to new data sources, meaning that those insights lag behind today’s pace of business.

Underneath the Big Data Hood

All of the traditional BI vendors have jumped on the big data wagon by hitching their cart to Hadoop. Hadoop is an open source storage and compute engine that is an infrastructure enabler here as it provides linear scalability using commodity hardware.  And Hadoop processes all types of structured, semi-structured and unstructured data. BI vendors generally offer connectivity to Hadoop via Hive, the open source data warehouse that is a part of Hadoop.  And while this gives traditional BI users access to data in Hadoop via an SQL-like (structured query language) interface, it ignores the core advantages of Hadoop and big data analytics.  Hive only deals with structured data (leaving out all of the other semi-structured and unstructured data in Hadoop) and requires technical skills in writing SQL to get at the data. What you end up with is limited analysis of a small subset of your data.

So lets be clear, there are major differences between a native big data analytics solution on Hadoop and connecting traditional BI solutions to Hadoop data via Hive. Using Hive is like looking at your business through a soda straw versus looking through a pair of wide-angle binoculars with big data analytics. The point of big data analytics and Hadoop is to turn the user loose on all of their data without the limitations of how much data, its diversity, or its constantly changing sources.

Datameer uses Hadoop natively as its back-end storage and compute engine for end-user focused, big data analytics.  Datameer and Hadoop don’t care whether the data is structured, semi-structured or unstructured.  Datameer use of Hadoop’s scalability and data flexibility means you don’t have to pre-model the data, so business users can quickly analyze anything and everything.  The end game is dramatically faster “time to insight” that fits the style and pace of business need in the 21^st century.

What does big data analytics do that my existing BI software doesn’t?

Ask any BI analyst or user – there is a tremendous amount of market noise about big data analytics being the answer to everything.

But of course, it’s not. Big data analytics isn’t about throwing away your existing BI, it’s about the new use cases that big data analytics brings to the table. With traditional BI, essentially all of the data is highly structured, usually transactional, and resides within a database, data warehouse or OLAP cube somewhere. With some small variations, all traditional BI uses the same process:  business users determine what questions they need answered and then the IT folks go find the data sources and build a data schema that answers those questions. Examples of traditional BI use cases include monthly sales or product reports, product profitability analysis and customer survey results.

Big data analytics are all about new use cases. Big data analytics and Hadoop are more flexible than traditional BI. Hadoop can store any data, no matter the source or whether its structured or unstructured. This enables creative discovery of your data and correlations and analysis across all of the available data without the limits of a data schema. The business, armed with these rich new data sources, can explore and ask questions around brand sentiment, product strategy and asset utilization that require both structured data together with weblogs, social media and other unstructured data. These analyses provide insights into how to offer prospects a more personalized user experience, a better understanding of customer behavior, or to make more relevant recommendations to customers based on what they’ve purchased in the past.

In short, the gist of big data analytics is to let users loose on all their data, no matter where it came from, in order to see a more complete picture.

Will Hadoop replace my Data Warehouse?

I get this question a lot. The answer is not unless you want it to. At least not for traditional BI queries. The dream of a central repository for all of one’s data has been around for years. Data warehousing technology was thought to be the ideal central repository of all data but companies find that the requirements of a rigid data schema and the need to rework the schema whenever a new data source is added simply is too time consuming for today’s fast paced business climate. The end result is a static data store that soon becomes out of date. And, of course, existing data warehouses are being challenged with growing varieties and volumes of data. One prospective customer I spoke with recently summed it up best:  “Our data warehouse is where data goes to die.”

Hadoop addresses the challenges around data variety, storing semi and unstructured data like web, application logs, and social media in their raw form as well as structured data. In some cases where companies’ existing data warehousing infrastructure is overwhelmed by huge volumes of structured data, Hadoop can serve as a lower-cost alternative to a proprietary database.

Hadoop is best viewed as a complimentary technology to a data warehouse. A data warehouse is usually the better option for reporting on moderate amounts of transactional data. And they can also serve as data sources for Hadoop. Once the data is in Hadoop, then Datameer can analyze any subset or all of it, structured or unstructured. Where Hadoop is most complimentary is when analyzing semi and unstructured data along with structured transaction or catalog data. Datameer’s spreadsheet-like UI makes those correlations and analyses very easy.

What about Hive and isn’t Hive free?

Companies exploring the big data and Hadoop ecosystem often look at Hive. Hive is a data warehouse infrastructure built on top of Hadoop that allows for querying and analysis of structured data. For companies that have specialized engineering talent that understand SQL (structured query language) and who don’t have semi and unstructured data components, Hive can appear as a viable fit.

However, many of our customers looked into Hive and found its limitations. Time to insight is slow due to the complex programming language required in joining multiple data sets as well as maintaining data schemas for analytics. Hive doesn’t provide tools for ETL, analysis or visualization so additional technologies and expertise are needed to tie those components together. With Datameer and its spreadsheet UI, no technical programming in SQL is needed.

Hive, like structured data stores used in traditional BI, requires tables and schemas that are then queried via a SQL-type of language. This approach carries the same limitations of many existing systems in that the questions that can be explored are limited to those that have data in the Hive schema rather than the full raw data that can be analyzed with Datameer. As I mentioned earlier, one of the strengths of Hadoop is the ability to do ad-hoc analysis directly on all the raw data and then correlate back to transactional data within a database. Forcing data into a schema with Hive negates the flexibility that  Hadoop provides.

Every day, we see companies using Datameer and Hadoop to get fast insights into customer behavior, company processes and business performance across all of their data.  BI and traditional data warehouses will continue to play a role going forward, but it’s the new use cases that are driving all the excitement around big data.

Two and a half years ago, Datameer was born. It started with six guys who all worked together for 15 years through six different companies. They built Datameer for one reason alone – to create a powerfully simple solution to help companies and organizations gain faster time to insight on any type and size of data.

Fast-forward to today. Because of the amazing demand for Datameer from financial firms to major retailers to online gaming companies, we’re now easily 10x the size of the company that started in Halle, Germany back in 2010. And we’re spread out all over the globe. So, last week, we did the unimaginable. We all took the time to convene from our various corners of the world to meet in person back where it all started. For a lot of us, it was the first time we had actually met face-to-face.

For five straight days and nights, we were truly one company. We talked a lot about Datameer, past, present, and future, and even had some customers come and talk to us about their use-cases. We sampled plenty of fine German food and drink, laughed a lot, and really took the time to get to know one-another. We even picked up an honorary German employee somewhere along the way named Wolfgang. Then very early Sunday morning, begrudgingly, we all said our goodbyes, and headed back to our various corners of the world to continue on our mission to make Datameer the best Big Data Analytics solution on the market.

So yeah, I can say with confidence that this week was different. In the best way possible.

“There are 4 fundamentally different problems in the world of “Big Data”.  First is collecting and storing data. The second is finding data when you need it. The third is turning that data into useful information. The fourth is differentiating what does and doesn’t need to be secure, and keeping it safe…

…I don’t think anyone is even close to cracking #3. Whoever comes up with that solution will be the business software winner. As to security, all that can be done is the best than can be done.”

As my colleague Alex discussed Hadoop security (number 4) previously, I thought I would respond to 1, 2 and 3 with some examples of how our customers accomplish these things using Datameer.

Collecting and Storing Data
A department store chain wanted to better understand their customer’s needs in order to maintain and increase wallet share.  The department store uses Datameer to import all types of data (transaction data, competitor pricing data, geolocation data and more) into Hadoop in the raw data format from where the data is generated.  Some of this data is then cleansed and transformed for analysis and some of it is exported into existing data warehouse technology for different use cases.  This department store now has access to more data, is able to make correlations between the different data sets, and is able to store  current data sources for longer time periods.

Finding Data when Needed
A financial services company has millions of transactions per day.  Due to the high cost of storing data, the company only kept a few months of transaction data in an MPP database and then transferred the data to tape which limited their data analysis capabilities.  The cost effectiveness of Hadoop allowed the company to keep one years worth of data and is planning on expanding the cluster to include more transactions.  Their challenge was the time it took to access and analyze the data writing one-off map reduce jobs for Hadoop.  With Datameer, the technical and business users now can easily access and analyze the data as needed with Datameer’s spreadsheet interface.

Turning Data into Useful Information
Customers using Datameer are doing an amazing job turning data into useful information.  An example is an internet security company that takes data from malware and other threats, uses ad hoc analysis and then builds solutions to solve threats to networks.  The data is in several formats (structured, semi-structured and unstructured).  Finding answers to questions requires ad hoc data access and exploration and Datameer grants access to all users direct access to the data.

To learn more about how people are using Datameer to solve problems, read our Big Data Analytics solution briefs on Retail, Telco, Internet Security, Financial Services and more.

This is done via real-time sensors.

As an enthusiastic data analyst, I realized I could use Datameer to get more insights about my parking problem. Sure, SFPark already gives you real-time information about parking (on the first page of http://sfpark.org/),  and analysts have already taken a look at pricing .

What I was looking for was a little different: an overall perspective of parking availability and change over a long period of time.

Datameer allows you to bring in data from a variety of different sources. In this case SF Park‘s API returns JSON data, so we used the built-in adaptor for this. We can easily schedule this import of data in an automated way to say, every 30 minutes or so to start building our time series analysis.

An interesting new feature about importing is you can now partition the data, in the same way Hive does, but in an easy user-friendly way, in this form:

You can then work with a subset of the data, in this case chosen to be on a day-granularity. More information about how to set up partitions here.

Once the data is imported, we can easily deconstruct the JSON data with our set of JSON functions. The complete way to do this and deal with JSON data downloaded from the web is described in details in our video section, but basically we end up with something like this:

We can now construct an analysis and visualize the data to infer some statistics about for example at what time in downtown San Francisco are the garages most full (this was run over a 3 month period in 2012):

It appears from this graph that if you work there, the earlier you arrive the better, because the garages get filled up pretty quickly; people seem to start leaving around 2pm (with a maximum availability of 32%), so it seems like the general trend is to work early in the day, perhaps because all of the financial institutions in that area?

After 5pm (1700) the general availability is around 42%, and it gets easier to park after that.Given that there are around 440,000 total spaces in San Francisco, does the day of the month make a difference in parking space availability? This graph shows that it doesn’t seem so:

The weekends (Feb 26, Mar 3) show the spaces are mostly unoccupied, whereas the average number of spaces occupied on weekdays is around 15,000. Of note, we have an outlier of over 20,000 spaces occupied on Feb 22, not sure what happened that day? (Please tell us if you know!).

Let’s see if there is any drastic difference in garage space occupation for this range of days, per garage or area:

It seems like the number of spaces available is fairly evenly distributed for the garages in the dashboard, except for the Golden Gateway one.

Let’s just add a sort and see the top occupied garages and areas to avoid:

It seems like overall, the Leavenworth area, as well as the south Embarcadero road seem to be pretty bad areas for parking.

What if you wanted to see the results of this study on a particular timeframe only like say on a per-day basis, without having to change the analysis? You can simply enable the result set to be partitioned, like demonstrated here, with this nifty sunburst to control the partition level:

This analysis is refreshed with the latest data on a continuous basis, so let us know if you want to see the latest results as it is being continuously updated.

This study could be further deepened by looking at the prices each garage is charging, and choosing the lowest-price one along with its availability.

Yesterday, Datameer announced the release of Datameer 1.4.  One of the updates is tightening security on data layer and the user layer.  As Natty brought this up in his post, “one of the more confusing topics in Hadoop is how authorization and authentication work in the system”.  Datameer customers have the same concerns as they expand users from a handful of data analysts to hundreds of business analysts.

Datameer manages both Hadoop and end-user security and provides additional functionality for authentication and authorization.  Below is a quick overview of Datameer security features.  You can read the details of the content in the datasheet Datameer Security (PDF).

Authentication
- Datameer provides LDAP / Active Directory (AD) authentication and integration
- Datameer supports connectivity to LDAPS (LDAP over SSL)
- Datameer supports operating in Hadoop environments secured by Kerberos

Authorization
- Datameer provides role-based access with delegation

For more information, here are some additional resources.
- Datasheet: Datameer Security
– Datameer Documentation
- Free Trial of Datameer

If you have any questions, contact us.

Eights months ago I met Stefan Groschupf, co-founder and CEO of Datameer. We spoke for an hour about the vision of Datameer, the company and products, my background and what I do for a living. I was impressed with Stefan. Datameer is developing products and solutions to help solve a new generation of needs and business problems resulting from extremely large amounts of new and strange data (Big Data, unstructured, machine-generated). Datameer has attracted a leadership team that has a vision.

The vision is attractive. There is plenty of hype around Big Data. Like Boris Evelson said, #8, I think “Big Data will move out of silos and into enterprise IT”. I am skeptical, like Neil Raden of Doopmarts and that the “proliferation of Hadoop” could lead to “isolated models, definitions, datasets not understood or available to the enterprise”. One of Deloitte’s Tech Trends for 2012  is that “Data Goes to Work”. The hype around Big Data is just hype unless businesses are actively using the products to solve business problems. Companies are using Datameer in production to provide their IT and business users with incredible insights into their unstructured and structured data. If you have data and want to explore it, download the trial version and try Datameer with your data.

I joined Datameer because I believe in the vision to fundamentally change the Business Intelligence and Analytics market. I want to help data driven companies make intelligent decisions.

Hadoop is the first open source technology that created a big market. Unlike past open source successes such as Linux and MySQL that brought low-cost alternatives to existing technology markets, Hadoop has led the creation of the multi-billion dollar big data analytics market. While a number of traditional, commercial software and hardware technologies have tried to address big data analytics, they are limited by high cost, lack of linear scalability and inability to process unstructured and structured data.

Hadoop is revolutionary in the way that analytics are done. So commercial interest has exploded with virtually every large company having an Hadoop project and smaller, innovative social media, gaming and Web 2.0 companies leveraging the scalability and cost effectiveness of Hadoop to break new ground in interaction analytics. Every major hardware vendor has some form of Hadoop offering now to help meet the demand for more computing power and data storage. Even Oracle, who questioned the relevance of NoSQL databases over the last several years, announced an Hadoop offering a few weeks back.

This week, Datameer and Microsoft announce their partnership to bring Datameer’s end-user BI platform to Microsoft’s new Azure-based Hadoop offering. We are excited about this partnership for three reasons. First, Microsoft’s embrace of Hadoop as a key analytics platform expands the Hadoop eco-system in a big way. Second, Microsoft made the spreadsheet the industry standard interface for basic number crunching with over 500,000,000 Excel users. And third, given the success of the spreadsheet user interface for both Microsoft and Datameer, this partnership is a perfect match to bring Hadoop-based spreadsheet analytics to every business user.

We welcome Microsoft to the Hadoop community and look forward to working with them and our joint customers to connect people to the world’s data.

Stefan Groschupf

This argument gave me the idea of comparing stats of the two quarterbacks for this weekend’s Super Bowl. But rather than look at overall career totals/averages (because we all know Brady’s overall stats will reign supreme), I decided to try to compare their QB stats for only “in the clutch”. When I say “in the clutch”, I mean which quarterback delivered the most when it counted, that is, when the game was on the line. For my definition of “clutch”, I will look at who passed for more touchdowns in the all important and closing 4^th quarter. And also who passed for more touchdowns in the 4^th quarter with less then 5 minutes remaining in the game. With the pressure on, game clock is dwindling, which quarterback reigns supreme?

To perform this analysis, I needed stats that broke down at the play-by-play level. Most NFL stat sites only give stats at an end of game level (i.e. final box score). I found such play-by-play data at  www.advancednflstats.com.

As you can see from the shot above, the data is stored at the individual play level, tracking what quarter, what time, and a description of the play. The website had data going back to 2002. Tom Brady’s rookie season was in 2000 and Eli Manning’s was 2007. So there was enough data to cover the majority of both quarterback’s careers. In fact there was well over 384,000 plays performed in the entire regular season NFL games, dating back to 2002.

Using the trial version of Datameer, I loaded this data and performed some aggregations. Didn’t need to write any code, simply used the wizards to guide me through my analysis.

My first step was to import or ingest the data. Data from the site came in multiple .csv files, one for each year dating back to 2002.

After specifying the data details, Datameer was able to parse through the data and recognize all the column headers and data types.

Once the data had been imported, I opened up a workbook and linked my play-by-play data into a worksheet.

Since this data represented all plays in the NFL for all teams, I used the filter wizard to get only the records for plays by Tom Brady and Eli Manning.

Since the play-by-play data contained descriptions of the play, I applied another filter to find all the touchdown passing play descriptions, but also weeded out interceptions, reversed calls, and non plays. And since we’re looking for stats “in the clutch”, I’m only concerned with the 4^th Quarter.

Next I created new columns to flag if the record was a play for Brady or Manning. Since all the details are in the play’s description field, I needed to use a “contains” function to check which quarterback the play was for. By simply double clicking into an empty column I was able to launch the wizard to help me configure the “contains” clause on the description field. I created two new columns, one for Brady and one for Manning.

Now that I have a record for every play and flag for both Brady and Manning, I could now create some analysis. By creating a new sheet and using the GROUPBY function, I grouped my data by the yearly football season.

I then performed a group count on my Brady and Manning boolean flags. One column for Brady and one for Manning.

I saved this workbook and ran it against the entire dataset.

You can now see the results, number of passing touchdowns for Brady and Manning, only in the 4^th quarter. A quick plot onto Datameer’s dashboard shows me the following graphs: red lines for Brady’s stats and blue lines for Manning’s.

Then by following the same process above, but now filtering for touchdowns in the 4^th quarter and less than 5 minutes remaining in the game, I get the following graphs:

This tells us that Eli Manning is actually better “in the clutch” compared to Tom Brady. Going back a couple years you will see that Manning has, for the most part, matched Brady’s stats. But this is all about “now” and coming into 2011, Manning scored 15 touchdowns in the 4^th quarter and 10 with only 5 minutes remaining in the game. Brady’s numbers for the same year are 12 TD’s in the 4^th quarter and 7 TD’s with only 5 minutes remaining in the game. So when the game is on the line and time is running out… Eli Manning is your elite QB!

While this data set is small, it shows how easily one can analyze data using Datameer. And since Datameer runs on Hadoop, we could easily scale up to billions of records.

So go ahead, download our trial version of Datameer and see what interesting stats you can come up for the Super Bowl.  And don’t hesitate to send us your results, who knows, it might be our next blog post!

http://datameer.com/products/download-trial.html

BTW, names have been omitted on advice of our lawyers.

It all started when I tried to sign up my then fiancé for a health care plan. After creating the account, the “request a quote” form was long, more than ten pages of questions about pre-existing health conditions. I ended up not having all of the information at my fingertips, luckily there was a save and continue later button. A few days later I went back and started typing in the domain name and my browser recommended the last URL of the domain. To my surprise, the form page with all my information opened. No redirect to a login page.

Interesting.

I copy/pasted the full URL into another browser and the page opened, no login required. Wow, personal info, credit card, health history – all there without login. The URL had an id=55237 parameter. Now I got curious and concerned at the same time. I changed the id=55236? OMG, it opened the page of a different person without any login. 55235 same thing. 44567, same thing. How could be such sensitive data not be secured?

I went on their homepage looking for a number to call. When I finally got through their voice menu, the receptionist asked me for my customer number. “I don’t have customer ID, but I would like to report a security problem on your website.” “Sir – without a customer number I can’t help you.” “Ahmm – seriously? Let me speak to your manager.” “Sir – without a customer ID I can’t put you through to my manager”. “No, you don’t understand – I managed to get access to all your customer data on your website and want to report this.” After endless minutes on hold (and me sweating that my personal and financial data was at risk), I hung up. There were other numbers listed but all ended up in the call center. I tried sending messages to their technical management folks on Linkedin, 30 minutes later – nothing.

I found the number for the PR department. A woman picked up and I explained that someone technical needed to call me back immediately. Finally a young man from the privacy department called me. At first, he thought I was trying to pull a prank. “We are the largest healthcare provider in California, we take security seriously…” I managed to convince him to open a browser and gave him the full URL. Long silence as I had him change the ID to a few different numbers. He became very nervous and promised to call me back in a few minutes. The next call came from a VP. “I want to inform you that we took our complete website offline investigating the security problem, all senior management is informed including the CEO.” Nice. Finally they took me seriously. To their credit, they kept me in the loop over the next few weeks as they crunched through the log files analyzing if IP’s might have accessed more than one customer page.

Interestingly enough, more than a year after this event, Datameer has become one of the leading big data, cyber security solution providers. Today a significant number of our customers are using Datameer to analyze log files to identify abnormal server access patterns and perform security forensics.  For example, we have customers that analyze their public websites, tracking pages and APIs to identify attacks early. A major private investment service provider is analyzing server logs from hundred of servers in their service oriented architecture to understand system interaction and behavior. Two of the leading anti-virus companies are using Datameer to identify threats early and understand pattern spread by analyzing honey pot log files and virus scanner signals.

BTW, my healthcare insurance application was rejected and the healthcare company is not yet a Datameer user.