Big Data

No need to worry about going "off topic" here
Forum rules
This forum is for general chit-chat, not necessarily APL-related. However, it's not for spam or for offensive or illegal comments.

Big Data

Postby andriambololona on Tue Feb 19, 2013 10:58 pm

Can we use BIG DATA with APL? Everyone talk BIG DATA evrywhere: in Business Intelligence, ....
andriambololona
 
Posts: 6
Joined: Tue Jun 15, 2010 10:06 am

Re: Big Data

Postby StephenTaylor on Thu Feb 21, 2013 9:09 am

Dyalog would make an excellent 'front end' for presenting data drawn from say kdb+ systems.

I recall we presented something like this in 2007 at the Kx Systems conference in Ireland and at the Dyalog conference in Princeton later the same year, but were unable to rouse much interest from the terabyters.
User avatar
StephenTaylor
 
Posts: 31
Joined: Thu May 28, 2009 8:20 am

Re: Big Data

Postby andriambololona on Mon Feb 25, 2013 10:44 am

Thank Stephen. I got it right. There are several debates about it, especially in the field of Business Intelligence.
I let you go to www.qlikview.com
andriambololona
 
Posts: 6
Joined: Tue Jun 15, 2010 10:06 am

Re: Big Data

Postby paulmansour on Tue Feb 26, 2013 1:01 pm

In addition to Kx as noted by Stephen, the big vendors have opened their own "big data" systems to the public. You can have a hosted database from Amazon (named "Red Shift"), and Google offers "Big Query". These are quasi-relational; a database consists of multiple tables. These all have an api that is probably no harder to access from APL than any other platform. Many of the so-called BI tools like Tableau and Qlikview can access them.

There are also "NOSQL" Databases like CouchDB from Apache, and mongoDB. These are key-value stores the foundations of which I don't think are too different from an APL component file. They have formalized sharding and replication and all that stuff to handle big data and system failure. Lots of websites with millions of users use these things. Implementing something like this would make a really nice Dyalog project...

So, you can use APL on data from big data systems, and you can use APL to write big data systems. However, most of the talk of big data is exactly that, talk. With 64 bit machines, hundreds of millions of rows don't even count as big data. I think it is better for the APL community to focus on "big problems" rather than "big data".
paulmansour
 
Posts: 302
Joined: Fri Oct 03, 2008 4:14 pm

Re: Big Data

Postby andriambololona on Sat Mar 02, 2013 11:16 am

Hi Paul
Thank you for your explanation on big data. It is really clear. All this is of the bla bla and it is some delusion
andriambololona
 
Posts: 6
Joined: Tue Jun 15, 2010 10:06 am


Return to Chat

Who is online

Users browsing this forum: No registered users and 1 guest