Big Data

[andriambololona]

Can we use BIG DATA with APL? Everyone talk BIG DATA evrywhere: in Business Intelligence, ....

[StephenTaylor]

Dyalog would make an excellent 'front end' for presenting data drawn from say kdb+ systems.

I recall we presented something like this in 2007 at the Kx Systems conference in Ireland and at the Dyalog conference in Princeton later the same year, but were unable to rouse much interest from the terabyters.

[andriambololona]

Thank Stephen. I got it right. There are several debates about it, especially in the field of Business Intelligence.
I let you go to www.qlikview.com

[paulmansour]

In addition to Kx as noted by Stephen, the big vendors have opened their own "big data" systems to the public. You can have a hosted database from Amazon (named "Red Shift"), and Google offers "Big Query". These are quasi-relational; a database consists of multiple tables. These all have an api that is probably no harder to access from APL than any other platform. Many of the so-called BI tools like Tableau and Qlikview can access them.

There are also "NOSQL" Databases like CouchDB from Apache, and mongoDB. These are key-value stores the foundations of which I don't think are too different from an APL component file. They have formalized sharding and replication and all that stuff to handle big data and system failure. Lots of websites with millions of users use these things. Implementing something like this would make a really nice Dyalog project...

So, you can use APL on data from big data systems, and you can use APL to write big data systems. However, most of the talk of big data is exactly that, talk. With 64 bit machines, hundreds of millions of rows don't even count as big data. I think it is better for the APL community to focus on "big problems" rather than "big data".

[andriambololona]

Hi Paul
Thank you for your explanation on big data. It is really clear. All this is of the bla bla and it is some delusion