Reading streaming data from external Websocket connection

APL-related discussions - a stream of APL consciousness.
Not sure where to start a discussion ? Here's the place to be
Forum rules
This forum is for discussing APL-related issues. If you think that the subject is off-topic, then the Chat forum is probably a better place for your thoughts !

Reading streaming data from external Websocket connection

Postby woody on Thu Aug 01, 2019 3:02 am

Greetings.

I need a few pointers to get me going in the right direction regarding how to connect an APL session to read streaming stock market data from an external Websocket service.

My primary goal is to read and STORE the streaming stock market data ... perhaps in an APL component file. It's a lot of data ... and a single day of stock quotes for one Stock Symbol might be 100,000 records (100 bytes each).

Here's the HTTP steps to setup the streaming HTTP URL from the Stock Market data broker.
I know how to do this ... but I am not sure how to connect to the streaming data.

curl -X POST "https://api.tradier.com/v1/markets/events/session" \
-H 'Authorization: Bearer <TOKEN>' \
-H 'Accept: application/json'

Response (JSON)
{
"stream": {
"url": "https://stream.tradier.com/v1/markets/events",
"sessionid": "c8638963-a6d4-4fb9-9bc6-e25fbd8c60c3"
}
}


But, what Dyalog tool would I use to open a Websocket connection ?
And what might be an example of getting the streaming data ... and appending it to an APL component file.

Do I simply use HTTPGet ... and LOOP to extract BLOBS of data from the stream ?
How do I get the streaming data without missing something from the stream as I Loop ?

Thanks in advance for the help on this.

Cheers,

//W
Woodley Butler
Automatonics, Inc.
"Find your head in the APL Cloud"
http://www.APLcloud.com
User avatar
woody
 
Posts: 114
Joined: Tue Dec 28, 2010 12:54 am
Location: Atlanta, Georgia USA

Re: Reading streaming data from external Websocket connectio

Postby Brian|Dyalog on Fri Aug 02, 2019 3:30 am

Hi Woodley,

I've taken a look at the online documentation for the streaming API for tradier.com. They are not using WebSocket protocol, but rather HTTP streaming which is achieved by leaving the socket open and using "chunked" transfer encoding.

HttpCommand supports chunked transfer encoding. However, it expects that the response from the server is a single response that's broken up into chunks of a more manageable size. That's not the case here; tradier.com is sending individual (stand alone) chunks in a stream.

To get streaming data from tradier.com, you'll need to write something that:
  1. Issues a POST request to get a streaming sessionid
  2. Issues a GET or POST request using that sessionid and whatever other parameters you need
  3. Sets up a listening loop in a separate thread to receive the chunks as the server pushes them out. Upon receipt of a chunk, your application code would process the data (probably in a new thread so as not to tie up the listening loop).
If there's no activity from the server for 15 minutes, the connection is closed, so you'll need to detect that condition and reestablish the connection.

The logic is pretty straightforward and HttpCommand has code that could be cloned and modified to help accomplish this.

I hope this helps!
/Brian
User avatar
Brian|Dyalog
 
Posts: 87
Joined: Thu Nov 26, 2009 4:02 pm
Location: West Henrietta, NY

Re: Reading streaming data from external Websocket connectio

Postby woody on Fri Aug 02, 2019 12:38 pm

Sincere thanks !

I'll give it a try.

Cheers,

//W
Woodley Butler
Automatonics, Inc.
"Find your head in the APL Cloud"
http://www.APLcloud.com
User avatar
woody
 
Posts: 114
Joined: Tue Dec 28, 2010 12:54 am
Location: Atlanta, Georgia USA


Return to APL Chat

Who is online

Users browsing this forum: No registered users and 1 guest