Reading streaming data from external Websocket connection

[woody]

Greetings.

I need a few pointers to get me going in the right direction regarding how to connect an APL session to read streaming stock market data from an external Websocket service.

My primary goal is to read and STORE the streaming stock market data ... perhaps in an APL component file. It's a lot of data ... and a single day of stock quotes for one Stock Symbol might be 100,000 records (100 bytes each).

Here's the HTTP steps to setup the streaming HTTP URL from the Stock Market data broker.
I know how to do this ... but I am not sure how to connect to the streaming data.

curl -X POST "https://api.tradier.com/v1/markets/events/session" \
-H 'Authorization: Bearer <TOKEN>' \
-H 'Accept: application/json'

Response (JSON)
{
"stream": {
"url": "https://stream.tradier.com/v1/markets/events",
"sessionid": "c8638963-a6d4-4fb9-9bc6-e25fbd8c60c3"
}
}


But, what Dyalog tool would I use to open a Websocket connection ?
And what might be an example of getting the streaming data ... and appending it to an APL component file.

Do I simply use HTTPGet ... and LOOP to extract BLOBS of data from the stream ?
How do I get the streaming data without missing something from the stream as I Loop ?

Thanks in advance for the help on this.

Cheers,

//W

[Brian|Dyalog]

Hi Woodley,

I've taken a look at the online documentation for the streaming API for tradier.com. They are not using WebSocket protocol, but rather HTTP streaming which is achieved by leaving the socket open and using "chunked" transfer encoding.

HttpCommand supports chunked transfer encoding. However, it expects that the response from the server is a single response that's broken up into chunks of a more manageable size. That's not the case here; tradier.com is sending individual (stand alone) chunks in a stream.

To get streaming data from tradier.com, you'll need to write something that:

1. Issues a POST request to get a streaming sessionid
2. Issues a GET or POST request using that sessionid and whatever other parameters you need
3. Sets up a listening loop in a separate thread to receive the chunks as the server pushes them out. Upon receipt of a chunk, your application code would process the data (probably in a new thread so as not to tie up the listening loop).

If there's no activity from the server for 15 minutes, the connection is closed, so you'll need to detect that condition and reestablish the connection.

The logic is pretty straightforward and HttpCommand has code that could be cloned and modified to help accomplish this.

I hope this helps!
/Brian

[woody]

Sincere thanks !

I'll give it a try.

Cheers,

//W