gzip

[paulmansour]

I would like to compress and un-compress data using gzip (and perhaps other compression type as well). I am trying to compress text data to send over HTTP.

I took a look at I-Beam 219. The result of compression is a two-item vector. I don't really understand the purpose of the first item, but it is required to decompress the data. Is this a useful avenue to continue on? Can I encode/decode content as gzip for the purposes of HTTP using I-Beam 219? How? Or is the purpose of 219 just for internal use with I-beam 220?

Does MiServer support gzip compression? I looked through the code but only could find scaffolding for compression (interfaces), not actual code that did the compression.

I know that Conga (and MiServer) supports "deflate" compression. It also has a method for determining if the deflate compression library is loaded. It Conga is succesfully initialized and working, is there any situation where the deflate library would not be loaded?


Thanks for any help.

[PGilbert]

I have posted on the Wiki at http://aplwiki.com/sfZip some utilities for zipping and unzipping text using Syncfusion libraries in case this is helpful.

[AndyS|Dyalog]

The executive overview: using DRC.flate.Deflate produces the same result as 2(219⌶) .. with a certain amount of adjusting from unsigned to signed integers:

      )load conga
      DRC.Init''

      sint←{⍵-256×⍵>127}
      utf8←'UTF-8'∘⎕ucs
      str←'empty←⍬ ⍝ include APL chars'
      conga←DRC.flate.Deflate utf8 str
      i219←256|2⊃2(219⌶)sint utf8 str
      conga≡i219
1

      )copy dfns cmpx
      cmpx'DRC.flate.Deflate utf8 str' '256|2⊃2(219⌶)sint utf8 str'
  DRC.flate.Deflate utf8 str → 2.8E¯4 |   0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
  256|2⊃2(219⌶)sint utf8 str → 9.5E¯6 | -97% ⎕

Deflate uses zlib rather than gzip, so the call to (219⌶) has a left argument of 2. For gzip, you'd use a left argument of 3.

The first element of the result of using (219⌶) for compression contains information about the compression library that was used, and hints that allow the interpreter to perform a faster and more efficient decompression. The second element contains the compressed data.

When using (219⌶) to decompress data, you must either give both the hints and the compressed data, or give just the compressed data but explicitly specify the compression library that was used. In most cases, I suspect that users will decompress by specifying the decompression library to be used, and simple raw data. So the two options are:

      comp←2(219⌶) sint utf8 str
      str≡utf8 256| 0(219⌶) comp
1
⍝ or
      str≡utf8 256| ¯2(219⌶) 2⊃comp
1

[I've just spotted that the documentation for decompression for (219⌶) does not state that if X is non-zero, then Y must be a vector of sints; that will be fixed in the very near future!]

If Conga is up and running, the deflate library will be present. As for MiServer, I'll find out the answer to that ..

[paulmansour]

Andy,

Thanks for this detailed response. I'm digesting it.

Paul

[Brian|Dyalog]

Hi Paul,

MiServer 3.0 does support gzip compression. When I implemented deflate compression support for MiServer 2, I created a generic interface in anticipation of eventually supporting other content encoders like gzip. The code to implement gzip can be found at https://github.com/Dyalog/MiServer/blob/development/Extensions/gzip.dyalog.

I hope this helps!

/Brian

[paulmansour]

Brian,

That's great, it looks like exactly what I was looking for. Thanks!

Paul

[paulmansour]

Ok, thanks everyone, I got gzip working well with I-Beam 219. Very easy!

I think what threw me initially was the documentation is not explicit that for decompression, if X, the left arg, is -1, -2 or -3, then Y, the right arg, is a simple vector. Granted the examples that follow in the doc show this, and I should have figured it out, but a mention of Y being simple, and the second element of the result of the compression, after "Otherwise, X ....' somewhere in the doc would have been helpful.

Thanks again all.

[AndyS|Dyalog]

Hi Paul

Given it's pantomime season, I would like to reply with "Oh Yes It Does" !

Well, it does as of 5 minutes ago, when I updated help.dyalog.com, and docs.dyalog.com (there's an updated CHM and LangRef PDF). The text in all three cases now reads:

"R is a sint_vector, containing the output of applying the decompression library to the input data, Y.

If X is a scalar or a one item vector, and has the value 0, then Y must be a vector of two items which is the result of previously calling 219⌶ to compress a sint_vector.

Otherwise, X is a scalar or one or two element vector and Y must be a sint_vector."