gzip
8 posts
• Page 1 of 1
gzip
I would like to compress and un-compress data using gzip (and perhaps other compression type as well). I am trying to compress text data to send over HTTP.
I took a look at I-Beam 219. The result of compression is a two-item vector. I don't really understand the purpose of the first item, but it is required to decompress the data. Is this a useful avenue to continue on? Can I encode/decode content as gzip for the purposes of HTTP using I-Beam 219? How? Or is the purpose of 219 just for internal use with I-beam 220?
Does MiServer support gzip compression? I looked through the code but only could find scaffolding for compression (interfaces), not actual code that did the compression.
I know that Conga (and MiServer) supports "deflate" compression. It also has a method for determining if the deflate compression library is loaded. It Conga is succesfully initialized and working, is there any situation where the deflate library would not be loaded?
Thanks for any help.
I took a look at I-Beam 219. The result of compression is a two-item vector. I don't really understand the purpose of the first item, but it is required to decompress the data. Is this a useful avenue to continue on? Can I encode/decode content as gzip for the purposes of HTTP using I-Beam 219? How? Or is the purpose of 219 just for internal use with I-beam 220?
Does MiServer support gzip compression? I looked through the code but only could find scaffolding for compression (interfaces), not actual code that did the compression.
I know that Conga (and MiServer) supports "deflate" compression. It also has a method for determining if the deflate compression library is loaded. It Conga is succesfully initialized and working, is there any situation where the deflate library would not be loaded?
Thanks for any help.
Last edited by paulmansour on Thu Dec 17, 2015 7:04 pm, edited 1 time in total.
- paulmansour
- Posts: 420
- Joined: Fri Oct 03, 2008 4:14 pm
Re: gzip
I have posted on the Wiki at http://aplwiki.com/sfZip some utilities for zipping and unzipping text using Syncfusion libraries in case this is helpful.
-
PGilbert - Posts: 436
- Joined: Sun Dec 13, 2009 8:46 pm
- Location: Montréal, Québec, Canada
Re: gzip
The executive overview: using DRC.flate.Deflate produces the same result as 2(219⌶) .. with a certain amount of adjusting from unsigned to signed integers:
Deflate uses zlib rather than gzip, so the call to (219⌶) has a left argument of 2. For gzip, you'd use a left argument of 3.
The first element of the result of using (219⌶) for compression contains information about the compression library that was used, and hints that allow the interpreter to perform a faster and more efficient decompression. The second element contains the compressed data.
When using (219⌶) to decompress data, you must either give both the hints and the compressed data, or give just the compressed data but explicitly specify the compression library that was used. In most cases, I suspect that users will decompress by specifying the decompression library to be used, and simple raw data. So the two options are:
[I've just spotted that the documentation for decompression for (219⌶) does not state that if X is non-zero, then Y must be a vector of sints; that will be fixed in the very near future!]
If Conga is up and running, the deflate library will be present. As for MiServer, I'll find out the answer to that ..
)load conga
DRC.Init''
sint←{⍵-256×⍵>127}
utf8←'UTF-8'∘⎕ucs
str←'empty←⍬ ⍝ include APL chars'
conga←DRC.flate.Deflate utf8 str
i219←256|2⊃2(219⌶)sint utf8 str
conga≡i219
1
)copy dfns cmpx
cmpx'DRC.flate.Deflate utf8 str' '256|2⊃2(219⌶)sint utf8 str'
DRC.flate.Deflate utf8 str → 2.8E¯4 | 0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
256|2⊃2(219⌶)sint utf8 str → 9.5E¯6 | -97% ⎕
Deflate uses zlib rather than gzip, so the call to (219⌶) has a left argument of 2. For gzip, you'd use a left argument of 3.
The first element of the result of using (219⌶) for compression contains information about the compression library that was used, and hints that allow the interpreter to perform a faster and more efficient decompression. The second element contains the compressed data.
When using (219⌶) to decompress data, you must either give both the hints and the compressed data, or give just the compressed data but explicitly specify the compression library that was used. In most cases, I suspect that users will decompress by specifying the decompression library to be used, and simple raw data. So the two options are:
comp←2(219⌶) sint utf8 str
str≡utf8 256| 0(219⌶) comp
1
⍝ or
str≡utf8 256| ¯2(219⌶) 2⊃comp
1
[I've just spotted that the documentation for decompression for (219⌶) does not state that if X is non-zero, then Y must be a vector of sints; that will be fixed in the very near future!]
If Conga is up and running, the deflate library will be present. As for MiServer, I'll find out the answer to that ..
-
AndyS|Dyalog - Posts: 257
- Joined: Tue May 12, 2009 6:06 pm
Re: gzip
Andy,
Thanks for this detailed response. I'm digesting it.
Paul
Thanks for this detailed response. I'm digesting it.
Paul
- paulmansour
- Posts: 420
- Joined: Fri Oct 03, 2008 4:14 pm
Re: gzip
Hi Paul,
MiServer 3.0 does support gzip compression. When I implemented deflate compression support for MiServer 2, I created a generic interface in anticipation of eventually supporting other content encoders like gzip. The code to implement gzip can be found at https://github.com/Dyalog/MiServer/blob/development/Extensions/gzip.dyalog.
I hope this helps!
/Brian
MiServer 3.0 does support gzip compression. When I implemented deflate compression support for MiServer 2, I created a generic interface in anticipation of eventually supporting other content encoders like gzip. The code to implement gzip can be found at https://github.com/Dyalog/MiServer/blob/development/Extensions/gzip.dyalog.
I hope this helps!
/Brian
-
Brian|Dyalog - Posts: 116
- Joined: Thu Nov 26, 2009 4:02 pm
- Location: West Henrietta, NY
Re: gzip
Brian,
That's great, it looks like exactly what I was looking for. Thanks!
Paul
That's great, it looks like exactly what I was looking for. Thanks!
Paul
- paulmansour
- Posts: 420
- Joined: Fri Oct 03, 2008 4:14 pm
Re: gzip
Ok, thanks everyone, I got gzip working well with I-Beam 219. Very easy!
I think what threw me initially was the documentation is not explicit that for decompression, if X, the left arg, is -1, -2 or -3, then Y, the right arg, is a simple vector. Granted the examples that follow in the doc show this, and I should have figured it out, but a mention of Y being simple, and the second element of the result of the compression, after "Otherwise, X ....' somewhere in the doc would have been helpful.
Thanks again all.
I think what threw me initially was the documentation is not explicit that for decompression, if X, the left arg, is -1, -2 or -3, then Y, the right arg, is a simple vector. Granted the examples that follow in the doc show this, and I should have figured it out, but a mention of Y being simple, and the second element of the result of the compression, after "Otherwise, X ....' somewhere in the doc would have been helpful.
Thanks again all.
- paulmansour
- Posts: 420
- Joined: Fri Oct 03, 2008 4:14 pm
Re: gzip
Hi Paul
Given it's pantomime season, I would like to reply with "Oh Yes It Does" !
Well, it does as of 5 minutes ago, when I updated help.dyalog.com, and docs.dyalog.com (there's an updated CHM and LangRef PDF). The text in all three cases now reads:
"R is a sint_vector, containing the output of applying the decompression library to the input data, Y.
If X is a scalar or a one item vector, and has the value 0, then Y must be a vector of two items which is the result of previously calling 219⌶ to compress a sint_vector.
Otherwise, X is a scalar or one or two element vector and Y must be a sint_vector."
Given it's pantomime season, I would like to reply with "Oh Yes It Does" !
Well, it does as of 5 minutes ago, when I updated help.dyalog.com, and docs.dyalog.com (there's an updated CHM and LangRef PDF). The text in all three cases now reads:
"R is a sint_vector, containing the output of applying the decompression library to the input data, Y.
If X is a scalar or a one item vector, and has the value 0, then Y must be a vector of two items which is the result of previously calling 219⌶ to compress a sint_vector.
Otherwise, X is a scalar or one or two element vector and Y must be a sint_vector."
-
AndyS|Dyalog - Posts: 257
- Joined: Tue May 12, 2009 6:06 pm
8 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 1 guest
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group