From owner-ppp-comp Thu Oct 21 16:52:18 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oq9nJ-00004ya; Thu, 21 Oct 93 16:51 PDT Sender: owner-ppp-comp X-Path: microsoft.com!tommyd From: Thomas Dimitri To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Thu, 21 Oct 93 16:45:10 TZ Message-ID: <9310212350.AA28635@netmail.microsoft.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk | From: Dave Carr | | Your reasons for the coherency byte seem valid for *some* types | of compression, and some types of lines. I don't think it is | a universal solution. Agreed. It is by no means a universal solution. From what I've seen, it is probably a good default solution. That's all. | In my case, I do not want to reset my tables. For one, it's | time consuming, and secondly, I've got a lot more model to lose. | Losing a 2K byte history is no big deal. One packet and your | history is back full. Not in my case. Ummm, well most string matches occur relatively close to the current string being looked at. That is, rarely do you find a good string match for something that occured 4000 bytes ago. Typically, there is at least a good match (maybe one byte or two bytes less) within the most recent 4000 bytes. That is why, inherently, almost all adaptive compression schemes build back up fast. Because the data stream is typically surrounded by data like it. Didn't you say even Gandalf's scheme could support a 4K or 8K buffer? | Of course, this coherency byte is based on some low error-rate | on the lines. At 10E-5, you'll be resetting pretty often. | V.42, MNP4, and other forms of error control are quite common. They will only be more apparent by the time this stuff gets out the door. Phone lines are moving closer and closer towards 100% digital. This 10E-5 error rate is typically NOT the case and therefore should not be the suggested default. | > Reasons: | > 1. The coherency byte is sufficient to detect out of sequence | > packets. | | Good, only on low error rate lines. Which is typically the case. | > 2. The coherency byte is sufficient to request a FLUSH of | > the history buffer. | | Another protocol to run when an error happens. How is this | better? And, start pitching all the frames while this reset | takes place? Or is each packet encoded separately? Each packet contains the coherency byte. Once a flush is requested the receiver can only accept packets which are not compressed or the packets sent after the history buffer was flushed. | > 3. The coherency byte takes less code, less memory, and | > is easier to implement than HDLC. | | True. | | > 3. The coherency byte takes less space (just one byte) | > and is faster to compute (on average) than the 16-bit | > CRC. | | If one byte is critical, use a better compression algorithm :-) It's not just one byte, it's several bytes because everyone appears to be using extra CRC checks. And it's extra code, and extra buffers to manage, and more bugs likely to occur and more interoperability problems, and typically slightly less throughput. All together, that's my point. The couple bytes saving is not alone worth it. --Thomas From owner-ppp-comp Thu Oct 21 17:20:16 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqAEb-00003Ba; Thu, 21 Oct 93 17:20 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Thu, 21 Oct 1993 17:19:54 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: Puddle Jumper, STAC, Gandalf, and MPPC" on Oct 21, 16:45, Thomas Dimitri writes:] > It's not just one byte, it's several bytes because everyone > appears to be using extra CRC checks. And it's extra code, > and extra buffers to manage, and more bugs likely to > occur and more interoperability problems, and typically > slightly less throughput. All together, that's my point. The couple > bytes saving is not alone worth it. --Thomas Not everyone will use CRC (HP PPC doesn't, STAC doesn't, Microsoft doesn't). There is room for lots of different techniques, and each compression algorithm is free to implement what works best for them. I have another compression technique that I am evaluating that doesn't look like it will need any error detection (sufficient redundancy), but it does take more CPU time. When you can disclose how your coherency byte works, we can do a good evaluations of it, and implement it when and where it makes sense. Assuming, of course, we don't have to pay royalties :-) -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Fri Oct 22 01:43:40 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqI5M-0000QYa; Fri, 22 Oct 93 01:43 PDT Sender: owner-ppp-comp X-Path: netcs.com!okorf From: Oliver Korfmacher To: ppp-comp@bungi.com Subject: Re: Comination Date: Fri, 22 Oct 93 9:42:33 MET Message-ID: <9310220842.AA15566@keks.netcs.com> References: <<9310211737.AA15185@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > > From: okorf@netcs.com, Oliver Korfmacher > > > Has anybody experience with TCP header compression over V.42bis or other > > compression techniques? > > > > We offer it as an unsupported feature, and it seems that V.42bis > > tends to become not very effective if many telnets with very small > > packets confuse its dictionary. > > > > Any comments? > > > > Oliver > > > What do you mean? > > Do you mean IP over PPP over v.42bis modems with VJ header compression? No. Sorry - what I actually mean is: TCP/telnet over VJ over IP over framed V.42bis - the type 20 in ppp-comp. The compressor compresses the VJ-packets, i.e. down to three (or smaller?) octets. This sometimes seems to confuse the decompressor, since it generates invalid VJ-packets, forcing the VJ to generate invalid IP-packets which are discarded by our upper layer. I havent yet found the time to investigate, but just wanted to ask if someone had a similar problem (may be our software bug, of course). Oliver From owner-ppp-comp Fri Oct 22 07:07:47 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqN9Q-0000Dla; Fri, 22 Oct 93 07:07 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 1993 09:53:24 -0400 (EDT) Message-ID: <9310221353.AA03042@donut.gandalf.ca> References: <<9310212350.AA28635@netmail.microsoft.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Ummm, well most string matches occur relatively close to the > current string being looked at. That is, rarely do you find a good > string match for something that occured 4000 bytes ago. Typically, > there is at least a good match (maybe one byte or two bytes less) > within the most recent 4000 bytes. That is why, inherently, > almost all adaptive compression schemes build back up > fast. Because the data stream is typically surrounded by data like it. I guess that's why PKZIP, INFO-ZIP, FREEZE, HPACK, ARJ, and just about every other archiver uses a 32K window :-) 2K and 4K are okay, but they are certainly not optimal. Let's face it, 2K and 4K buffers are used to limit the memory and increase the speed in your algorithms. You can't be using up 256K of the PC's memory for compression now can you? Back you numbers up. Your claim about matches is may be true, given the "locality-of-reference" inherently found in data. But the "longest match" has been shown to be within a few percent of the "best match". State a percentage lost by limiting the size of the history. (Sorry for the buzzwords Vernon). Your claim here is only true, as far as I can tell, for *static* encoding. And since that's all you seem to be doing, it's a *local* truth. But then again, no archiver would make it in the top 10 with only static encoding. Please don't apply it my algorithm. That's a false statement. Your claim also is bunk in the presence of multiple streams. At best, you must support multiple histories to even have a full packet of history from this session. > Didn't you say even Gandalf's scheme could support a 4K > or 8K buffer? 4K, 8K, 16K, 32K, and higher if you have time and memory. > V.42, MNP4, and other forms of error control are quite > common. You still haven't said whether the error control was turned of in your tests. If these protocols are enabled, they are doing the job of LAPB, AND TAKING THE SAME BANDWIDTH!!!! > They will only be more apparent by the time > this stuff gets out the door. Phone lines are moving closer > and closer towards 100% digital. This 10E-5 error rate is typically > NOT the case and therefore should not be the suggested default. And here we go again, every network is digital. I'll believe it when Bell Canada drops in our ISDN lines here at work maybe in December. > Which is typically the case. Typical maybe in the U.S. Quote some numbers outside the U.S. > It's not just one byte, it's several bytes because everyone > appears to be using extra CRC checks. I guess I'm the oddball then, because there is none in our code. Maybe I should add it, maybe I shouldn't. I wouldn't be adding it to save a byte per packet though. > And it's extra code, > and extra buffers to manage, and more bugs likely to > occur and more interoperability problems, and typically > slightly less throughput. All together, that's my point. The couple > bytes saving is not alone worth it. --Thomas If 8K of code is too much for a software giant to handle, were all in trouble. It always seems that those without working LAPB code keep telling the same story. Let's face it, come up with all the schemes you want, LAPB does the job, does it well, and is at worst a byte per packet more than any scheme you can come up with. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Oct 22 08:52:33 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqOm3-00007aa; Fri, 22 Oct 93 08:51 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Comination Date: Fri, 22 Oct 93 09:50:29 -0600 Message-ID: <9310221550.AA22637@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: okorf@netcs.com, Oliver Korfmacher > > > > > Has anybody experience with TCP header compression over V.42bis or other > > > compression techniques? > No. Sorry - what I actually mean is: TCP/telnet over VJ over IP over > framed V.42bis - the type 20 in ppp-comp. > The compressor compresses the VJ-packets, i.e. down to three (or smaller?) > octets. This sometimes seems to confuse the decompressor, since it generates > invalid VJ-packets, forcing the VJ to generate invalid IP-packets > which are discarded by our upper layer. > > I havent yet found the time to investigate, but just wanted to ask if > someone had a similar problem (may be our software bug, of course). I don't see how it could be anything other than your software bug, whether in your v.42bis or VJ-compression code. v.42bis and VJ compression are both "lossless". Simply composing them does not produce a function that changes any bits. If the v.42bis protocol regularly messed up 3 byte sequences, how would the zillions of people with v.32bis+v.42+v.42bis modems feel? Vernon Schryver, vjs@sgi.com From owner-ppp-comp Fri Oct 22 09:00:45 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqOth-00007ra; Fri, 22 Oct 93 08:59 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: LZW compression Date: Fri, 22 Oct 1993 11:56:21 -0400 (EDT) Message-ID: <9310221556.AA03073@donut.gandalf.ca> References: <<9310212117.AA16612@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > One claim that stands out is that LZW requires ~500KByte of memory. In can vary from 32K to 512 K depending on the codesize and the data structures (trie, hash). There is even a 17-bit version, which limits expansion to 6% worse case. > I do not understand this, given that for many years netnews was > transfered to and from PDP-11's, using 12-bit compression. That's either an encoder, or a decoder. We'll need both. What was the code size, 12-bits. It may have used a smaller hash table as well. > Predictor's 64KB table is not small when viewed on a PDP-11. For the memory used, Predictor does not use memory effectively. It was built for speed. > > A statement in the archives is that LZW seemed to compress better than > Predictor but that gross file transfer times were higher. I could not > tell enough about the implementation in which this was observed to know > if it might not have been caused by packets being so compressed that > they were delayed, causing the unspecified file transfer algorithm to > timeout and then ask for a retransmission. Or if the delays were > caused by the increased CPU load of LZW on the machines involved. The > phrased in the archives "not all implementations are equal" is true in > many different way. LZW is fast. But it still has to search hash tables. Unfortunately, it must search the whole hash chain, so as to avoid creating duplicate entries in the compression table. To say it is slower is relative. The fastest chip(s) on the market is RSIC's (James Storers company). It achieves an *output* rate of 300 Mb/s. Yes, megabits! It's an LZW variant called AP, and the concept is in his book (stochastic pipe). Even predictor would have trouble keeping up with dedicated hardware. > Since the compression ID's are up to 20, we ought to add one for > straight BSD-UNIX `compress`, with negotiatable 12 or 16-bit. I don't > know which strategy for handling expansion. The 17-bit version limits expansion to 6% worse case, with no expansion checking involved. The first 64K entries are all possible 2 character strings. Of course, you don't need to actually store them, so the RAM requirements are the same as 16-bit. > Described as "UNIX > compress" to frighten off the patent leeches and allow pure software > implemtations like Morningstar, Novell, Silicon Graphics, BSDI, Sun, > IBM, and HP to use it without fear. (several of those are fearlessly > shipping BSD (!NOT USL!) derived compress commands.) With sample code > grabbed from your favorite BSD archive site. Perhaps I'll try to contact the patent holder of the 17-bit method. Maybe, he'll donate the code to a good cause. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Oct 22 09:33:55 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqPPn-0000ZYa; Fri, 22 Oct 93 09:32 PDT Sender: owner-ppp-comp X-Path: sgw.xyplex.com!sgw From: sgw@sgw.xyplex.com (Scott Wasson) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 93 12:29:47 EDT Message-ID: <9310221629.AA14967@eng.xyplex.com> References: <<9310221353.AA03042@donut.gandalf.ca>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > And it's extra code, > > and extra buffers to manage, and more bugs likely to > > occur and more interoperability problems, and typically > > slightly less throughput. All together, that's my point. The couple > > bytes saving is not alone worth it. --Thomas > > If 8K of code is too much for a software giant to handle, were all in > trouble. It always seems that those without working LAPB code keep > telling the same story. > > Let's face it, come up with all the schemes you want, LAPB does the > job, does it well, and is at worst a byte per packet more than any > scheme you can come up with. > -- > Dave Carr | dcarr@gandalf.ca | It's what you learn, > Principal Designer | TEL (613) 723-6500 | after you know it all, > Gandalf Data Limited | FAX (613) 226-1717 | that counts. > You may come up with any argument you want, but there are those who believe that doing guaranteed-delivery at the link layer, and doing it again at the network layer, is a bad idea. This is one of those religious wars that will not be resolved in our lifetimes. So I'd really rather see us move away from discussing the benefits/detriments of LAPB. The only way around this problem is to: 1. Make LAPB optional for those who want it and those who don't. 2. For link-layer features that require synchronization (such as compression), talk about two solutions: one involving LAPB, and one involving some means of detecting loss of dictionary synchronization. Whether that's done by computing the CRC on the original data, or by embedding a index count, I don't care. Both modes of operation must be supported to appease both religious camps, otherwise this mailing list will still be discussing compression encapsulation well into the next century. We have before us two equally-implementable technologies. Let's just spec both and be done with this, Ok? +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+ | Scott G. Wasson sgwasson@eng.xyplex.com | | Xyplex, Internetworking & Media Voice: (508) 952-4746 | | 295 Foster St. Littleton, MA 01460 Fax: (508) 952-4702 | +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+ From owner-ppp-comp Fri Oct 22 10:57:04 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqQjC-00006Ua; Fri, 22 Oct 93 10:56 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 1993 10:56:25 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: Puddle Jumper, STAC, Gandalf, and MPPC" on Oct 22, 12:29, Scott Wasson writes:] > The only way around this problem is to: > > 1. Make LAPB optional for those who want it and those who don't. Done. It is completely optional. > 2. For link-layer features that require synchronization (such as > compression), talk about two solutions: one involving LAPB, and one > involving some means of detecting loss of dictionary synchronization. > Whether that's done by computing the CRC on the original data, or by > embedding a index count, I don't care. Done. If you notice (or care) about a decompression failure, you can blip the CCP, and restart. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Fri Oct 22 12:58:40 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqSd0-0000Ofa; Fri, 22 Oct 93 12:58 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 93 13:57:38 -0600 Message-ID: <9310221957.AA24233@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Done. If you notice (or care) about a decompression failure, you can > blip the CCP, and restart. > > Dave Rand That raises a question of what should happen to network packets while CCP is neither dead nor "Opened". Can normal network packets (not CCP) be sent? Should they be handled or discarded if received? This concerns Bill Simpson's proposed strategy for omitting the 0xfd for Puddle Jumper. Since the compression protocol used before a "blip" may differ from the one after, this policy must be decided for all compression protocols. Since the CCP draft right now says nothing on the issue, it seems to me that you may send network packets and must accept them concurrently with CCP negotiations. Personally, I find the idea of omitting the protocol byte and the consequent strangeness during CCP negotiations neat but over the edge. It's too much of a kludge for saving a single byte. I'd instead rather see the real protocol byte compressed along with it's data. You would expect a good compression protocol to compress to nearly nothing not only the higher layer header (even if VJ-compressed) but also any fixed premable such as 0x21. Regardless of the choice we make, it must be written into the CCP RFC. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Fri Oct 22 13:03:09 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqShO-0000QDa; Fri, 22 Oct 93 13:02 PDT Sender: owner-ppp-comp X-Path: microsoft.com!tommyd From: Thomas Dimitri To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 93 12:56:40 TZ Message-ID: <9310222002.AB07599@netmail.microsoft.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk | From: Scott Wasson | | > > And it's extra code, | > > and extra buffers to manage, and more bugs likely to | > > occur and more interoperability problems, and typically | > > slightly less throughput. All together, that's my point. The couple | > > bytes saving is not alone worth it. --Thomas | > | > If 8K of code is too much for a software giant to handle, were all in | > trouble. It always seems that those without working LAPB code keep | > telling the same story. | > | > Let's face it, come up with all the schemes you want, LAPB does the | > job, does it well, and is at worst a byte per packet more than any | > scheme you can come up with. | > -- | > Dave Carr | dcarr@gandalf.ca | It's what you learn, | > Principal Designer | TEL (613) 723-6500 | after you know it all, | > Gandalf Data Limited | FAX (613) 226-1717 | that counts. | > | | You may come up with any argument you want, but there are those who believe | that doing guaranteed-delivery at the link layer, and doing it again | at the network layer, is a bad idea. This is one of those religious | wars that will not be resolved in our lifetimes. So I'd really rather | see us move away from discussing the benefits/detriments of LAPB. | | The only way around this problem is to: | | 1. Make LAPB optional for those who want it and those who don't. | | 2. For link-layer features that require synchronization (such as | compression), talk about two solutions: one involving LAPB, and one | involving some means of detecting loss of dictionary synchronization. | Whether that's done by computing the CRC on the original data, or by | embedding a index count, I don't care. | | Both modes of operation must be supported to appease both religious | camps, otherwise this mailing list will still be discussing | compression encapsulation well into the next century. We have before | us two equally-implementable technologies. Let's just spec both and | be done with this, Ok? We are in violent agreement!!! I was just trying to clarify this point further and suggest that all implementations should try to at minimum, implement the "index count" because it is easy to implement. But, hey, if people are gonna fuss this much about adding a souped up counter (because it is too much work or whatever the reason), consider the issue dropped. Now, what I REALLY want is a CRC at the PPP async level, one at the compression level, one in TCP/IP, one in the file system, and one in the app. I mean, it's the 90's. You can't just go around trusting every layer you interface with :-) From owner-ppp-comp Fri Oct 22 13:22:43 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqT0H-0000D7a; Fri, 22 Oct 93 13:22 PDT Sender: owner-ppp-comp X-Path: Novell.COM!Dave_Rand From: Dave_Rand@Novell.COM (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 1993 13:21:46 PDT Message-ID: <9310222021.AA15720@va.SJF.Novell.COM> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk On Oct 22, 13:57, Vernon Schryver wrote: } } Since the CCP draft right now says nothing on the issue, it seems to me } that you may send network packets and must accept them concurrently } with CCP negotiations. Agreed. } Regardless of the choice we make, it must be written into the CCP RFC. I'll add it now. -- From owner-ppp-comp Fri Oct 22 13:58:13 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqTXz-0000cla; Fri, 22 Oct 93 13:57 PDT Sender: owner-ppp-comp X-Path: sgw.xyplex.com!sgw From: sgw@sgw.xyplex.com (Scott Wasson) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 93 16:55:26 EDT Message-ID: <9310222055.AA15435@eng.xyplex.com> References: <<9310221957.AA24233@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk According to Vernon Schryver: > > > > Done. If you notice (or care) about a decompression failure, you can > > blip the CCP, and restart. > > > > Dave Rand > > > That raises a question of what should happen to network packets while > CCP is neither dead nor "Opened". Can normal network packets (not CCP) > be sent? Should they be handled or discarded if received? > Any time CCP is not "Opened", it's dead to compressed data transfer. I don't see the ambiguity. I also don't see any reason why other network packets can't be sent even after CCP has Opened. They should simply be handled by their respective NCPs. Here's a no-no: you have to be really careful if you send bridged packets to the Bridged PPP protocol ID (0x0031) AND compressed bridged packets to the compression protocol ID (0x00??), with the 0x0031 embedded, compressed. We should state in big bold letters that once CCP Opens, all received traffic must pass through the decompression data path, regardless of whether it's actually compressed. This is to ensure packet order. If compressed bridge packets are followed by uncompressed bridge packets, "thou must not" allow the uncompressed packets to bypass all that decompression that's going on. Those packets have to sit around and wait for the compressed bridged packets to be forwarded. In general, the data link layer should never reorder packets, so I think this issue applies in general, not just to bridging. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+ | Scott G. Wasson sgwasson@eng.xyplex.com | | Xyplex, Internetworking & Media Voice: (508) 952-4746 | | 295 Foster St. Littleton, MA 01460 Fax: (508) 952-4702 | +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+ From owner-ppp-comp Fri Oct 22 15:53:21 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oqVLo-0000bPa; Fri, 22 Oct 93 15:52 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Puddle Jumper, STAC, Gandalf, and MPPC Date: Fri, 22 Oct 93 16:52:11 -0600 Message-ID: <9310222252.AA25248@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >From: Scott G. Wasson sgwasson@eng.xyplex.com | > > That raises a question of what should happen to network packets while > > CCP is neither dead nor "Opened". Can normal network packets (not CCP) > > be sent? Should they be handled or discarded if received? > > Any time CCP is not "Opened", it's dead to compressed data transfer. > I don't see the ambiguity. I also don't see any reason why other > network packets can't be sent even after CCP has Opened. They should simply > be handled by their respective NCPs. Please read Bill's recent notes about Puddle Jumper. > Here's a no-no: you have to be really careful if you send bridged > packets to the Bridged PPP protocol ID (0x0031) AND compressed bridged > packets to the compression protocol ID (0x00??), with the 0x0031 embedded, > compressed. We should state in big bold letters that once CCP > Opens, all received traffic must pass through the decompression data > path, regardless of whether it's actually compressed. This is to > ensure packet order. That is wrong. Unless you do as Bill Simpson wanted, and shut down all network traffic for the duration of CCP negotiation, you have no way of knowing whether or not any network (bridge or other flavor) PPP packet had anything to do with compression when it left the transmitter. As the CCP draft now reads, and as Dave says he will make explicit, you cannot reliably run network (i.e. non-CCP) packets through your decompressor. All the decompressor can do is pass all non-CCP packets through without looking at them. There is an unfixable race or ambiguity between exactly when the sender of a packet thinks compression is on and when the receiver agrees, or when the receiver agrees and when the sender hears that agreement in a Configure-ACK. Bill proposed to fix the race by synchronizing the start and end of CCP with turning off normal traffic and by discarding any packet that leaked. (Yes, dropping a packet is Not Good for any higher layer protocol.) > If compressed bridge packets are followed by uncompressed bridge > packets, "thou must not" allow the uncompressed packets to bypass all > that decompression that's going on. Those packets have to sit around > and wait for the compressed bridged packets to be forwarded. Maybe so, although I don't see why. Even if you impose that restriction, you still have that unfixable race. > In general, the data link layer should never reorder packets, so I > think this issue applies in general, not just to bridging. No, that does not apply in general, but only to a large number of broken protocols and/or broken implementations. There are too many link layers that have no way to prevent reordering packets. Consider "dual-rail" or dual-MAC FDDI. Consider what happens in any bridged network during transients. It is easy to construct scenarios where packets are reordered by routing transients caused by spanning tree responses to coming and going links. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Sun Oct 24 05:08:43 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0or4ET-00006ka; Sun, 24 Oct 93 05:07 PDT Sender: owner-ppp-comp X-Path: um.cc.umich.edu!bill.simpson From: "William Allen Simpson" To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Fri, 22 Oct 93 15:08:05 EDT Message-ID: <1578.bill.simpson@um.cc.umich.edu> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk (I sent this before, but it didn't echo. Here it is slightly updated.) There are a number of problems with the latest CCP draft, all of which were pointed out in earlier drafts. Some which I have not indicated in writing -- but did tell Dave on the phone 2 weeks ago -- have not been fixed either. > CCP Configuration Options allow negotiation of compression algorithms > and their parameters. CCP uses the same Configuration Option format > defined for LCP [1], with a separate set of Options. > Unfortunately, this is not true as written. The format of the Type, Length, Value for PPP Configuration Options is well established. > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > | Type | Length | Comp. Types | > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > | Length | options... > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > The Length MUST be one octet. There is no need for two lengths, as the length of the values part is easily determined by the overall length. Each Algorithm will have its own option number, followed by its values. There will be no mucking about assigning "special" numbers to "common" algorithms. Assign them in sequence, as drafts are published. Here is the list I have used, based on actual submitted drafts: 0 OUI 1 Predictor 1 2 Predictor 2 3 bogus "options" type, reassign 4 Puddle Jumper 5 Stacker LZS 6 Gandalf FZA 7 Microsoft Citing the Amsterdam Minutes: There will be one option number per compression algorithm, with a special one for proprietary algorithms. The general format of these is: COMPRESSION CONTROL PROTOCOL Option Length Parameters as required by the algorithm. > A summary of the Proprietary Compression Algorithm Configuration > Option format is shown below. The fields are transmitted from left > to right. > > 0 1 2 3 > 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > | Type | Length | OUI MS octet | > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > | OUI remaining octects | Length | options... > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > Type > > 2 > The Type for this will be 0. There will be one length field, followed by the OUI field, which is followed by its values. There is no need for two lengths, since the OUI is fixed in length. > B. Common compression algorithm definitions > This section will be removed and put in its own draft. CCP provides the means to negotiate, not any particular compression technique. Arguing about compression algorithms has slowed the process. KISS. We are not required to implement Predictor as part of this standard. Therefore, it MUST NOT be included in the proposed standard. Citing the Amsterdam Minutes: Although we wanted to do this, PPP Working Group does not recommend one of them for universal implementation. The reason is that we cannot, under IETF rules and marketplace sense, require everyone to license code or silicon from a single vendor, and the one unencumbered algorithm we have found has significant (64K per link) memory requirements. We therefore only provide the means to negotiate them. Finally, there are several bugs in the Predictor code, which crashes. I can only recommend that the separate Predictor "Informational" not contain buggy code. Bill.Simpson@um.cc.umich.edu From owner-ppp-comp Mon Oct 25 08:41:53 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orU0e-00008Fa; Mon, 25 Oct 93 08:39 PDT Sender: owner-ppp-comp X-Path: opal.acc.com!art From: art@opal.acc.com (Art Berggreen) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 93 08:38:19 PDT Message-ID: <9310251538.AA21874@opal.acc.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >> A summary of the Proprietary Compression Algorithm Configuration >> Option format is shown below. The fields are transmitted from left >> to right. >> >> 0 1 2 3 >> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 >> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ >> | Type | Length | OUI MS octet | >> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ >> | OUI remaining octects | Length | options... >> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ >> >> Type >> >> 2 >> >The Type for this will be 0. There will be one length field, followed >by the OUI field, which is followed by its values. There is no need for >two lengths, since the OUI is fixed in length. Hmm, if we are going to use IEEE OUIs, they only identify a vendor, not a specific protocol. If a single vendor supports multiple private compression algorithms, how are they identified? Do we need at least another byte for OUI specific codepoints? Also, I'm assuming that following sort of preference negotiation is supported (in order of preference): Private-A -> Public-B -> Private-C -> Public-D Art From owner-ppp-comp Mon Oct 25 09:16:23 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orUaS-00008qa; Mon, 25 Oct 93 09:16 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 1993 09:15:58 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: New CCP draft (rev 01)" on Oct 25, 8:38, Art Berggreen writes:] > > Hmm, if we are going to use IEEE OUIs, they only identify a vendor, not > a specific protocol. If a single vendor supports multiple private > compression algorithms, how are they identified? Do we need at least > another byte for OUI specific codepoints? Quoted from the draft: Multiple vendor-specific proprietary compression types may be implemented by the option field, which may contain algorithm selection information, negotiated options such as dictionary size, or any other information required. If the information in the option field is unrecognized, a Configure-Reject MUST be sent. If the information in the option field is recognized, but certain value(s) are unavailable, a Configure-Nak MAY be sent with the appropriate values modified. Any unrecognized proprietary compression configure request MUST have a Configure-Reject sent back. So it is up to each vendor to offer a way to negotiate various compression algorithms, if they wish to do so. Some will not wish to do so. Some may want lots of different algorithms - since we don't specify what goes in the option area, each vendor is free to put as much or as little configuration as they desire. > > Also, I'm assuming that following sort of preference negotiation is > supported (in order of preference): > > Private-A -> Public-B -> Private-C -> Public-D > As indicated in the draft: This Configuration Option provides a way to negotiate the use of a proprietary compression protocol. By default, such compression is not enabled. Since the first matching compression will be used, it is recommended that any known OUI compression options be transmitted first, before the common options are used. Before accepting this option, the implementation must verify that the Organizationally Unique Identifier identifies a proprietary algorithm that the implementation can decompress, and that any vendor specific negotiation options are fully understood. If no OUIs are supported by an implementation, a Configure-Reject may be sent back for any type 2 Compression Type Negotiation Option. But in reality, you should check for an "appropriate" compression technique, not default to your private technique, on a per link basis. You would not want to run (for example) an algorithm that can only source data at 1K/sec on a 56K link. So, after determining the link speed, you would offer the "best" algorithm for the link speed first, then the other algorithms that you support (unless, of course, your proprietary algorithm had speed- sensitive compression or some such). One thing that I tried to do early on was to devise a way for a PPP link to figure out the "best" compression to use (like PKARC did - unlike ZIP/PKZIP). PKARC actually ran several different algorithms on the data, and picked the one with the lowest number of output bytes. That is *very* CPU intensive, and was replaced in PKZIP with an algorithm that looks at the size of the file, and the contents of the first few blocks. We don't really have the luxury of knowing the size of the file, nor the contents in advance. Now, I just look at the speed of the link, and pick the "best" algorithm. As Jim O'Donnell pointed out - in his and my implementations, we have to contend with multiple CPU speeds. In the case of the X86, variations in CPU type can have a *radical* effect on the choice of algorithm. Due to other hardware concerns (external caches, mostly), pattern-match compressors may perform *much* better on certain platforms. I don't have any good way of predicting this, so I'll have to run each compressor prior to the link coming up to determine its speed on the current platform. If anyone has a better way of determining the "best" compressor list, I would sure like to know about it. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Mon Oct 25 09:59:13 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orVFq-00005xa; Mon, 25 Oct 93 09:58 PDT Sender: owner-ppp-comp X-Path: opal.acc.com!art From: art@opal.acc.com (Art Berggreen) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 93 09:58:22 PDT Message-ID: <9310251658.AA21939@opal.acc.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > >[In the message entitled "Re: New CCP draft (rev 01)" on Oct 25, 8:38, Art Berggreen writes:] >So it is up to each vendor to offer a way to negotiate various compression >algorithms, if they wish to do so. Some will not wish to do so. Some may want >lots of different algorithms - since we don't specify what goes in the >option area, each vendor is free to put as much or as little configuration >as they desire. Things left up to implementors to figure out generally just cause complications. I would welcome having how the OUI specific codepoints are encoded being standardized. But then, I just view all of my compression algorithms as available choices. Some have been documented as "standard" and some are "private". I already prefer not to negotiate them in totally different ways. If one of my "private" protocols gets standarized, how it gets negotiated suddenly changes. >> >> Also, I'm assuming that following sort of preference negotiation is >> supported (in order of preference): >> >> Private-A -> Public-B -> Private-C -> Public-D > >But in reality, you should check for an "appropriate" compression technique, >not default to your private technique, on a per link basis. You would not >want to run (for example) an algorithm that can only source data at 1K/sec >on a 56K link. So, after determining the link speed, you would offer >the "best" algorithm for the link speed first, then the other algorithms that >you support (unless, of course, your proprietary algorithm had speed- >sensitive compression or some such). And after carefull and magical internal incantations, my box has decided on the following preferences: Private-A -> Public-B -> Private-C -> Public-D >If anyone has a better way of determining the "best" compressor list, I would >sure like to know about it. As you pointed out, this is a difficult problem, full of engineering tradeoffs, with no single answer. Art From owner-ppp-comp Mon Oct 25 10:58:13 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orWAs-00008ja; Mon, 25 Oct 93 10:57 PDT Sender: owner-ppp-comp X-Path: Novell.COM!Dave_Rand From: Dave_Rand@Novell.COM (Dave Rand) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 1993 10:56:59 PDT Message-ID: <9310251756.AA10847@va.SJF.Novell.COM> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk On Oct 25, 9:58, Art Berggreen wrote: } Things left up to implementors to figure out generally just cause complications. } I would welcome having how the OUI specific codepoints are encoded being } standardized. But then, I just view all of my compression algorithms } as available choices. Some have been documented as "standard" and some are } "private". I already prefer not to negotiate them in totally different } ways. If one of my "private" protocols gets standarized, how it gets } negotiated suddenly changes. Sounds OK to me... want to use the first byte (or two?) of the OUI negotiation field, and mandate a min length of one/two? Of course, the rest of the field will change on a per compression algorithm basis. In this case, let's say you want to do a custom variation of a WaffleIron compressor, SteamBed compressor, and TrashMasher compressor... you would offer OUI-1, OUI-2, OUI-3 with the appropriate options, of course. Sound ok? If so, I'll add that to the next round. I would suggest a single byte of algorithm choice, with the note that a value of 255 means to look at the options in a vendor-unique way to determine the compression method. -- From owner-ppp-comp Mon Oct 25 11:15:00 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orWR4-0000F4a; Mon, 25 Oct 93 11:14 PDT Sender: owner-ppp-comp X-Path: opal.acc.com!art From: art@opal.acc.com (Art Berggreen) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 93 11:13:56 PDT Message-ID: <9310251813.AA21964@opal.acc.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > >Sounds OK to me... want to use the first byte (or two?) of the OUI negotiation >field, and mandate a min length of one/two? Of course, the rest of the >field will change on a per compression algorithm basis. In this case, >let's say you want to do a custom variation of a WaffleIron compressor, >SteamBed compressor, and TrashMasher compressor... you would offer > >OUI-1, OUI-2, OUI-3 > >with the appropriate options, of course. > >Sound ok? If so, I'll add that to the next round. I would suggest a single >byte of algorithm choice, with the note that a value of 255 means to look >at the options in a vendor-unique way to determine the compression method. Sounds acceptable to me. One byte ought to be sufficient (famous last words). Art From owner-ppp-comp Mon Oct 25 11:31:00 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orWgO-0000Esa; Mon, 25 Oct 93 11:30 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 1993 11:29:45 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: New CCP draft (rev 01)" on Oct 25, 9:58, Art Berggreen writes:] > And after carefull and magical internal incantations, my box has decided > on the following preferences: > Private-A -> Public-B -> Private-C -> Public-D I missed your question on first reading. Yes, you may offer OUI and standard compression intermixed. As well, if you implement a "standard" compression engine in a proprietary way, you are not oblidged to offer the "standard" negotiation for it, if you don't want to. We can *always* negotiate to no compression. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Mon Oct 25 12:57:09 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orY24-0000RHa; Mon, 25 Oct 93 12:56 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Mon, 25 Oct 93 13:55:47 -0600 Message-ID: <9310251955.AA21286@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Art Berggreen writes: > >So it is up to each vendor to offer a way to negotiate various compression > >algorithms, if they wish to do so. Some will not wish to do so. Some may want > >lots of different algorithms - since we don't specify what goes in the > >option area, each vendor is free to put as much or as little configuration > >as they desire. > > Things left up to implementors to figure out generally just cause > complications. True. That's the biggest problem with PPP in general. > I would welcome having how the OUI specific codepoints are encoded being > standardized. It is impossible to have it both ways, to both let consenting boxes do abitrarily strange things in the future and to standardize what they do now. The choice of algorithm strikes me as one of the more trivial parameters that would need to be negotiated under an OUI. The choice between run-length and LZ is not significantly different than the choice among LZ dictionary sizes, or LZ variations. > But then, I just view all of my compression algorithms > as available choices. Some have been documented as "standard" and some are > "private". I already prefer not to negotiate them in totally different > ways. Don't we all! But you need not negotiating anything by OUI. Simply get numbers assigned to whatever you want now. > If one of my "private" protocols gets standarized, how it gets > negotiated suddenly changes. That is not exactly true. You would end up have to always negotiate your proprietary algorithm both ways, to always Config-Req your algorithm first by OUI and then by standard number. Old versions of your box would answer the OUI. New, incompatible versions would COnfig-REJ the OUI and accept the standard. Transitional versions could handle both and so would Config-ACK the OUI. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Mon Oct 25 19:40:47 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oreKi-0000D7a; Mon, 25 Oct 93 19:40 PDT Sender: owner-ppp-comp X-Path: mcimail.com!STAC/STAC/BobL%Stac_Electronics From: BobL To: ppp-comp Subject: Royalty Free Stacker LZS Compression For Date: Mon, 25 Oct 93 22:50 GMT Message-ID: <11931025225011/MAILMAN000NA4EM@mcimail.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk ---------------------------------------- TO: IETF PPP, Data Compression Working Group Stac would like the IETF PPP to consider a prospective standards proposal in which Stac would provide a ROYALTY FREE limited license for the Stacker LZS data compression format for specified Internetworking applications. I am still working on the details, but I wanted to forward this information to this group as soon as possible. I believe that we want to find a good default compression algorithm for PPP. Currently, the Predictor algorithm is under consideration. After experimenting with the Predictor algorithm for a short while, I conclude that this would be a poor choice. The compression ratio that it produces is not nearly as high as LZS. As the packet size (or block size) decreases, it gets even worse. The Stacker LZS algorithm is much better. Up to two times better compression than Predictor Software speed up to 1 Mbyte/s compression Software speed up to 2 Mbytes/s decompression Hardware speed up to 12 Mbytes/s compression Hardware speed up to 16 Mbytes/s decompression Available in hardware or software, cross compatible It costs the same as Predictor, $0 Algorithm is already a pending ANSI standard Many shipping WAN bridges/routers that support data compression are already using Stacker LZS Novell will be incorporating Stacker LZS in future versions of Netware, as well as standardizing on LZS for all their compression needs in the future Here is a preliminary description of the proposed limited license: License for LZS compression format - ROYALTY FREE The LZS compression format will be fully described in a standard. A royalty free limited license can be obtained from Stac Electronics which allows the licensee to implement the LZS compression format. A C source code version will be provided. License for optimized assembly language software from Stac - $10K Stac has developed hand coded assembly language versions of the LZS algorithm for the 80x86, 680x0, and the i960. For a one-time license fee (royalty free), Stac will provide a highly optimized library, and product support. Hardware chips - $20 to $65 depending upon quantities and speed Stac has an entire family of data compression chips with various speeds and features. Stac has been shipping compression chips since 1989. Here is the current list of chips that are available: 9703 - 1.0 Mbyte/s 9704 - 1.8 Mbytes/s 9705 - 2.5 Mbytes/s, supports over 1600 histories 9706 - 2.5 Mbytes/s, supports over 1600 histories 9730 - 12 Mbytes/s, available Q194. I will be attending the IETF meeting in Houston. I will present this proposal in more detail at this meeting. I will also share some of the results of my experiments with compression ratio, speed, and effects of smaller packets. I will be performing these experiments between now and then. Stac's patented LZS format has been marketed by Stac for several years now. Even though issues of patents and trade secrets seem to hover over every different compression format, at least Stacker LZS has a track record. The claim that a data compression format is patent free should not be the reason for choosing it to become a standard. Please forward your comments to me. I would like to know if Stac's proposed LZS licensing will meet the needs for this working group, and the needs of the manufacturers that we all represent. This proposal is in response to Microsoft's royalty free offer. By the way, Stac has an LZS evaluation program available that runs on a PC. It may be used to evaluate the speed and compression ratio of the LZS compression algorithm. If you would like a copy of this program mailed to you, please contact me. Otherwise, I will bring copies with me to Houston, so you can pick one up from me there. Robert Lutz Stac Electronics Phone (619) 431-7474, FAX (619) 431-0947 Email:Stac/Stac/BobL%Stac_Electronics@mcimail.com From owner-ppp-comp Mon Oct 25 21:44:02 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orgFg-0000PZa; Mon, 25 Oct 93 21:43 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Mon, 25 Oct 93 22:42:32 -0600 Message-ID: <9310260442.AA24313@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: Robert Lutz > > Stac would like the IETF PPP to consider a prospective standards proposal in > which Stac would provide a ROYALTY FREE limited license for the Stacker LZS > data compression format for specified Internetworking applications.... That is welcome news, and STAC should be congratulated. Unfortunately, I don't think this does the trick for two important classes of PPP implementations: 1. the public domain PPP implementations must not require anyone to get permission from anyone. A lisense that has no explicit fee is not free if you must spend time and money to negotiate it. 2. As all of us know who work for commercial outfits and who have gone around merry-go-rounds like getting the almost-free BSD lisenses, the cost to negotiate such a free lisense is a matter of at least several thousand dollars for any commerical organization, if you assume your own lawyers will not insist on negotiating any modifications of STAC's lisense. If your lawyers need to prove their value to the company this quarter, if there is something they have good reason to dislike in the STAC wording, negotiating the lisense could be very expensive. Even in the best case, where the STAC lisense is liked, I bet I would spend almost as much time encouraging and nagging lawyers as writing an optimized assembly language implementation of any plausible compression algorithm. (There can't be much code if it is to be fast, and so it can't take much time to write.) STAC seems to think the job is worth at least $10,000, which seems reasonable for 2-6 weeks, and of the same order of magnitude as the internal cost to negotiate a "free" lisense. For case (1) and for workstation vendors in case (2), there is a genuinely free, higher performance alternative to Predictor. It is the UNIX compress command implementation of LZW. UNIX compress as embodied in the freely and widely distributed BSD source has the following features: - dynamic table clearing when the ratio drops. - automatic turning off of compression when the overall result is not smaller than the input. - dynamic choice of symbol width within predetermined limits. - no lisense required for workstation vendors, since we are all already shipping it and most of us have been shipping it much longer than the PPP working group has existed. - heavily used for many years in networks (on modem and other point-to-point links to transfer netnews) for longer than the PPP working group has existed. - a good performing symbol width requires less than 64KBytes of RAM on both sender and receive, or less than Predictor. - it seems that the UNIX command spends a significant part of its total time reading and writing bytes. The CPU cost of the algorithm itself is on the same order as the cost of I/O, which is more than Predictor, but not a problem for modern, fast machines. - source widely and freely available with no additional lisense for most computer vendors. The UNIX compress algorithm consists essentially of: a. look up byte in hash table, including dealing with collisions and re-probing. b. add to the hash table or not, c. possibly emit a symbol. Notice that looks rather like Predictor. The potentially expensive part is (a), which as usual can be made as cheap as you wish by increasing the size of the hash table. (Recall the ancient rule of thumb that with alpha < .9, you need about 1.1 probes per lookup.) Assuming nothing unexpected happens, I hope to be shipping an implementation that asks for UNIX-compress first and Predictor 1 second. Given two such algorithms, I have absolutely no interest in getting an Officer of the Company to order the corporate lawyers to start their familiar 6 month dance, and not be sure I'll have permission to ship when the music stops. I have publication deadlines far quicker than it is reasonable to expect lawyers to move. Maybe in a future release I'll have time and interest in more algorithms or in something that will take non-programming work. I think the "UNIX compress" listed in the recent CCP draft should be based on a quick and obvious optimization of the UNIX source, removing all of the previous-version compatibility stuff, always enabling "block compression", replacing variables with manifest constants, and so on. With the "bits" being the single parameter that is negotiated. If people do not have a copy of the UNIX compress source, let me know Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 00:57:21 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orjGp-0000DVa; Tue, 26 Oct 93 00:56 PDT Sender: owner-ppp-comp X-Path: netcs.com!okorf From: Oliver Korfmacher To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 8:55:56 MET Message-ID: <9310260755.AA21720@keks.netcs.com> References: <<11931025225011/MAILMAN000NA4EM@mcimail.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > TO: IETF PPP, Data Compression Working Group > > [..] > > The Stacker LZS algorithm is much better. > > Up to two times better compression than Predictor > Software speed up to 1 Mbyte/s compression > Software speed up to 2 Mbytes/s decompression > Hardware speed up to 12 Mbytes/s compression > Hardware speed up to 16 Mbytes/s decompression > Available in hardware or software, cross compatible > It costs the same as Predictor, $0 > Algorithm is already a pending ANSI standard > Many shipping WAN bridges/routers that support > data compression are already using Stacker LZS > Novell will be incorporating Stacker LZS in future versions > of Netware, as well as standardizing on LZS for all their > compression needs in the future > > Here is a preliminary description of the proposed limited license: > > License for LZS compression format - ROYALTY FREE > > The LZS compression format will be fully described in a standard. A royalty > free limited license can be obtained from Stac Electronics which allows the > licensee to implement the LZS compression format. A C source code version > will be provided. > > License for optimized assembly language software from Stac - $10K > > Stac has developed hand coded assembly language versions of the LZS > algorithm for the 80x86, 680x0, and the i960. For a one-time license fee > (royalty free), Stac will provide a highly optimized library, and product > support. Does the above given performance figures (esp. software speed values) are for the 'free' source code or for the assembly language implementations? Oliver From owner-ppp-comp Tue Oct 26 07:16:40 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orpBy-00003pa; Tue, 26 Oct 93 07:16 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 10:13:48 -0400 (EDT) Message-ID: <9310261413.AA05310@donut.gandalf.ca> References: <<11931025225011/MAILMAN000NA4EM@mcimail.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Stac would like the IETF PPP to consider a prospective standards proposal in > which Stac would provide a ROYALTY FREE limited license for the Stacker LZS > data compression format for specified Internetworking applications. I am > still working on the details, but I wanted to forward this information to > this group as soon as possible. Can you discuss the "limited license"? I would assume it grants the user indemnity from the major patents, namely: Waterworth patent 4,701,745 STAC patent 5,016,009 IBM patent 4,054,951 Granted, the IBM patent has just expired (last week). Did this expiration have anything to do with the recent offerings? Further, does this indemnity cover various "spins" of the algorithm? Are we free to improve the algorithm. These are options which are available to all public domain algorithms. Note the various "spins" taking place on Predictor. Or are we restricted to producing a "STAC" compatable data stream. > I believe that we want to find a good default compression algorithm for PPP. > Currently, the Predictor algorithm is under consideration. After > experimenting with the Predictor algorithm for a short while, I conclude > that this would be a poor choice. The compression ratio that it produces is > not nearly as high as LZS. As the packet size (or block size) decreases, it > gets even worse. Funny, with an arithmetic backend, Predictor outperforms STAC. I would like the freedom to enhance LZS. This shouldn't be a surprise. LZ77 derivatives in theory have been shown to be equivalent to a Markov based compressor. > The Stacker LZS algorithm is much better. > > Up to two times better compression than Predictor State tests please. > It costs the same as Predictor, $0 Great! > Algorithm is already a pending ANSI standard Good, but certainly not great. > Many shipping WAN bridges/routers that support > data compression are already using Stacker LZS Yes, and it would provide us with a (lower) compression compatability mode. But hey, I won't look a gift horse in the mouth. > Novell will be incorporating Stacker LZS in future versions > of Netware, as well as standardizing on LZS for all their > compression needs in the future Commercial plug. This is the Internet! > Here is the current list of chips that are available: It would be nice to obtain data sheets, in postscript format over the Internet. If these aren't available, please send data sheets. FAX number in my signature. > Stac's patented LZS format has been marketed by Stac for several years now. > Even though issues of patents and trade secrets seem to hover over every > different compression format, at least Stacker LZS has a track record. Yes. STAC holds some nice patents, STAC has some innovations, and STAC has bought some. LZS format is *not* patented. You cannot patent step codes. The closest patent is the IBM patent covering the use of literal flags, and it just expired. Sure, the encoding is mentioned in your patents, but it hasn't and can't be claimed due to prior art. Your claims, the only part in the patents that matter, have to do with hash refreshing and storing offsets instead of absolute links in the hash list. The other patents have to do with a very big shift register/CAM approach, which are only applicable to hardware implmentations. While all of these patents deserve merit, the actual encoding is patent free, and I could produce a STAC compatable compressor/decompressor without violating any STAC patents. Of course, I'm no patent lawyer. I'm just going by what I have heard. Feel free to quote patent numbers supporting your claim. > The claim that a data compression format is patent free should not be the > reason for choosing it to become a standard. But you just got finished stating it was patented. But freedom from patent infringement *is* one of the most important features of a standard algorithm. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Tue Oct 26 08:15:36 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orq6o-0000Aoa; Tue, 26 Oct 93 08:14 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 11:12:23 -0400 (EDT) Message-ID: <9310261512.AA05429@donut.gandalf.ca> References: <<9310260442.AA24313@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > UNIX compress as embodied in the freely and widely distributed BSD source > has the following features: > - dynamic table clearing when the ratio drops. True. But there is a definite pause in the data flow when this happens. For low speed links it's not a problem, but adaptive methods are preferrable. > - automatic turning off of compression when the overall result > is not smaller than the input. > > - dynamic choice of symbol width within predetermined limits. I like this scalability. Every algorithm should have it. > - no lisense required for workstation vendors, since we are all What about for router/bridge vendors? > - it seems that the UNIX command spends a significant part of > its total time reading and writing bytes. The CPU cost of the > algorithm itself is on the same order as the cost of I/O, which > is more than Predictor, but not a problem for modern, fast > machines. LZW, like Predictor, was built for speed. What does the Rand Goodness Factor for LZW work out to :-) > - source widely and freely available with no additional lisense > for most computer vendors. I agree with you, Unix Compress looks like a fine choice for workstations. It doesn't do much however for the rest of us. > (Recall the ancient rule of > thumb that with alpha < .9, you need about 1.1 probes per lookup.) Now your starting to sound like a mathematician :-) -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Tue Oct 26 08:24:24 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orqFe-0000DSa; Tue, 26 Oct 93 08:24 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 08:23:55 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: Royalty Free Stacker LZS Compression For" on Oct 26, 11:12, Dave Carr writes:] > > - automatic turning off of compression when the overall result > > is not smaller than the input. This is part of the UNIX compress program, but I don't think that it can be done easily within a packet-by-packet approach. UNIX compress looks at the input and output file sizes, and if the output is bigger, it deletes it. This does not apply to "pipe" mode (obviously), and thus would not apply to a packet-by-packet implementation. Expansion (bounded, yes, but expansion none the less) needs to be addressed in the implementation. I did this in my implementation by using a stream modem of delivery (similar to predictor 2) rather than packet-by- packet. > > LZW, like Predictor, was built for speed. What does the Rand Goodness > Factor for LZW work out to :-) I implemented this on our system, but it did not work well at speeds beyond "really slow" (like 19200 bps, sync). I could have done a better job optimizing the code, but... Compression ratio, on the other hand, was *really* good on repetative packets! -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Tue Oct 26 10:11:26 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orrv3-000097a; Tue, 26 Oct 93 10:10 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 10:27:50 -0600 Message-ID: <9310261627.AA28407@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > Novell will be incorporating Stacker LZS in future versions > > of Netware, as well as standardizing on LZS for all their > > compression needs in the future > > Commercial plug. This is the Internet! I disagree. Because of the size of Novell's customer base, that statement is as important to us as any statement about public domain code. That statement implies that a large fraction of PPP installations might find "Stacker LZS" convenient. Of course, that statement also contains obvious marketing hyperbolic nonsense. Because Novell is now also UNIX, Novell will have to continue to support both the System V.3 `pack` as well as the SVR4 (and BSD) `compress` for a long, long time. > > Here is the current list of chips that are available: > > It would be nice to obtain data sheets, ... I agree those chips are the important part of the STAC proposal. It seems STAC is being smart about their business. They are basically giving away the software ($10,000 is all but free in the commercial world), while keeping enough control to ensure that their chips will be useful. If STAC manages to prevent popular variations of their algorithm, their chips could be a good deal for part of the PPP community. It would be nice if hard wired PPP router boxes could compress data on low speed lines (<= 10Mbit/sec). It doesn't matter very much which algorithm you pick, but to get volumes up enough to keep costs low enough to afford compression hardware, you need to minimize the number of choices. However, the software part of the PPP community with roots in the "SLIP replacement" IETF working group has different needs from the part that came from the "common router box protocol" IETF working group. (Those two were merged to form the PPP working group in about 1988. Or was it 1989?). The software part of this community requires something that is completely free with no lisense paperwork at all, and that performs "good enough." We currently have at least two such algorithms, Predictor and "UNIX compress". If STAC could make their code completely free of both fees and paperwork, their scheme would be a candidate to replace Predictor and UNIX compress. None of us want to have to implement more than one compression algorithm or even the mechanisms to pick one of several. Unfortunately, such an open lisense would risk losing control so that the chips might not be useful in as many installations. I think that risk could be made small enough by publishing the C implemenation in the STAC RFC, but that judgement is their responsibility and not ours. About the hassles of "free" lisenses killing standards--is anyone using the 48-bit PPP checksum? Does anyone think it will ever be used? Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 10:18:39 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ors2K-0000Bsa; Tue, 26 Oct 93 10:18 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 10:18:17 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Royalty Free Stacker LZS Compression For" on Oct 25, 22:50, BobL writes:] > The Stacker LZS algorithm is much better. > > Up to two times better compression than Predictor Document this, please. > Software speed up to 1 Mbyte/s compression > Software speed up to 2 Mbytes/s decompression I have been unable to reproduce this. I have seen a compression speed of <200Kbytes per second on a 486/33. > Novell will be incorporating Stacker LZS in future versions > of Netware, as well as standardizing on LZS for all their > compression needs in the future I feel it is improper to make claims on behalf of another company. I will, however, say that "standardizing" on a single algorithm would be a poor business decision for any company. I will see if I can get an Official Novell Opinion on this statement, since I don't speak for Novell. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Tue Oct 26 10:51:18 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orsXv-0000CTa; Tue, 26 Oct 93 10:51 PDT Sender: owner-ppp-comp X-Path: acc.com!fbaker From: fbaker@acc.com (Fred Baker) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 10:07:36 -0800 Message-ID: <9310261708.AB17122@fennel.acc.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >Does the above given performance figures (esp. software speed values) >are for the 'free' source code or for the assembly language implementations? Given the C source, can you not generate optimized assembly that will accomplish the same algorithm? ============================================================================= Don't blame ACC; they think I'm nuts too! From owner-ppp-comp Tue Oct 26 12:42:15 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oruHI-0000DRa; Tue, 26 Oct 93 12:42 PDT Sender: owner-ppp-comp X-Path: um.cc.umich.edu!bill.simpson From: "William Allen Simpson" To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Tue, 26 Oct 93 14:49:25 EDT Message-ID: <1584.bill.simpson@um.cc.umich.edu> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Hmm, if we are going to use IEEE OUIs, they only identify a vendor, not > a specific protocol. If a single vendor supports multiple private > compression algorithms, how are they identified? Do we need at least > another byte for OUI specific codepoints? > Goodness, I hope that the poor vendors don't have to support more than one proprietary protocol! I suggest that each vendor handle multiple internal protocols as a value field of their own definition. The negotiation mechanism certainly won't need to know about it. > Also, I'm assuming that following sort of preference negotiation is > supported (in order of preference): > > Private-A -> Public-B -> Private-C -> Public-D > Just list the options in the order you want them. If you are talking to another of your own boxes, the peer will Ack, but this just means it understands them all. You only really use the first listed. If you send 4, the peer will probably Reject most of them. Say the peer Rejects Private-A and Public-D. Then, you send your favorite of the leftovers Public-B, which you are now certain the peer understands. The peer Acks. Nothing could be simpler! There is apparently some confusion about the operation of Nak and Reject. - Nak changes the "values" part of an option. - Reject rejects the "entire" option. So, in the above example, when you send Public-B, it might have the history value "5". The peer could Nak with value "4". You would then send value "4" in the next Request, since you know that this is an acceptable value for the peer. The peer MUST Ack. Again, simple! Bill.Simpson@um.cc.umich.edu From owner-ppp-comp Tue Oct 26 13:18:52 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oruqj-00006Ca; Tue, 26 Oct 93 13:18 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: UNIX compress encapsulation--forget what I just said Date: Tue, 26 Oct 93 14:02:58 -0600 Message-ID: <9310262002.AA29711@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk On reflection, I think a different encapsulation would be better. The encapsulation for unreliable links: 1. must detect lost packets. It should not worry about corruption or lost bytes, since the PPP CRC detects those. CRC-16 is clearly sufficent, but its cost per byte is about the same as the Predictor cycles/byte and not much less than the LZW cycles/byte. "Less is better." 2. must be quick and easy. So I propose the following: 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+ | CCP Protocol| | Identifier | +-+-+-+-+-+-+-+ |F| Seqno_hi | +-+-+-+-+-+-+-+ | Seqno_mid | +-+-+-+-+-+-+-+ | Seqno_lo | +-+-+-+-+-+-+-+ | Data | +-+-+-+-+-+-+-+ Where the Protocol Identifier specifies CCP, and so is always 0xfd, and may be compressed to 8 bits, but may be as long as 16 bits. F is 1 if the data is compressed, and 0 if not. The data may be uncompressed if the transmitter found it "expanded" instead. Whether the data is uncompressed or not, the LZW state machine is changed identically. Seqno_hi, Seqno_mid, and Seqno_lo comprise a 23-bit packet sequence number. The sequence number after 2**23-1 or 8,388,607 is 0. In other words, the sequence number "wraps" in the usual way. The sequence number is reset to zero whenever the CCP state machine is reset, including as the result of the receiver inferring lost packets by an unexpected sequence number. When the receiver encounters an unexpected sequence number, it MUST send a new configure-request, and the CCP will drop out of the opened state. Upon receipt of the configure-ack, the LZW tables are initialized, and compression can be resumed without data loss. Data consists of the protocol identifier and Info fields of the original packet, compressed or not as indicated by F. The protocol identifier of the original packet MAY be compressed as described in [1], regardless of whether the Protocol field of the CCP Protocol Identifier is compressed or whether PPP Protocol field compression has been negotiated. The CCP Data field is followed by the end of the PPP frame, including the PPP CRC and flag bits or byte. The data is compressed with "block_compress" turned on, with no magic number transmitted, with the code width negotiated by CCP configure-requests and not transmitted in the data, and without any of the rest of the `UNIX compress` self-identifying header. The last code of a packet WILL be padded with 1's to reach an even byte boundary. An implementation of UNIX Compress PPP compression on unreliable links SHOULD ensure that every transition by the LCP state machine out of the opened state also causes the CCP state machine to leave the open state. This encapsulation detects any single lost of up to 8,388,608 packets or at least 40MBytes. Any loss of more than one burst of consecutive packets is also detected, provided no single burst involves more than 8,388,608 packets. The obvious, next smaller sequence number field would not detect losses of larger than 32768 packets or a silence as short as 30 seconds on a 56Kbit/second line. Notice that in the most common case, the data in this encapsulation is as well aligned modulo 4 as the CCP packet itself. ----- Below is my rationalization for UNIX compress PPP compression: UNIX compress as embodied in the freely and widely distributed BSD source has the following features: - dynamic table clearing when the ratio drops. - automatic turning off of compression when the overall result is not smaller than the input. - dynamic choice of code width within predetermined limits. - no lisense required for workstation vendors, since we are all already shipping it and most of us have been shipping it much longer than the PPP working group has existed. - heavily used for many years in networks (on modem and other point-to-point links to transfer netnews) for longer than the PPP working group has existed. - a good performing code width requires less than 64KBytes of RAM on both sender and receive, or less than Predictor. - it seems that the UNIX command spends a significant part of its total time reading and writing bytes. The CPU cost of the algorithm itself is on the same order as the cost of I/O, which is more than Predictor, but not a problem for modern, fast machines. - source widely and freely available with no additional lisense for most computer vendors. vjs From owner-ppp-comp Tue Oct 26 13:19:27 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orurL-00006va; Tue, 26 Oct 93 13:19 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: more CCP draft text Date: Tue, 26 Oct 93 13:59:42 -0600 Message-ID: <9310261959.AA29669@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk Does anyone besides me think that something like the following should be added to the CCP draft?: An implementation of UNIX Compress PPP compression on unreliable links SHOULD ensure that every transition by the LCP state machine out of the opened state also causes the CCP state machine to leave the open state. vjs From owner-ppp-comp Tue Oct 26 13:19:50 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orurc-0000Pna; Tue, 26 Oct 93 13:19 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 12:15:39 -0600 Message-ID: <9310261815.AA29104@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > > - automatic turning off of compression when the overall result > > > is not smaller than the input. > > This is part of the UNIX compress program, but I don't think that it > can be done easily within a packet-by-packet approach. UNIX compress > looks at the input and output file sizes, and if the output is bigger, > it deletes it. This does not apply to "pipe" mode (obviously), and > thus would not apply to a packet-by-packet implementation. Expansion > (bounded, yes, but expansion none the less) needs to be addressed > in the implementation. I did this in my implementation by using a > stream modem of delivery (similar to predictor 2) rather than packet-by- > packet. I disagree. On each packet, look to see if the compressed data is bigger than the original. If it is bigger, then transmit the uncompressed version. Use the exactly the same encapsulation as Predictor, with a 1-bit "compression failed" indicator. Do the obvious in the receiver. The main point is that the ancient UNIX compress command is smart enough to not compress the incompressible, and so patent nonsense is irrelevant. (Yes, I think an extra, redundant CRC is a perfectly acceptible, entirely fast enough mechanism for doing UNIX-compress based compression over unreliable links. Anyone is free to propose a concrete alternative.) > > LZW, like Predictor, was built for speed. What does the Rand Goodness > > Factor for LZW work out to :-) > > I implemented this on our system, but it did not work well at speeds > beyond "really slow" (like 19200 bps, sync). I could have done a > better job optimizing the code, but... That's why it makes sense to have both Predictor and UNIX compress. One for PC-XT's and other VAX 11/750 wannabes, and one for real systems. > Compression ratio, on the other hand, was *really* good on repetative > packets! One of the things James Woods suggested in his 1984 note was preloading the table with interesting things. He liked adding blanks because "most net English [sic] (and also numerical tables) are 15% space.". Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 13:19:52 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orurh-00002Ma; Tue, 26 Oct 93 13:19 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 12:03:32 -0600 Message-ID: <9310261803.AA29023@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > UNIX compress as embodied in the freely and widely distributed BSD source > > has the following features: > > > - dynamic table clearing when the ratio drops. > > True. But there is a definite pause in the data flow when this > happens. For low speed links it's not a problem, but adaptive methods > are preferrable. What "definite pause" are you talking about? You cannot be saying that interesting CPU's find that it takes so long to clear a < 64KB table that they stop outputing or suffer input overruns. If that were true, how do they handle a fill-zero page fault, given that everyone's page size today is at least 4096? In the unlikely event that you do mean that, then there is still no problem. Simply put a "generation or serial number" into each entry in your table, and clear the whole only when the number wraps back to zero. 3 bits is enough to reduce the cost of clearing the table to 13%. Another scheme to completely eliminate any such pauses would be to have two tables, one in use and one spare. At the beginning clear one table and use it. Each time you add an entry to the active table, clear two entries in the spare table. You never clear the UNIX compress table until it is full, by which time the spare one will be clear. So you simply switch tables when the time comes to clear. Let's separate the algorithm embodied by the UNIX compress source from what any half-competent programmer would do to port it to any particular machine. Or are you refering to James Wood's experiments reported in July, 1984 using a 128KBbyte table for 12-bit LZW instead of a hash table? And 2MByte for 16-bit? Naively learing a 128KByte table once took a lot of time. He wrote that it took 6 seconds on his 2MB 11/750. Woods mentioned a hack to make the clearing fast enough even on such ancient relics. I don't think such times nor tables are relevant here. Woods noted that one use of compression is for "slow networks", for TCP/IP over Ethernet that did 50KBytes/sec. Things change. My commerical TCP/IP/FDDI implementation does >12MByte/sec (speed of light) with about 45% of a low cost 1992 workstation's CPU. > > - no lisense required for workstation vendors, since we are all > > What about for router/bridge vendors? As I have said a zillion times since this chimera of a PPP working group was formed by the unholy merging of the SLIP and router working groups, router vendors have their own constraints. In the absense of all other considerations, common solutions are best, but it is not necessary or even rational or reasonable to require a common solution. Such a requirement is irrational because the public domain and workstation software worlds would ignore it except to laugh. > > - source widely and freely available with no additional lisense > > for most computer vendors. > > I agree with you, Unix Compress looks like a fine choice for workstations. > It doesn't do much however for the rest of us. You have my sympathy, but unless you can offer a completely paperwork-free alternative to Predictor and UNIX-compress, sympathy is all you're going to get. I've put the BSD Net-2 UNIX compress source in the ~ftp/other/ppp-comp directory. There is both a tar.Z and a .shar. Don't be frightened by the code. After you remove the backward compatibility stuff, and do the obvious simplifications, it is quite simple. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 14:09:33 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orvcr-0000cKa; Tue, 26 Oct 93 14:08 PDT Sender: owner-ppp-comp X-Path: um.cc.umich.edu!bill.simpson From: "William Allen Simpson" To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression Date: Tue, 26 Oct 93 16:11:24 EDT Message-ID: <1587.bill.simpson@um.cc.umich.edu> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk I am very happy that they decided to do this! > Unfortunately, such an open lisense would risk losing control so that > the chips might not be useful in as many installations. I think that > risk could be made small enough by publishing the C implemenation in > the STAC RFC, but that judgement is their responsibility and not ours. > So you think that they SHOULD publish the code? I was telling them no, best to just publish the data format, and give the free C source to people who bother to ask for the "free" license. > About the hassles of "free" lisenses killing standards--is anyone using > the 48-bit PPP checksum? Does anyone think it will ever be used? > Unfortunately, DEC decided not to make it free. They decided they wanted $20,000. Since it improves the performance on one (1) packet during negotiation, it was a pretty worthless offer. I just used a work-around (in lcpext). Therefore, it will never be used. As for Stacker as a "standard", I thought we agreed not to agree, just negotiate??? I don't think that Stacker can ever become an IETF standard. Our standards require multiple independently developed interoperable implementations. Stacker would be only one implementation, with source. I recommend "Informational". The CCP would be the IETF standard. The algorithms would be informational. May the most widely used "win" -- without a formal decision! Bill.Simpson@um.cc.umich.edu From owner-ppp-comp Tue Oct 26 14:40:45 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orw7y-0000DSa; Tue, 26 Oct 93 14:40 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 17:38:10 -0400 (EDT) Message-ID: <9310262138.AA06366@donut.gandalf.ca> References: <<9310261815.AA29104@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > > > > - automatic turning off of compression when the overall result > > > > is not smaller than the input. > > I disagree. > On each packet, look to see if the compressed data is bigger than the > original. If it is bigger, then transmit the uncompressed version. > Use the exactly the same encapsulation as Predictor, with a 1-bit > "compression failed" indicator. Do the obvious in the receiver. Yes, but to make it work, you must: (1) Reset the compressors dictionary; OR Not practical, for any size dictionary. (2) Add a hash table to the decoder; The memory requirements just went up. > The main point is that the ancient UNIX compress command is smart > enough to not compress the incompressible, and so patent nonsense is > irrelevant. Yes, sending through the original data would be "obvious to one skilled in the art". I haven't seen any patents which cover it, and I've read quite a few. > One of the things James Woods suggested in his 1984 note was preloading > the table with interesting things. He liked adding blanks because > "most net English [sic] (and also numerical tables) are 15% space.". For LZW this may be a win on (real) short sessions. For V.42 bis, it may or may not help, depending on the V.42 bis settings. For LZ77 algorithms, it's never a win. The reason is the LZ77 encoder can say "back up one character and copy 50", to compress a string of 51 blanks. Your better off doing run length encoding first. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Tue Oct 26 15:00:16 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orwQK-0000Qza; Tue, 26 Oct 93 14:59 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 1993 17:56:50 -0400 (EDT) Message-ID: <9310262156.AA06381@donut.gandalf.ca> References: <<9310261803.AA29023@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > What "definite pause" are you talking about? You cannot be saying that > interesting CPU's find that it takes so long to clear a < 64KB table > that they stop outputing or suffer input overruns. If that were true, > how do they handle a fill-zero page fault, given that everyone's page > size today is at least 4096? What's a page? Is that something like a segment? :-) No seriously, filling a 64K (shorts) table will require require 8K burst cycles, at roughly 10 clocks each on my processor. That's about 4 msec. Now, to keep the pipe full, I must always queue at least that much on each link. That adds 4 msec of latency. Considering an algorithm like predictor might have 1.5 msec of latency on a maximum length packet, the 4 msec latency would seem a bit big. I have no such pauses in FZA. In fact, I took pains to avoid these pauses. The "amortised cost" of your approach may be good, but the worst case is significantly worse. A lot of real time design work is dealing with the worst case. Your just moving the your latency problem to another layer. It's easier for other layers if you have a known delay. > > In the unlikely event that you do mean that, then there is still no > problem. Simply put a "generation or serial number" into each entry > in your table, and clear the whole only when the number wraps back > to zero. 3 bits is enough to reduce the cost of clearing the table > to 13%. I'd like to see the code for clearing part of a LZW table. > > Another scheme to completely eliminate any such pauses would be to have > two tables, one in use and one spare. At the beginning clear one table Patented. :-( > Or are you refering to James Wood's experiments reported in July, 1984 Never read it. Just basing it on my experiments. > You have my sympathy, but unless you can offer a completely paperwork-free > alternative to Predictor and UNIX-compress, sympathy is all you're > going to get. I don't need sympathy, I've got FZA, Predictor, and now STAC. Why would I need LZW? -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Tue Oct 26 15:05:02 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0orwUo-0000UYa; Tue, 26 Oct 93 15:04 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: UNIX compress encapsulation Date: Tue, 26 Oct 93 15:07:27 -0600 Message-ID: <9310262107.AA00431@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >From: tommyd@microsoft.com > ... I'm curious > as to why you choose a 23 bit counter instead of a 15 bit counter? As I tried to say in my message today, a simple 15 bit packet counter would not detect line drop outs as short as 32768 packets, and since a CCP packet can be as small as about 8 bytes, could not detect line drop outs as short as 200KBbytes or 1.6Mbits, which is only about a second on a T1 line. I find it easy to imagine a 10 second burst of 80 Byte packets being lost between the transmitter and receiver with no indication to the receiver except a temporary loss of synchronization. vjs From owner-ppp-comp Tue Oct 26 17:14:01 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0oryWH-0000Cea; Tue, 26 Oct 93 17:13 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Tue, 26 Oct 93 18:11:47 -0600 Message-ID: <9310270011.AA01703@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >From: Bill.Simpson@um.cc.umich.edu > ... > > Private-A -> Public-B -> Private-C -> Public-D > If you send 4, the peer will probably Reject most of them. Say the peer > Rejects Private-A and Public-D. Then, you send your favorite of the > leftovers Public-B, which you are now certain the peer understands. > The peer Acks. Nothing could be simpler! As I understand CCP, this cannot not be done. You're required to put all of your public choices into a single option, aren't you? That's part of why you (Bill) and I complained so much about a REJ for only part of one of those mongo-public requests. And why Dave added words say you should REJ an option only if you understand none of it. (Or have I read too much into the existence of the multiple algorithms in option type 1?) I still think it would be cleanest if each public config. request (i.e. CCP option type 1) would have exactly one (1) requested algorithm, that such a scheme would be fast enough and far easier to code correctly. However, no one seems to agree me, and I don't want to argue about it. > There is apparently some confusion about the operation of Nak and Reject. > > - Nak changes the "values" part of an option. > > - Reject rejects the "entire" option. I think we have gotten that words to that effect in the CCP draft. > So, in the above example, when you send Public-B, it might have the > history value "5". The peer could Nak with value "4". You would then > send value "4" in the next Request, since you know that this is an > acceptable value for the peer. The peer MUST Ack. Again, simple! Are you saying that the more than one multi-algorithm type 1 config option might be in a single CCP config. request? If so, why do we need the complexity of multiple algorithms in a single config-option? vjs From owner-ppp-comp Tue Oct 26 19:36:16 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0os0jr-0000TYa; Tue, 26 Oct 93 19:35 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression Date: Tue, 26 Oct 93 20:34:55 -0600 Message-ID: <9310270234.AA02607@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > I am very happy that they decided to do this! I would be happy to, if they do it completely. Having to negotiate a "free" lisense is not quite enough to get over the hump. > > Unfortunately, such an open lisense would risk losing control so that > > the chips might not be useful in as many installations. I think that > > risk could be made small enough by publishing the C implemenation in > > the STAC RFC, but that judgement is their responsibility and not ours. > > So you think that they SHOULD publish the code? I was telling them no, > best to just publish the data format, and give the free C source to > people who bother to ask for the "free" license. If STAC promises to not sue anyone with deep pockets who implements to their interface without having negotiated a "free" lisense, and if their interface is sufficently well defined to allow someone to implement something interoperable, then I agree the C source is only nice and not necessary. However, I read the message as saying "If you sign a free lisense, only then you may use implement LZS algorithm. Otherwise, expect to spend time in court." A no-charge lisense that requires an official corporate signature is not remotely really free. > > About the hassles of "free" lisenses killing standards--is anyone using > > the 48-bit PPP checksum? Does anyone think it will ever be used? > > > Unfortunately, DEC decided not to make it free. They decided they > wanted $20,000. Since it improves the performance on one (1) packet > during negotiation, it was a pretty worthless offer. I just used a > work-around (in lcpext). > > Therefore, it will never be used. No, and yes. No, a no-royaties $20,000 lisense is almost as free as a $0.20 lisense for practical commerical purposes. $20,000 is a minor increase in the cost of product development. $20,000 would buy only 0.05 to 10 extra test machines. $20,000 is good for no more than 2 man-months of programmer time. How far will $20,000 in an advertising budget? Or a trade show booth? Yes, it is a minor improvement not worth the $10,000 or more necessary to get through the paperwork hoops regardless of the check you'd have to get sent to DEC, and so will never be used. I was trying to point out the similarity to between the 48-bit checksum and what I understand to be the current STAC offer. Good, but not quite good enough. > As for Stacker as a "standard", I thought we agreed not to agree, > just negotiate??? > > I don't think that Stacker can ever become an IETF standard. Our > standards require multiple independently developed interoperable > implementations. Stacker would be only one implementation, with > source. I recommend "Informational". Wouldn't it be nice to have a single, common, moderately high performance, low cost algorithm that would always be available? Wouldn't it be nice to chuck all of that CCP negotiation code Morningstar is going to have to have? Wouldn't it be nice to not have to have at least two and perhaps more compression and decompression paths for data? We better agree on some things now, starting with the length fields in CCP and continuing through Predictor, PuddleJumper, and LZW encapsulations or our computers will never have anything to negotiate. Still, I agree. The chances of agreement on a single default compression algorithm here are smaller than the chances that Microsoft and Novell/USL will agree that BSDI's definition of "multi-user operating system for 80*86's" is the single, right answer. (I trust everyone knows about Windows-NT, System V, BSD lawsuits, etc.) Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 20:40:27 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0os1ja-0000bta; Tue, 26 Oct 93 20:39 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 21:38:43 -0600 Message-ID: <9310270338.AA03126@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: Dave Carr > ... > I have no such pauses in FZA.... I really wish you would either make FZA public, open and genuinely free or stop talking about it. While I accept your unsupported claims about its wonderfulness, they are irrelevant. No one I know is going to pay money to you or anyone else for a PPP compression algorithm. No, I still will not sign an non-disclosure agreement for the priviledge of learning that your claims for FZA are true. I already accept them as true but entirely uninteresting, because you said FZA is patent pending and requires a lisense. > > In the unlikely event that you do mean that, then there is still no > > problem. Simply put a "generation or serial number" into each entry > > in your table, and clear the whole only when the number wraps back > > to zero. 3 bits is enough to reduce the cost of clearing the table > > to 13%. > > I'd like to see the code for clearing part of a LZW table. You don't clear part of it, you clear all of it. The 12.5% improvement is on average, not worst case. Consider a 4th scheme, where you separate the generation numbers into another table and clear only them. Using a separate table of 1 bit generation numbers would cost the average compression time about 6 instruction cycles and 1 data read (MIPS R3000) per hash probe and reduce the cost to clear the table by a factor of 16, to be paid on only 50% of the table clears. > > Another scheme to completely eliminate any such pauses would be to have > > two tables, one in use and one spare. At the beginning clear one table > > Patented. :-( Such a patent is unmitigated manure. Such coloring schemes have been around for decades in garbage collectors, and are now obvious to everyone. What is clearing the LZW hash table/list except garbage collecting? I'm convinced most software patents are filed to feed the egos of people who want to believe they can invent something. Software patents may be defended by lawyers and accountants out to make a quick buck or stifle competition, but if enough programmers had even a little integrety, nonsense like the patents you keep waving in my face would never exist. (I think there are some software ideas that are worthy of patents; it's just that you keep waving the garbage majority.) The only rational way to handle stupid patents is to ignore them. They don't matter in this particular discussion, since the way you clear your dictionary should not be part of any IETF standard. Do you, who keeps dragging out this patent nonsense, do a patent search for every 50 lines of code you write? (I presume you agree that any of the 4 schemes I've sketched to mitigate the clearing of the LZW dictionary would take less than 100 lines of C.) > > You have my sympathy, but unless you can offer a completely paperwork-free > > alternative to Predictor and UNIX-compress, sympathy is all you're > > going to get. > > I don't need sympathy, I've got FZA, Predictor, and now STAC. Why would > I need LZW? You have STAC only if you pay your lawyers to ok the lisense for it. I presume you have a lisense for FZA, whatever it is. You say LZW is too expensive for your hardware. Fine. You're happy, so I'm happy. I do not expect Silicon Graphics to sign a lisense for anyone's compression scheme**. That could leave me with Predictor, which I've been convinced is not as effective as LZW. UNIX compress requires no lisense for me. I have no fears about clearing a trivial 64KByte buffer at lethargic data rates of <=10Mbit/sec. So I'll do UNIX compress. Q.E.D. Given our separate markets, it is unlikely that my code and your code will ever exchange LCP packets outside a test bed, if there. Our code will negotiate Predictor or nothing, and everything will be peachy. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Tue Oct 26 21:04:49 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0os27G-0000bpa; Tue, 26 Oct 93 21:04 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Tue, 26 Oct 93 22:03:01 -0600 Message-ID: <9310270403.AA03285@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk From: dcarr@gandalf.ca > > > > > - automatic turning off of compression when the overall result > > > > > is not smaller than the input. > > > > I disagree. > > On each packet, look to see if the compressed data is bigger than the > > original. If it is bigger, then transmit the uncompressed version. > > Use the exactly the same encapsulation as Predictor, with a 1-bit > > "compression failed" indicator. Do the obvious in the receiver. > > Yes, but to make it work, you must: > > (1) Reset the compressors dictionary; OR > Not practical, for any size dictionary. Not necessarily impractical. I know someone who wants to do 9 and 10 bit LZW. Those tables are small enough to snapshot with bcopy() before every packet. > (2) Add a hash table to the decoder; > > The memory requirements just went up. Yes, the memory requirement "just went up" to less than Predictor. Besides, you forgot: (3) use a generation number for each packet's change to the table, and roll back changes using a separate bit mask. (For a 4 bit generation number, use a single 16-bit mask.) When you have burned enough generation numbers, gabage collect them. Give me a little time, and I can come up with (4), (5), (6), and (7). > > One of the things James Woods suggested in his 1984 note was preloading > > the table with interesting things. He liked adding blanks because > > "most net English [sic] (and also numerical tables) are 15% space.". > > For LZW this may be a win on (real) short sessions. For V.42 bis, it may or > may not help, depending on the V.42 bis settings. For LZ77 algorithms, it's > never a win. The reason is the LZ77 encoder can say "back up one character > and copy 50", to compress a string of 51 blanks. Your better off doing > run length encoding first. I don't think that Woods was worried about strings of blanks, and that run-length encoding would not be useful for English text. I think Woods was refering to the blanks between words. Consider the rule of thumb that says average words are 5 or 6 characters long. The blanks after each word in such text amount to about 15%. I understand his idea to be to add "X-blank for all characters X". In the few UNIX-compress experiments I've tried where I expected run-length coding to be the only answer, compressing chunks of /dev/zero, UNIX-compress did just fine. You don't need the ultimate compression. A factor of 10 or 20 best case is more than good enough here. Leave 200:1 to the special situations like MPEG. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Wed Oct 27 08:48:12 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osD6L-00005xa; Wed, 27 Oct 93 08:47 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression Date: Wed, 27 Oct 1993 11:45:30 -0400 (EDT) Message-ID: <9310271545.AA06774@donut.gandalf.ca> References: <<1587.bill.simpson@um.cc.umich.edu>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > So you think that they SHOULD publish the code? I was telling them no, > best to just publish the data format, and give the free C source to > people who bother to ask for the "free" license. The format of LZS may have been published. It's in their patents. Unless of course they don't stick to it. > I don't think that Stacker can ever become an IETF standard. Our > standards require multiple independently developed interoperable > implementations. Stacker would be only one implementation, with > source. I recommend "Informational". I believe it is possible to create a compatible version if they publish a description. It it follows the description in their patents, it should be a no-brainer to write a compatible version. Of course, it would be nice if the compatable version could use some of the patented parts of LZS. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Wed Oct 27 09:33:36 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osDo3-0000Bfa; Wed, 27 Oct 93 09:33 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 1993 12:30:11 -0400 (EDT) Message-ID: <9310271630.AA06814@donut.gandalf.ca> References: <<9310270338.AA03126@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > about its wonderfulness, they are irrelevant. No one I know is going > to pay money to you or anyone else for a PPP compression algorithm. How wrong you are. But enough said on FZA. I was merely trying to point out that a periodic function such as the dictionary reset has real-time impacts, which may be unacceptable in some cases. A good example is LZS, which was the original thread. In LZS (from the STAC patents), they use an amortised hash refresh. By comparision INFO-ZIP uses a periodic hash refresh. INFO-ZIP pauses to rehash the entire 32K history buffer, and recreate the hash table and hash links. As it turns out, the INFO-ZIP method is faster overall than the amortised method used in STAC algorithm. But it has a latency aspect. Obviously, STAC thought that this wasn't acceptable either. STAC has hard real-time constraints. INFO-ZIP does not. > accept them as true but entirely uninteresting, because you said FZA is > patent pending and requires a lisense. Wrong again. FZA is not patent-pending. I don't believe I said it was. If I did, then I was wrong. A license is however required. > > > Another scheme to completely eliminate any such pauses would be to have > > > two tables, one in use and one spare. At the beginning clear one table > > > > Patented. :-( > > > Such a patent is unmitigated manure. Such coloring schemes have been > around for decades in garbage collectors, and are now obvious to everyone. > What is clearing the LZW hash table/list except garbage collecting? There are *several* patents on LZW which may appear to be "obvious to one skilled in the art", or "invalidated by prior art". Nonetheless, they still exist. It costs time and money to invalidate them. Often it's cheaper to license the patent then to fight it. I'm not defending the patent system, but I'm not ignoring it either. > I'm convinced most software patents are filed to feed the egos of > people who want to believe they can invent something. Software patents > may be defended by lawyers and accountants out to make a quick buck or > stifle competition, but if enough programmers had even a little > integrety, nonsense like the patents you keep waving in my face would > never exist. (I think there are some software ideas that are worthy of > patents; it's just that you keep waving the garbage majority.) I'm in agreement here. In fact, filing one about 18 months ago just proved to me how riduculous the whole patent system is. Most of the people in the patent office don't have a clue what the patent is about, they merely look at the wording of the claims. If it looks unique, it's passes. > The only rational way to handle stupid patents is to ignore them. They > don't matter in this particular discussion, since the way you clear > your dictionary should not be part of any IETF standard. Your right. Clearing the table is in UNIX COMPRESS, and the method of clearing cannot (in theory) be patented. But FYI, here are some patents that have been granted. Do with them what you want: U.S. 4,847,619 Performance Based Reset of Data Compression Dictionary Kato et. al. Hewlett-Packard Company. Granted Jul.11,1989 Filed Oct.19, 1987 U.S. 4,881,075 Method and Appartus for Adaptive Data Compression Weng. Digital Equipment Corporation Granted Nov. 14,1989 Filed Oct 15,1987 > Do you, who > keeps dragging out this patent nonsense, do a patent search for every > 50 lines of code you write? Since most compressors tend to be a one or two hundred lines of code, this is not extreme. Just the hashing alone is worthy of a patent search. There are several patents (Graybill, STAC) just on the hashing. Often, there are design choices, and the only decisive factor is whether one option is patented. Actually, I'm not the only one concerned with patents. Remember Dave Rand? There are probably others. > (I presume you agree that any of the 4 > schemes I've sketched to mitigate the clearing of the LZW dictionary > would take less than 100 lines of C.) I agree the schemes can be implemented in less 100 lines of code. Probably a lot less. I think your taking my comments a bit wrong. I'm not trying to shoot down LZW for you or anyone else. I merely disemminating my knowledge and research in the field. If no-one wants it, I'll gladly shut up. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Wed Oct 27 09:54:33 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osE8M-0000BFa; Wed, 27 Oct 93 09:54 PDT Sender: owner-ppp-comp X-Path: um.cc.umich.edu!bill.simpson From: "William Allen Simpson" To: ppp-comp@bungi.com Subject: Re: New CCP draft (rev 01) Date: Wed, 27 Oct 93 09:28:23 EDT Message-ID: <1589.bill.simpson@um.cc.umich.edu> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) > As I understand CCP, this cannot not be done. > ... > If so, why do we need the complexity of multiple algorithms in > a single config-option? > We don't. It's illegal. The CCP draft is wrong. It _will_ be fixed, if I have to write one myself. (I have already discussed this with Fred, and agreed to hold off until after Houston.) Bill.Simpson@um.cc.umich.edu From owner-ppp-comp Wed Oct 27 10:07:08 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osEJk-0000C3a; Wed, 27 Oct 93 10:05 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 1993 13:03:01 -0400 (EDT) Message-ID: <9310271703.AA06841@donut.gandalf.ca> References: <<9310270403.AA03285@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Besides, you forgot: Oh there's lot's of things I haven't thought of. That's the beauty of forums such as this. > (3) use a generation number for each packet's change to the table, and > roll back changes using a separate bit mask. (For a 4 bit > generation number, use a single 16-bit mask.) > When you have burned enough generation numbers, gabage collect them. Yes, that's possible. Again, an O(N) solution, but only when you need to roll back. > I don't think that Woods was worried about strings of blanks, and that > run-length encoding would not be useful for English text. I think > Woods was refering to the blanks between words. Consider the rule of > thumb that says average words are 5 or 6 characters long. The blanks > after each word in such text amount to about 15%. I understand his > idea to be to add "X-blank for all characters X". Well actually you would also PREFIX each character in the base dictionary creating a total of 512 new strings in the dictionary at initialization. > In the few UNIX-compress experiments I've tried where I expected > run-length coding to be the only answer, compressing chunks of > /dev/zero, UNIX-compress did just fine. The effects of run length encoding counteract the realitively slow learning (one character at a time) of Unix Compress. Other LZW variants such as Storers AP, or V.42bis (with appropriate settings) get the same effect without run length. > You don't need the ultimate > compression. A factor of 10 or 20 best case is more than good enough > here. Leave 200:1 to the special situations like MPEG. Nah. It looks too "me-too"-ish on the data sheet :-) If something is "almost free" in terms of CPU and memory, take it. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Wed Oct 27 12:54:07 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osGwN-0000Pfa; Wed, 27 Oct 93 12:53 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 93 13:53:04 -0600 Message-ID: <9310271953.AA08103@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > Do you, who > > keeps dragging out this patent nonsense, do a patent search for every > > 50 lines of code you write? > > Since most compressors tend to be a one or two hundred lines of code, > this is not extreme. Just the hashing alone is worthy of a patent search. > There are several patents (Graybill, STAC) just on the hashing. > Often, there are design choices, and the only decisive factor is whether > one option is patented. > > Actually, I'm not the only one concerned with patents. Remember Dave Rand? > There are probably others. You missed my point, or I've made a bad assumption about what you do for a living. Do you use classic algorithms and data structures or invent your own only in PPP compression, or work only on PPP compression? Some people produce only the classic, big company number of 500 lines of debugged, commented code per year. Other people do 10 or 100 times that much. If I had to do a patent search for every 50 or 100 lines, I would never get anything done. Doing a patent search for every data structure or algorithm you design would effectively stop your work, unless you spend a month or more for each one. Besides being a waste of time as far as producing cod, patent searches are worthless in and of themselves, since it is both practially and formally impossible to determine if an arbitrary algorithm is the same as any of the patented algorithms. (Consider the halting problem.) Concern about patents is necessary to ensure that an IETF standard does not implicitly or explicitly require an implementation to infringe on a valid patent. Your and other people's investigations on those questions are very valuable. However, other than that, how are patent concerns relevant to an IETF mailing list? The STAC patents seem quite relevant, since it sounds as if any implementation of LZS would be presumed to infringe, and an unlisensed implementation would likely draw court orders. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Wed Oct 27 15:08:42 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osJ1c-0000Oma; Wed, 27 Oct 93 15:07 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 1993 18:02:57 -0400 (EDT) Message-ID: <9310272203.AA06918@donut.gandalf.ca> References: <<9310271953.AA08103@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > You missed my point, or I've made a bad assumption about what you do > for a living. I'm a principal designer. If you know what that is, please tell me so I can tell my boss and I can have a job description to go with the fancy title. I write less than 300 lines of code a *year*, and hopefully it doesn't have any bugs in it. I tend to write code that requires a lot of delicate handling (drivers) or have the need for speed (packet filters). Compression for me is a hobby, that sometimes interferes with my work. My company doesn't view it as a full-time job. I spend time architecting products, brainstorming, and the like, and haven't done "project" type work for 5 or more years. > Do you use classic algorithms and data structures or invent your own > only in PPP compression, or work only on PPP compression? I've done compressors for stat-muxes, and bridges. Now for PPP. I tend to research not invent. There just isn't that much new in data compression. Limpel and Ziv did some work, and all that's really happened since is arithmetic coding. All the patents and paper are really different applications of the same stuff, faster, better, ... engineering not mathematics. Thank god, otherwise us engineers would be lost. Now, engineers tend to use 3 structures, linked-lists, ring buffers, and (what was that third one, scratching his head) and occasionally a binary tree. 99% of the code in the world uses these structures. Compression requires fancier structures. There's a reward for using them, speed. Some of the best compression data structures aren't in textbooks. > Some people > produce only the classic, big company number of 500 lines of debugged, > commented code per year. Too high for me. They might use me as a programmer if I was that fast :-) > Other people do 10 or 100 times that much. Yeh, we've got guys like that too. > If I had to do a patent search for every 50 or 100 lines, I would never > get anything done. Doing a patent search for every data structure or > algorithm you design would effectively stop your work, unless you spend > a month or more for each one. I agree. Let me at least point out to you the problem, if you don't have time to do it. Don't rip into me then. *I ain't do it for my health you know*. > Besides being a waste of time as far as > producing cod, patent searches are worthless in and of themselves, > since it is both practially and formally impossible to determine if an > arbitrary algorithm is the same as any of the patented algorithms. > (Consider the halting problem.) Oh my, now your diving into that math/compsci techno-garble. I thought you were an engineer too :-) I would be damn hard to know that if someone had pirated FTP code running in their box since it's internal and could have been arbitrarily modified to confuse/defeat a disassembler. It's a whole different story to claim for example that a router is V.42bis compliant, without paying for V.42bis. The algorithm dictates the bit encoding, and the algorithm is covered by several patents. > However, other than that, how are patent concerns relevant to an IETF > mailing list? That's probably the limit. I can't (nor do I want to stop) you from using a patented algorithm. But there may be a newcomer on the list who doesn't know the patents and needs the information. I'm just disemminating it. I know it, they may not. I'm trying to be a good guy. I'm not trying to be offensive. > The STAC patents seem quite relevant, since it sounds as if any > implementation of LZS would be presumed to infringe, and an unlisensed > implementation would likely draw court orders. Buzzzt wrong! It is quite the opposite. STAC cannot patent that bit encoding. It can be generated with an LZSS encoder, public domain code for some ten years. That's an LZ77 characteristic. Many compatible compressors/decompressors with different source. In fact the STAC patents that I have seem never claim the encoding. There may have been other patents that I have missed when I did my search, but I would find it hard to believe that STAC can patent any class of simple "step codes" or "unary codes". There patents, and they are quite valid in my mind, deal with amortised hash updates and a 2048 entry shift register-comparator for finding matches. That's it. Now I be lying if I said I didn't want the patents in the public domain. The patents themselves are quite useful. On the other hand, I could have had a STAC compatable compressor years ago. It's not even rocket science. I don't even think that the current round of court battle on reverse engineering would apply if I sat down with a line analyser and worked out the bit encodings. I could be wrong here. But hey, they gave me a head start by publishing patents. If they wanted it a secret, don't publish. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Wed Oct 27 15:25:31 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osJHg-00000ka; Wed, 27 Oct 93 15:24 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 1993 18:02:57 -0400 (EDT) Message-ID: <9310272203.AA06918@donut.gandalf.ca> References: <<9310271953.AA08103@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > You missed my point, or I've made a bad assumption about what you do > for a living. I'm a principal designer. If you know what that is, please tell me so I can tell my boss and I can have a job description to go with the fancy title. I write less than 300 lines of code a *year*, and hopefully it doesn't have any bugs in it. I tend to write code that requires a lot of delicate handling (drivers) or have the need for speed (packet filters). Compression for me is a hobby, that sometimes interferes with my work. My company doesn't view it as a full-time job. I spend time architecting products, brainstorming, and the like, and haven't done "project" type work for 5 or more years. > Do you use classic algorithms and data structures or invent your own > only in PPP compression, or work only on PPP compression? I've done compressors for stat-muxes, and bridges. Now for PPP. I tend to research not invent. There just isn't that much new in data compression. Limpel and Ziv did some work, and all that's really happened since is arithmetic coding. All the patents and paper are really different applications of the same stuff, faster, better, ... engineering not mathematics. Thank god, otherwise us engineers would be lost. Now, engineers tend to use 3 structures, linked-lists, ring buffers, and (what was that third one, scratching his head) and occasionally a binary tree. 99% of the code in the world uses these structures. Compression requires fancier structures. There's a reward for using them, speed. Some of the best compression data structures aren't in textbooks. > Some people > produce only the classic, big company number of 500 lines of debugged, > commented code per year. Too high for me. They might use me as a programmer if I was that fast :-) > Other people do 10 or 100 times that much. Yeh, we've got guys like that too. > If I had to do a patent search for every 50 or 100 lines, I would never > get anything done. Doing a patent search for every data structure or > algorithm you design would effectively stop your work, unless you spend > a month or more for each one. I agree. Let me at least point out to you the problem, if you don't have time to do it. Don't rip into me then. *I ain't do it for my health you know*. > Besides being a waste of time as far as > producing cod, patent searches are worthless in and of themselves, > since it is both practially and formally impossible to determine if an > arbitrary algorithm is the same as any of the patented algorithms. > (Consider the halting problem.) Oh my, now your diving into that math/compsci techno-garble. I thought you were an engineer too :-) I would be damn hard to know that if someone had pirated FTP code running in their box since it's internal and could have been arbitrarily modified to confuse/defeat a disassembler. It's a whole different story to claim for example that a router is V.42bis compliant, without paying for V.42bis. The algorithm dictates the bit encoding, and the algorithm is covered by several patents. > However, other than that, how are patent concerns relevant to an IETF > mailing list? That's probably the limit. I can't (nor do I want to stop) you from using a patented algorithm. But there may be a newcomer on the list who doesn't know the patents and needs the information. I'm just disemminating it. I know it, they may not. I'm trying to be a good guy. I'm not trying to be offensive. > The STAC patents seem quite relevant, since it sounds as if any > implementation of LZS would be presumed to infringe, and an unlisensed > implementation would likely draw court orders. Buzzzt wrong! It is quite the opposite. STAC cannot patent that bit encoding. It can be generated with an LZSS encoder, public domain code for some ten years. That's an LZ77 characteristic. Many compatible compressors/decompressors with different source. In fact the STAC patents that I have seem never claim the encoding. There may have been other patents that I have missed when I did my search, but I would find it hard to believe that STAC can patent any class of simple "step codes" or "unary codes". There patents, and they are quite valid in my mind, deal with amortised hash updates and a 2048 entry shift register-comparator for finding matches. That's it. Now I be lying if I said I didn't want the patents in the public domain. The patents themselves are quite useful. On the other hand, I could have had a STAC compatable compressor years ago. It's not even rocket science. I don't even think that the current round of court battle on reverse engineering would apply if I sat down with a line analyser and worked out the bit encodings. I could be wrong here. But hey, they gave me a head start by publishing patents. If they wanted it a secret, don't publish. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Wed Oct 27 15:42:48 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osJYO-0000FKa; Wed, 27 Oct 93 15:41 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 93 16:39:48 -0600 Message-ID: <9310272239.AA09035@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > (3) use a generation number for each packet's change to the table, and > > roll back changes using a separate bit mask. (For a 4 bit > > generation number, use a single 16-bit mask.) > > When you have burned enough generation numbers, gabage collect them. > > Yes, that's possible. Again, an O(N) solution, but only when you need > to roll back. That it is O(N) is not quite right, since for fixed dictionary size, LZW is itself order N. All of the costs per byte or per packet for all dictionary rolling back or forward are bounded by a constant, given that fixed dictionary size. What matters more in protocol designs like this, where N is formally infinite and so O(N) has an academic meaning, if any, is the constant hidden in the O() notation. For example, the previous complaint about the time to flush the UNIX-compress dictionary had nothing to do with the fact that its cost is a constant times the number of bytes. Your complaint was entirely about the constant, and my suggestions only reduced that constant. In this case, what matters is reducing the constant number of cycles/byte or cycles/packet required roll back a packet's worth of dictionary changes. That hack I described above is a trivial "coloring" elaboration of "change bits," obvious to anyone who has piddled with database design and implementation. (While you didn't force or let me to produce (4), (5), (6), or (7), I can't resist pointing out that more than one of the various techniques used in databases to rollback changes suggest useful hacks for your problem of rolling back compression dictionary changes. Please don't bother mentioning database patents.) In any case, I do not think it is desirable the rollback the changes. It is better to ship the uncompressed/unexpanded data and let the receiver update its dictionary. Otherwise, you can get stuck with a less than full UNIX compress dictionary on the sender forever. Consider the likelihood of whether the very first packet expands or compresses, and if you don't update the receiver's dictionary, the likelihood that you would never, ever send a compressed packet. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Wed Oct 27 17:32:07 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osLGu-0000SCa; Wed, 27 Oct 93 17:31 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Royalty Free Stacker LZS Compression For Date: Wed, 27 Oct 93 18:30:21 -0600 Message-ID: <9310280030.AA09980@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > I tend to write code that requires a lot of delicate > handling (drivers) or have the need for speed (packet filters). > ... I and some of my collegues do the same kind of code, but 3,000 to 30,000 lines/year/person. E.g. FDDI that goes at lightspeed, graphics microcode, X servers, compilers, (no doubt you've heard of MIPS compilers and SGI graphics) and even packet filters (snooping on FDDI packets requires fast hardware, fast software, cheating, or all 3). > ... > Now, engineers tend to use 3 structures, linked-lists, ring buffers, and > (what was that third one, scratching his head) and occasionally a binary > tree. 99% of the code in the world uses these structures. Compression > requires fancier structures. There's a reward for using them, speed. > Some of the best compression data structures aren't in textbooks. Well, each to their own. I find it is effective to use the right structure for the job, without paying attention to the craft unions for each structure that would prefer only members of the hashwrights guild touch hashes, and so on. Some of my fellow old farts and I occassionally gather to bemoan the new Computer Industry, where no one knows much of anything about anything, and everyone even seems affraid to do anything not in Knuth vol 1 or 3, and almost everyone is too afraid of feeling dumb to even open those old books. We grumble that almost all code should be optimized for speed or size or both, that almost all code should be written with the most appropriate tools, and that only code to be run only once or twice, like the quick tests written in /tmp/foo.c or typed directly at a compiler should not use the right structures and the right algorithms. > ... > It's a whole different story to claim for example that a router is V.42bis > compliant, without paying for V.42bis. The algorithm dictates the bit > encoding, and the algorithm is covered by several patents. Exactly. For v.42bis, patents matter. > > The STAC patents seem quite relevant, since it sounds as if any > > implementation of LZS would be presumed to infringe, and an unlisensed > > implementation would likely draw court orders. > > Buzzzt wrong! It is quite the opposite. STAC cannot patent that bit > encoding. It can be generated with an LZSS encoder, public domain code > for some ten years. That's an LZ77 characteristic. Many compatible > compressors/decompressors with different source. You missed my point, which was that anyone generating STAC's bit encoding without having lisensed STAC's code or chips would probably find it necessary to prove to a court or at least a lawyer they had not infringed STAC's patents. STAC would presume and get a court to presume infringement in the absense of clear evidence to the contrary, just like any other patent holder. This is the same case as v.42bis. (I presume you agree that a box that generated and parsed v.42bis using literal demonic magic would not require a v.42bis lisense, although it might be difficult to convince anyone that the box is compliant with the v.42bis standard or to sell it to most churches.) Ok, so please, let's talk about patents only when they are relevant to IETF standards. Please omit your patent objections for things like how one clears an arbitrary data structure, even if it is a compression hash table. Your repeatitions of the "it's all patented" song give aid and comfort to those who would destroy the essense computer programming, who would convert every programmer into a Standards Committee Goer or salescritter. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Wed Oct 27 20:12:08 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osNlz-00005na; Wed, 27 Oct 93 20:11 PDT Sender: owner-ppp-comp X-Path: mcimail.com!STAC/STAC/BobL%Stac_Electronics From: BobL To: ppp-comp Subject: Stac Questions Answered Date: Thu, 28 Oct 93 02:17 GMT Message-ID: <43931028021734/0005015674ND2EM@mcimail.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk I would like to clarify a few issues which have been raised concerning Stac Electronics and its LZS compression algorithm STANDARDS I have been told that some people have interpreted Stac's royalty free offer as a request to standardize on a SINGLE compression algorithm. This is not what I am trying to suggest. I support the concept that there should be hooks for other data compression algorithms to accommodate future data compression improvements and to support proprietary solutions. In fact, the Stacker LZS data compression standards that have been developed in ANSI, and in QIC have hooks for alternate compression algorithms. My suggestion would be to use a negotiating protocol (CCP) that supports multiple compression algorithms. But, to replace the Predictor recommended compression algorithm with the Stacker LZS format. I believe that if a GOOD data compression format is chosen as the recommended solution, then most implementations will use it. Manufacturers will not need to integrate dozens of compression algorithms, hoping that a match will be found with the equipment it is communicating with. But, this will still allow each manufacturer to implement whatever proprietary solutions they desire. Even Novell's technology agreement with Stac Electronics leaves the door open for other compression technologies. COMPRESSION RATIO I stated in my earlier email that Stacker LZS compresses about twice as well as Predictor. I have been asked to substantiate this statement. I have found this to be true for almost every file I try. I will bring a full spectrum of test results with me to the Houston meeting. I have not yet completed all the test runs. But, let me offer a couple of examples. These examples assume that the compression history is reset after each 16Kbytes: COMMAND.COM that is supplied with MS-DOS 6.0 Stacker LZS -> 1.38:1 to 1.50:1 Predictor -> 1.18:1 README.TXT that is supplied with MS-DOS 6.0 Stacker LZS -> 1.84:1 to 2.36:1 Predictor -> 1.66:1 Stacker LZS supports a "performance" parameter which accounts for the range of values listed. In both these examples, the potential network bandwidth gained by Stacker LZS can be double the bandwidth gained by predictor. COMPRESSION SPEED Using the same two example files on a 386DX @ 33MHz: COMMAND.COM Stacker LZS -> 134K to 81Kbytes/s Predictor -> 219Kbytes/s README.TXT Stacker LZS -> 173K to 75Kbytes/s Predictor -> 232Kbytes/s The C source code version of Stacker LZS and Predictor are used in the above examples. In both examples, Stacker LZS can perform at about 60% to 75% of the speed of Predictor and always produce superior compression ratios. On a 486DX2 @ 66MHz, the C source code can run over 800Kbytes/s and the hand assembly code version can run over 1Mbyte/s. Of course the hardware chips that are available run much faster. Up to 2.5Mbytes/s today, and up to 12Mbytes/s in a few months. LICENSE REQUIREMENTS I am not 100% up to speed on IETF and IAB procedures, but I do not believe that a license free technology is required for an IAB document. In fact, I don't think that even a royalty free license is required. Please correct me if I am incorrect. Anyway, I believe that the royalty free license is a very good offer. This can save each manufacturer a significant amount of money on development and royalties. In addition, it is to Stac's benefit to keep this license as simple as possible. Just as a single licensee does not want to spend money on legal fees, Stac cannot hire a lawyer for each FREE license that is requested. LIMITED LICENSE The royalty free license will be very short and straightforward. I am not skilled in interpreting licenses, but the following is my understanding of what it will be: It will allow the licensee to implement the Stacker LZS format using the techniques described in the patents that Stac holds for Internetworking applications. It will not allow for modifications to the compression format. I believe that the stability of the compression format is important. This will guarantee compatibility across application boundaries (DOS file system, LAN, etc.) and also ensure compatibility with the hardware solutions. There is no reason to tweak this format. If another compression format is developed in the future, then it can be implemented as defined in the CCP just like any other new compression scheme. TECHNICAL DOCUMENTATION I have received several requests for compression chip data sheets. I would be happy to mail data sheets to anybody who requests them. Each data sheet is over 60 pages, so I cannot fax or email them. If you would like documentation, please send me your name and address. GENERAL FEATURES I would like to summarize why I believe that the Stacker LZS format would be a better choice than the Predictor algorithm as the recommended (not the only) compression standard. - 2x better compression than Predictor. - Adjustable compression speed vs. compression ratio (requires no negotiation between sender and receiver). - Only 12Kbytes required for compression history and hash tables. - Supports multiple compression histories, or a single history. - C source compiles to only 3K object code (Intel). - History reset requires no memory initialization. - Memory never needs to be cleared or preset. - Sender may reset compression history at any time w/o notifying receiver. - No negotiable parameters required. - Hardware implementations available today. - Costs the same as Predictor, $0. - Already implemented in several standards. - Shipping products already using Stacker LZS. - Future support by Novell. Robert Lutz Stac Electronics Phone (619) 431-7474, FAX (619) 431-0947 Email:Stac/Stac/BobL%Stac_Electronics@mcimail.com From owner-ppp-comp Thu Oct 28 01:35:01 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osSnp-00003Ba; Thu, 28 Oct 93 01:33 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Thu, 28 Oct 1993 01:33:42 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Stac Questions Answered" on Oct 28, 2:17, BobL writes:] > COMPRESSION RATIO > > I stated in my earlier email that Stacker LZS compresses about twice as well > as Predictor. I have been asked to substantiate this statement. > > I have found this to be true for almost every file I try. I will bring a > full spectrum of test results with me to the Houston meeting. I have not > yet completed all the test runs. But, let me offer a couple of examples. > These examples assume that the compression history is reset after each > 16Kbytes: > > COMMAND.COM that is supplied with MS-DOS 6.0 > Stacker LZS -> 1.38:1 to 1.50:1 > Predictor -> 1.18:1 > > README.TXT that is supplied with MS-DOS 6.0 > Stacker LZS -> 1.84:1 to 2.36:1 > Predictor -> 1.66:1 These are similar issues to what we faced a while ago with the FZA algorithm, among others. Almost all algorithms will "do better" when confronted with a file. With PPP, we can't assume that the next packet will be part of the same file, nor even the same TYPE of data. This is where the "rand test file" came from. Single file compression just doesn't show what "real" (whatever that is) compression will be. Please try the tests on packetized data, preferably from multiple sources. A method for interleaving the data can be found in the comp.doc/comp.ps document available on sgi.com:other/ppp-comp, or you can invent your own method of interleaving the data from multiple files, if you prefer a non-IPX approach. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Thu Oct 28 07:44:39 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osYZJ-00003Ia; Thu, 28 Oct 93 07:43 PDT Sender: owner-ppp-comp X-Path: um.cc.umich.edu!bill.simpson From: "William Allen Simpson" To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Thu, 28 Oct 93 09:40:15 EDT Message-ID: <1599.bill.simpson@um.cc.umich.edu> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: BobL > I would like to clarify a few issues which have been raised concerning Stac > Electronics and its LZS compression algorithm > Excellent. Thank you. > In addition, it is to Stac's benefit to keep this license as simple as > possible. Just as a single licensee does not want to spend money on legal > fees, Stac cannot hire a lawyer for each FREE license that is requested. > Could you bring copies of your licensing agreement with you to IETF? When DEC's Tony Lauck did that at the Boston IETF in '92, it was wonderfully clarifying. Bill.Simpson@um.cc.umich.edu From owner-ppp-comp Thu Oct 28 07:53:31 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osYi2-00002Ra; Thu, 28 Oct 93 07:52 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Thu, 28 Oct 1993 07:52:10 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: Stac Questions Answered" on Oct 28, 1:33, Dave Rand writes:] > > These are similar issues to what we faced a while ago with the FZA > algorithm, among others. Dave just pointed out to me that this could be construed as criticism of his handling of the FZA issues. This was not my intent. When Dave made similar claims about FZA, he backed them up by creating a test setup so that he could run my test data through the FZA algorithm, and we have since published those numbers (in comp.doc/comp.ps). Compressing individual files, or even compressing a single file shipped with ftp (for example) will yeild much better compression ratios for almost all of the algorithms, but are not representative of "real life" PPP connections (at least for our products). When comparing compression algorithms and methods, we must all use some form of real-life data. My technique (explained in the comp.doc/ comp.ps) was to simulate multiple stations accessing multiple files, in an IPX environment. This is representative of what our users do with our PPP links. They don't just connect one workstation, and transfer one file. Using this test, which combines different types of data, as well as different sources/destinations, is as close as I could get to real life. I have, from time to time, tried to analyze the data that Dave Carr provided (Sniffer trace files), so that we could add some more data to the compression "suite". I have not completed this, yet. Based on my test data, the maximum compression ratio (2.774 with HPACKA) to the minimum compression ratio (1.118 with RLE) is a 2.5:1 ratio. Very few of the algorithms have "2 times better compression" than any of the other algorithms - in fact, RLE to FZA/Info-zip/HPACK are about the only ones. FZA is 1.767 to 1.488 times better than STAC. FZA is 1.615 times better than Predictor. Depending on the options, STAC is 1.086 times better than Predictor, or Predictor is 1.093 times better than STAC. Of course, "times better" refers only to compression ratio, and ignores speed. With hardware, STAC is many times better than either FZA or Predictor, or most any of the other algorithms. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Thu Oct 28 09:17:27 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osa23-00006ua; Thu, 28 Oct 93 09:17 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Thu, 28 Oct 93 10:16:06 -0600 Message-ID: <9310281616.AA15580@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > From: Robert Lutz > COMMAND.COM that is supplied with MS-DOS 6.0 > Stacker LZS -> 1.38:1 to 1.50:1 > Predictor -> 1.18:1 > > README.TXT that is supplied with MS-DOS 6.0 > Stacker LZS -> 1.84:1 to 2.36:1 > Predictor -> 1.66:1 Those ratios are not very different. UNIX-compress reduces the 6.0 command.com by 1.23, the readme.txt on the MSDOS 6.0 upgrade diskette by 2.52. Based on those ratios, it is clear that the right default compression algorighm is UNIX-compress. (Actually, it is clear that Predictor does well enough, and Predictor's completely free lisense for everyone makes it the only rational choice for everyone except those trying to get their hands into other people's pockets, unless you want to settle on run-length.) That readme.txt and command.com are very small files, only 61857 and 52925 bytes long, and far too short to allow LZW or Predictor to do as well as they will in real life. We are designing a network protocol here. Let's not worry much about streams of 50 or fewer packets. Command.com would be only 37 packets uncompressed. > ... > COMMAND.COM > Stacker LZS -> 134K to 81Kbytes/s > Predictor -> 219Kbytes/s > > README.TXT > Stacker LZS -> 173K to 75Kbytes/s > Predictor -> 232Kbytes/s Those numbers make LZS look too slow. I do not and cannot know about the LZS numbers, but a 386DX33 should do at least 2 or 3 times better than 200KByte/sec for Predictor. My bet is that most of those Predictor seconds are spent in whichever DOS file I/O package you prefer. In other words, I do not see enough signal in the noise to support the obvious conclusion that LZS is unacceptibly slow. > ... > I am not 100% up to speed on IETF and IAB procedures, but I do not believe > that a license free technology is required for an IAB document. In fact, I > don't think that even a royalty free license is required. Please correct me > if I am incorrect. Anyway, I believe that the royalty free license is a > very good offer. This can save each manufacturer a significant amount of > money on development and royalties. > > In addition, it is to Stac's benefit to keep this license as simple as > possible. Just as a single licensee does not want to spend money on legal > fees, Stac cannot hire a lawyer for each FREE license that is requested. You are wrong about the IEFT and IAB. You are doing as I said, and trying to protect STAC's business. Worse, you are trying to prevent genuinely cost free alternatives. > ... > The royalty free license will be very short and straightforward... That's what they all say. I have no doubt that your lisense is as short and simple as possible. It is also true that only an incompetent idiot anxious to be fired or sued for failing to exercise due diligence would sign it without spending significant money on lawyer time, and it is also likely that the lawyer of every potential signer with annual revenues of more than $1M will insist on "minor clarifications," and that negotiations over those minor modifiations will take weeks to months and require at least 10's and probably hundreds of hours of $100-$500/hour lawyer time. > ... > - Costs the same as Predictor, $0. That is either an intentionally false statement, commonly known as a lie, or you know absolutely nothing of real life lisensing and everything you say about your lisense must be ignored. Predictor costs $0. LZS costs between $5,000 and $30,000, although STAC gets none of the money. I do not believe you honestly think that signing any lisense is free, that anyone would sign any lisense without getting competent legal advice, or that you honestly think that such advice is free. For crying out loud! Just the cost of postage to get a copy of your lisence and then return it to you signed is not "$0". I am insensed that you would suggest replacing Predictor, a completely free algorithm, with yours, and demand that everyone in the world ask you for permisson to write compressed PPP code. However, the proprietery, costly LZS does not appear perform well enough to be a serious competator. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Thu Oct 28 09:47:30 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osaUb-00003Ia; Thu, 28 Oct 93 09:46 PDT Sender: owner-ppp-comp X-Path: dlr From: dlr (Dave Rand) To: ppp-comp@bungi.com Subject: IETF Date: Thu, 28 Oct 93 09:46 PDT Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk I will be arriving on Sunday evening in Houston, around 10pm. I'll be staying at the Sheraton. If anyone wants to get together, please leave a message for me there. Sorry for the late posting, but I (just now) confirmed my travel plans. From owner-ppp-comp Thu Oct 28 13:41:32 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ose9Q-0000TLa; Thu, 28 Oct 93 13:40 PDT Sender: owner-ppp-comp X-Path: hprnls7.rose.hp.com!jim From: Jim Petty To: ppp-comp@bungi.com Subject: HP PPC Date: Thu, 28 Oct 93 13:38:51 PDT Message-ID: <9310282038.AA23045@hprnls7.rose.hp.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk Yo Here is the HP PPC RFC. I was hoping the lawyers would be done by know but I thought I had better send it out before IETF. I think Vernon said that lawyers add 6 months to any agreement. This seems deragotory toward that class of people, and definitly not politically correct. Besides I don't think they add more than 5 months to any process. This is my take on the patent issue. Unisys owns the patent for the basic LZW algorithm, or at least 181 separate parts thereof (lawyerese?). Unisys sometimes contests the patent, I understand they went after Stacker. But in other cases they have not, namely UNIX compress. For a standards process we must be legal. What HP is attempting to do is basically what Dave Langley spoke about at the last IETF. HP will simplify the process for acquiring the patent rights for HP PPC. HP did this for another compression algorithm known as DCLZ. The way it works is HP tries to have a very simple contract (is that possible?), you pay HP some fee ($20,000 seems to be the going rate) and HP will handle the paper work and give the money to Unisys. If you already have an agreement with Unisys, HP will give away it's own patent rights on improvements and it is basically free (except for the lawyer fees). Of course, the preceding and what eventually happens are strictly coincidental. Jim -- *********************************************************************** "No matter where you go, there you are." Jim Petty (jpetty@hprnd.rose.hp.com) (916)785-3353 Fax: (916)786-9185 Hewlett-Packard (M/S R3NF3) 8000 Foothills Blvd Roseville, CA 95747 *********************************************************************** Network Working Group Jim Petty Internet Draft Hewlett-Packard expires in six months October 1993 PPP Hewlett-Packard Packet-by-Packet Compression (HP PPC) Protocol draft-ietf-pppext-hpppc-nn.txt Status of this Memo This document is the product of the Point-to-Point Protocol Working Group of the Internet Engineering Task Force (IETF). Comments should be submitted to the ietf-ppp@ucdavis.edu mailing list. Distribution of this memo is unlimited. This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a ``working draft'' or ``work in progress.'' Please check the 1id-abstracts.txt listing contained in the internet-drafts Shadow Directories on nic.ddn.mil, nnsc.nsf.net, nic.nordu.net, ftp.nisc.sri.com, or munnari.oz.au to learn the current status of any Internet Draft. Abstract The Point-to-Point Protocol (PPP) [1] provides a standard method for transporting multi-protocol datagrams over point-to-point links. The PPP Compression Control Protocol [2] provides a method to negotiate and utilize compression protocols over PPP encapsulated links. This document describes the use of the HP PPC compression algorithm for compressing PPP encapsulated packets. Petty expires in six months [Page i] DRAFT Hewlett-Packard PPC October 1993 1. Introduction The HP PPC compression algorithm is an improvement on the existing implementations of the LZ variant known as LZ2. In particular the scheme uses an intermediate dictionary initialization. This method allows for a dictionary with fewer entries than the alphabet size. Thus, compression is possible with very small dictionaries, requiring very little memory. This is particularly attractive for networks where each packet is compressed independently. Other enhancements include run length encoding and parameter initialization based on input packet size. Since this is a packet by packet scheme, only one compression dictionary is needed per link and no reliable link is required or necessary. If the compressed packet is larger than the input packet, the input packet is sent uncompressed. 1.1. Licensing This paragraph will contain some lawyer and management approved words. The contact person for evaluation under NDA and licensing is: Mary Ryan Hewlett-Packard MS R3NF3 8000 Foothills Blvd Roseville, CA 95747 (916)785-5744 Fax: (916)786-9185 EMail: Mary_Ryan@hp5200.desk.hp.com Petty expires in six months [Page 1] DRAFT Hewlett-Packard PPC October 1993 2. HP PPC Packets Before any HP PPC packets may be communicated, PPP must reach the Network-Layer Protocol phase, and the CCP Control Protocol must reach the Opened state. Exactly one HP PPC datagram is encapsulated in the PPP Information field, where the PPP Protocol field indicates type hex 00FD (compressed datagram). The maximum length of the HP PPC datagram transmitted over a PPP link is the same as the maximum length of the Information field of a PPP encapsulated packet. Reliability and Sequencing Each HP PPC packet is considered a separate entity. Therefore, the compression tables are reset for each packet. Reliable links are not necessary. Packets need not be delivered in sequence. Data Expansion Although the compression algorithm might occasionally expand a data packet, there is no expansion in HP PPC since such packets are sent uncompressed. Petty expires in six months [Page 2] DRAFT Hewlett-Packard PPC October 1993 2.1. Packet Format The encapsulation is the same for every packet. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PPP Protocol |C| Uncompressed Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Compressed Data ... +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ PPP Protocol The PPP Protocol field is described in the Point-to-Point Protocol Encapsulation [1]. When the HP PPC compression protocol is successfully negotiated by the PPP Compression Control Protocol [2], the value is 00FD hex. This value MAY be compressed when Protocol-Field-Compression is negotiated. C In the off chance that a packet expanded during compression, this bit is set. 0 = compressed 1 = uncompressed Uncompressed Length This is the length of the uncompressed data. It is used as a limit during the decompression process. 3. Configuration Option No configuration option is required. Petty expires in six months [Page 3] DRAFT Hewlett-Packard PPC October 1993 Security Considerations Security considerations are not discussed in this memo. References [1] Simpson, W.A., "The Point-to-Point Protocol (PPP)", work in progress. [2] Rand, D., "The PPP Compression Control Protocol (CCP)", work in progress. [3] Lempel, Abraham and Seroussi, Gadiel., "Compression Using Small Dictionaries with Applications to Network Packets", HP Technical Report HPL-92-112, Patent application, HP docket No. 1092545, 1993. Acknowledgments Original algorithm and mathematical advice provided by Gadiel Seroussi (HP). Initial testing of algorithm by Gadiel Seroussi and Dave Langley (HP). Bill Simpson provided the table of contents macros. Petty expires in six months [Page 4] DRAFT Hewlett-Packard PPC October 1993 Chair's Address The working group can be contacted via the current chair: Fred Baker Advanced Computer Communications 315 Bollay Drive Santa Barbara, California 93117 EMail: fbaker@acc.com Author's Address Questions about this memo can also be directed to: Jim Petty Hewlett-Packard Company 8000 Foothills Boulevard, MS R3NF3 Roseville, CA 95747 (916)785-5744 Fax: (916)786-9185 EMail: jpetty@hprnd.rose.hp.com Petty expires in six months [Page 5] DRAFT Hewlett-Packard PPC October 1993 TTTTaaaabbbblllleeee ooooffff CCCCoooonnnntttteeeennnnttttssss 1. Introduction .......................................... 1 1.1 Licensing ....................................... 1 2. HP PPC Packets ........................................ 2 2.1 Packet Format ................................... 3 3. Configuration Option .................................. 3 SECURITY CONSIDERATIONS ...................................... 4 REFERENCES ................................................... 4 ACKNOWLEDGEMENTS ............................................. 4 CHAIR'S ADDRESS .............................................. 5 AUTHOR'S ADDRESS ............................................. 5 From owner-ppp-comp Thu Oct 28 15:32:01 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osfsc-0000aea; Thu, 28 Oct 93 15:31 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: HP PPC Date: Thu, 28 Oct 93 16:30:31 -0600 Message-ID: <9310282230.AA18271@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Here is the HP PPC RFC. I was hoping the lawyers would be done by know > but I thought I had better send it out before IETF. I think Vernon said > that lawyers add 6 months to any agreement. This seems deragotory toward > that class of people, and definitly not politically correct. Besides > I don't think they add more than 5 months to any process. HP needs more creative lawyers, or perhaps more lawyers so they can devote more attention to each individual contract. SGI's never add less than 6 months. As far as I know, people inside SGI are still fighting about the 4.4BSD lisense more than a year after we decided to sign (for 4.4beta). The 4.3BSD lisense took only about a year. > This is my take on the patent issue. Unisys owns the patent for the basic > LZW algorithm, or at least 181 separate parts thereof (lawyerese?). > Unisys sometimes contests the patent, I understand they went after Stacker. > But in other cases they have not, namely UNIX compress. Could the difference between Stacker and UNIX compress be that one was given away for nothing and the other was sold for real money, and/or that one was pure software and the other involved hardware? > For a standards > process we must be legal. That's a good argument for Predictor, at least for the public domain world, the lingua franca, whatever gets put into the code they talk about in comp.protocols.ppp. > What HP is attempting to do is basically what Dave Langley spoke about at > the last IETF. HP will simplify the process for acquiring the patent > rights for HP PPC. HP did this for another compression algorithm known > as DCLZ. The way it works is HP tries to have a very simple contract > (is that possible?), you pay HP some fee ($20,000 seems to be the > going rate) and HP will handle the paper work and give the money to Unisys. > > If you already have an agreement with Unisys, HP will give away it's own > patent rights on improvements and it is basically free (except for the > lawyer fees). I don't object to LZS, as one of not-quite-free-but-less-than-$50,000- and-no-royalties alternatives. What is insulting is the claim that LZS is completely free, and so should replace Predictor. What are the performance numbers for the HP scheme? Is it the "HPACKA" with nice numbers in Dave Rand's table? At least part of that paper, including the table, should be published as an informational RFC. It would be interesting to compare the performance of current LZS algorithms with the values reported in that table. Run length patented in 1986--sheesh, what nonsense. In 1979 I was shipping significant volumes of a small business system with a file backup subsystem using run-length coding, based on run-length ideas I'd picked up from a popular time sharing system (Berkeley's Project Genie, later SDS and then Xerox) of the mid-1960's. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Fri Oct 29 06:18:12 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osthw-0000XMa; Fri, 29 Oct 93 06:17 PDT Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: HP PPC Date: Fri, 29 Oct 1993 09:15:12 -0400 (EDT) Message-ID: <9310291315.AA08916@donut.gandalf.ca> References: <<9310282230.AA18271@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > What are the performance numbers for the HP scheme? Is it the "HPACKA" > with nice numbers in Dave Rand's table? They should be so lucky. No, HPACK (I don't know why there is an A after it), is a portable archiver written by Peter Gutmann. See comp.compression FAQ for obtaining the source. As noted in the Rand test, it is an LZA derivative. Slow as hell, but always seems to come out on top (compression ratio only) in testing against ZIP, ARJ, Freeze,... There is a more recent version out too, slightly better than 0.75. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Oct 29 08:45:41 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osw0n-0000Awa; Fri, 29 Oct 93 08:45 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: HP PPC Date: Fri, 29 Oct 93 09:44:24 -0600 Message-ID: <9310291544.AA23893@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk I'm serious about publishing at least that table as an RFC. It looks like it would make STAC go away, at least until they have something interesting. Other parts of the document are a little dated, but the table will be relevant for a long time. vjs From owner-ppp-comp Fri Oct 29 10:00:33 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0osx9t-0000W8a; Fri, 29 Oct 93 09:58 PDT Sender: owner-ppp-comp X-Path: hprnls7.rose.hp.com!jim From: Jim Petty To: ppp-comp@bungi.com Subject: Re: HP PPC Date: Fri, 29 Oct 93 9:56:38 PDT Message-ID: <9310291656.AA23997@hprnls7.rose.hp.com> References: <<9310282230.AA18271@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk Yo again > > HP needs more creative lawyers, > Did someone say we needed more lawyers, creative or otherwise? > > Could the difference between Stacker and UNIX compress be that one was > given away for nothing and the other was sold for real money, and/or that > one was pure software and the other involved hardware? > Probably logistical. > > What are the performance numbers for the HP scheme? Is it the "HPACKA" > with nice numbers in Dave Rand's table? No, I think that is 'H PACK A'. The numbers I have would be comparing apples to oranges. But..... We selected this algorithm because it will work on any link, uses very little memory, and runs fast enough for our low end routers. Because it is packet-by-packet, in any mix of data packets, each packeti, if the same packet came through again, would compress exactly the same, everytime. On the downside the compression ratio will not be as good as any running dictionary type. The compression ratio is directly connected to packet size. The larger the packet the better the ratio, assuming the data is compressible. I haven't run Dave Rand's test on this, but I believe the compression ratio would be similar to Predictor, take more CPU, and less memory. There has been a lot of banter lately about picking one compression algorithm as the defacto standard. I believe it is situational. If you have clean dedicated lines at 56K, use a good running dictionary type algorithm (Stacker?). If you have PPP over X.25 where separate dictionaries for each circuit is prohibitive, a packet-by-packet scheme would be useful (HP PPC?). If you are running at 512K and doing compression in software, a fast scheme would be great (Predictor?). > > At least part of that paper, including the table, should be published > as an informational RFC. Gee, Dave, another RFC. > Run length patented in 1986--sheesh, what nonsense. In 1979 I was > shipping significant volumes of a small business system with a file > backup subsystem using run-length coding, based on run-length ideas I'd > picked up from a popular time sharing system (Berkeley's Project Genie, > later SDS and then Xerox) of the mid-1960's. If I wrote anything that hinted that HP had patented run-length, I couldn't find it. I was just trying to describe the algorithm. > > Vernon Schryver, vjs@sgi.com Jim -- *********************************************************************** "No matter where you go, there you are." Jim Petty (jpetty@hprnd.rose.hp.com) (916)785-3353 Fax: (916)786-9185 Hewlett-Packard (M/S R3NF1) 8000 Foothills Blvd Roseville, CA 95747 *********************************************************************** From owner-ppp-comp Fri Oct 29 13:21:49 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ot0KD-0000Sea; Fri, 29 Oct 93 13:21 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: HP PPC Date: Fri, 29 Oct 93 14:20:39 -0600 Message-ID: <9310292020.AA25472@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Jim Petty (jpetty@hprnd.rose.hp.com) > > Run length patented in 1986--sheesh, what nonsense.... > If I wrote anything that hinted that HP had patented run-length, I couldn't > find it. I was just trying to describe the algorithm. No, sorry. I skimmed Dave Rand's paper hoping for performance numbers for yours, and encountered the reference to 1986 run-length patents. Run length in 1986! It would take a lot of something or other to look at that list of stupidity, greed, and attempted frauds, and then willing admit ever been involved in the filing of any patent, even a worthy one. (I'm inclined to think there was something genuinely novel and worthy somewhere in LZ and Hufmann encoding, although the flood of bogus claims over the years have obscured exactly what it was, at least for me.) vjs From owner-ppp-comp Fri Oct 29 14:52:19 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ot1jh-0000a8a; Fri, 29 Oct 93 14:51 PDT Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: CCP transmission Date: Fri, 29 Oct 93 15:35:40 -0600 Message-ID: <9310292135.AA00857@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk What is the receiver supposed to do with its transmissions when it "drops out of open" because it has decided the receive compression path has lost synchronization? The only thing I can see to do is to stop transmitting compressed packets as well as send the Configure-Request and start ignoring received compressed packets. Then wait for the peer to renegotiate what it wants to receive as well as answer your own demand for what you want to receive. It's too bad there are not two CCP protocols or state machines for unreliable links, one for the transmit path and one for the receive path. It's too bad because one of the nice things about using unreliable links even for compression is that the transmit and receive paths are mostly decoupled. Oh, well, I guess the basic assumption with compression with unreliable links that involve inter-packet state is that synchronization is hardly ever lost. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Fri Oct 29 19:01:52 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ot5cs-00003Ka; Fri, 29 Oct 93 19:01 PDT Sender: owner-ppp-comp X-Path: mcimail.com!STAC/STAC/BobL%Stac_Electronics From: BobL To: ppp-comp Subject: Re: Stac Questions Answered Date: Sat, 30 Oct 93 01:24 GMT Message-ID: <43931030012434/0005015674ND4EM@mcimail.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk In response to email from Vernon Schryver - >> COMMAND.COM that is supplied with MS-DOS 6.0 >> Stacker LZS -> 1.38:1 to 1.50:1 >> Predictor -> 1.18:1 >> >> README.TXT that is supplied with MS-DOS 6.0 >> Stacker LZS -> 1.84:1 to 2.36:1 >> Predictor -> 1.66:1 > >Those ratios are not very different. I disagree. The compression ratio's listed above are quite different. For instance, let's assume that a 1Mbyte file transfer (of a type similar to COMMAND.COM) requires about 140 seconds to transfer on a 56Kb/s line. (This ignores misc. overhead, but it is simply a baseline). The time required to transfer this file with various compression algorithms are: None -> 140 seconds Predictor -> 118 seconds LZS -> 93 seconds As you can see, LZS can reduce an additional 25 seconds off the original transfer time for a total savings of 47 seconds. 25 seconds additional savings is even more than the total savings of Predictor's 22 seconds. If you are claiming that 25 seconds is insignificant, then you must believe that the Predictor algorithm is completely useless. If you don't believe that Predictor's compression performance is beneficial, then why are you arguing to keep it as the recommended standard? >That readme.txt and command.com are very small files, only 61857 and >52925 bytes long, and far too short to allow LZW or Predictor to do as >well as they will in real life. We are designing a network protocol >here. Let's not worry much about streams of 50 or fewer packets. >Command.com would be only 37 packets uncompressed. I guess this comes down to the question of what is "real life". Dave Rand has explained to me that he has created an artificially stream of possible Internetworking traffic by breaking up several files into smaller chunks and streaming it into a single file. As soon as I gain access to this test file, I will run the compression algorithms through it. I see three different ways data may be transferred through a compressing router. 1) All data will pass through a single compression dictionary. The dictionary will be reset only if an error is detected or optionally, if data expansion occurs on a block of data. This solution requires only a single dictionary, and minimal logic for compression. The compression ratio will be mediocre because each time the data stream switches between virtual links, the data type will change, and the current state of the dictionary will be unrelated to the new data stream. An advantage of this model is that if only a single virtual link is active, the compression ratio will be at its maximum because the dictionary will not be reset, and the data type remains the same. However, as soon as additional virtual links are created, the compression ratio begins to fall. This appears to be the model that Dave Rand's tests simulate. 2) All data will pass through a single compression dictionary. The dictionary will be reset automatically after every "n" bytes. Different source-destination pairs will be switched in these "n" byte intervals. This solution also requires only a single dictionary, and minimal logic for compression. The compression ratio will be slightly worse than the model #1 assuming that the block sizes are about the same. An advantage of this model over model #1 is that there needs to be no special handshaking for a dictionary reset. I have been simulating this model with "n" set at 16Kbytes. 3) Each virtual link is assigned its own compression dictionary. Each dictionary is reset only if an error is detected, or optionally if data expansion occurs on a block of data. This solution requires multiple compression dictionaries and sophisticated logic to maintain the relationships between separate compression dictionaries and virtual links. This model will always produce the best compression ratio compared to the other two models because each dictionary is dedicated to the data type of each virtual link. This model is simulated by simply compressing single files, resetting the compression dictionary at the end of each file. I don't know which of these models will be used by Internetworking manufacturers. But we should probably be thinking about all of these unless it is clear which model will be the only model used. There may be other working models that I am not thinking of as well. As another data point, I found a file labeled novell.enc.z at the same location that our archived ppp-comp emails are kept. It appears to be the Sniffer Trace File that Dave Rand mentioned in an earlier email. This file is quite large (924149 bytes). The results I got for this file (without ever resetting the dictionary) is: Stacker LZS -> 2.11:1 Predictor -> 1.53:1 This file is plenty large to allow Predictor's hash table to be fully utilized. And still the Stacker LZS algorithm compresses significantly better. >> ... >> COMMAND.COM >> Stacker LZS -> 134K to 81Kbytes/s >> Predictor -> 219Kbytes/s >> >> README.TXT >> Stacker LZS -> 173K to 75Kbytes/s >> Predictor -> 232Kbytes/s > >Those numbers make LZS look too slow. I do not and cannot know about >the LZS numbers, but a 386DX33 should do at least 2 or 3 times better >than 200KByte/sec for Predictor. My bet is that most of those >Predictor seconds are spent in whichever DOS file I/O package you >prefer. >In other words, I do not see enough signal in the noise to support >the obvious conclusion that LZS is unacceptibly slow. The evaluation program that I wrote was from the C source code that was provided in the CCP description and the C source code of the Stacker LZS algorithm. I used the same C compiler (Microsoft) for both with the same optimization options. There is no DOS overhead in my results. My speed test program reads in the first 16Kbytes of a file and compresses this 16K buffer over and over again for 10 seconds. It then calculates the throughput based on the number of bytes compressed. No file I/O is performed at all during the speed test. One reason why Predictor may be slower than you expected is because I reset the hash table every 16Kbytes in this particular test run. This requires clearing the entire 64Kbyte for each block. If I don't reset the dictionary, Predictor runs at 260Kbytes/s in this example. >> ... >> I am not 100% up to speed on IETF and IAB procedures, but I do not believe >> that a license free technology is required for an IAB document. In fact, I >> don't think that even a royalty free license is required. Please correct me >> if I am incorrect. Anyway, I believe that the royalty free license is a >> very good offer. This can save each manufacturer a significant amount of >> money on development and royalties. >> >> In addition, it is to Stac's benefit to keep this license as simple as >> possible. Just as a single licensee does not want to spend money on legal >> fees, Stac cannot hire a lawyer for each FREE license that is requested. > >You are wrong about the IEFT and IAB. >You are doing as I said, and trying to protect STAC's business. >Worse, you are trying to prevent genuinely cost free alternatives. I admit that I am new to the policies of the Internet Standards Process. But I got my information from reading RFC1310, "The Internet Standards Process". The latest revision of this document dated September 1993 states the following: 5.4.2. Standards Track Documents (A) ISOC will not propose, adopt, or continue to maintain any standards, including but not limited to standards labelled Proposed, Draft or Internet Standards, which can only be practiced using technology or works that are subject to known copyrights, patents or patent applications, or other rights, except with the prior written assurance of the owner of rights that: l. ISOC may, without cost, freely implement and use the technology or works in its standards work; 2. upon adoption and during maintenance of an Internet Standard, any party will be able to obtain the right to implement and use the technology or works under specified, reasonable, non-discriminatory terms; and 3. the party giving the assurance has the right and power to grant the licenses and knows of no other copyrights, patents, patent applications, or other rights that may prevent ISOC and members of the Internet community from implementing and operating under the standard. Based on this official procedure, I am not sure why you think that I am wrong about this. Can you please explain further? Also, just because Stac Electronics does business by selling data compression chips, does not mean that this is my sole motivation. I am attempting to offer a superior technology to improve the performance of Internetworking products that adhere to the standards that we are creating. I thought that my inputs would be appreciated, and that the software technology that Stac is offering at no cost would be seriously considered. >> ... >> - Costs the same as Predictor, $0. > >That is either an intentionally false statement, commonly known as a >lie, or you know absolutely nothing of real life lisensing and >everything you say about your lisense must be ignored. > >Predictor costs $0. LZS costs between $5,000 and $30,000, although >STAC gets none of the money. I do not believe you honestly think that >signing any lisense is free, that anyone would sign any lisense without >getting competent legal advice, or that you honestly think that such >advice is free. > >For crying out loud! Just the cost of postage to get a copy of your >lisence and then return it to you signed is not "$0". Hmmm. Let me just mention that based on your argument, the use of the Predictor algorithm may not be entirely free either. Wouldn't you need to verify the patent free status of the Predictor algorithm? >I am insensed that you would suggest replacing Predictor, a completely >free algorithm, with yours, and demand that everyone in the world ask >you for permisson to write compressed PPP code. > >However, the proprietery, costly LZS does not appear perform well >enough to be a serious competator. > > >Vernon Schryver, vjs@sgi.com Vernon, I hope that I have addressed most of your issues. I am simply attempting to offer a technically superior compression algorithm. Also, I believe that it is important to review all the pros and cons of each algorithm to determine which would be the best recommended solution. We shouldn't discard a potential solution based on a single issues. Robert Lutz Stac Electronics Phone (619) 431-7474, FAX (619) 431-0947 Email:Stac/Stac/BobL%Stac_Electronics@mcimail.com From owner-ppp-comp Mon Nov 1 09:23:40 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ou2tj-00007Va; Mon, 1 Nov 93 09:18 PST Sender: owner-ppp-comp X-Path: zircon.acc.com!rms From: rms@zircon.acc.com (Ron Stoughton) To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Mon, 1 Nov 93 12:18:23 EST Message-ID: <9311011718.AA02336@zircon.acc.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > >> ... > >> COMMAND.COM > >> Stacker LZS -> 134K to 81Kbytes/s > >> Predictor -> 219Kbytes/s > >> > >> README.TXT > >> Stacker LZS -> 173K to 75Kbytes/s > >> Predictor -> 232Kbytes/s > > There is no DOS overhead in my results. My speed test program reads in the > first 16Kbytes of a file and compresses this 16K buffer over and over again > for 10 seconds. It then calculates the throughput based on the number of > bytes compressed. No file I/O is performed at all during the speed test. There has been a lot of discussion about the speed of one algorithm versus another, and these comparisons seem to be focused on the speed at which a particular algorithm can generate compressed output. Standing alone, this is not a meaningful measure of the algorithm's performance. For example, if I had an algorithm which compressed 1,000,000:1, I might well expect the output to be low. The throughput as measured at the output of the compression engine tells me the maximum speed of a transmission line I can support before the effectiveness is diminished because I cannot keep it full. A better measure would be the EFFECTIVE throughput which is the output rate times the compression ratio. rms@acc.com From owner-ppp-comp Mon Nov 1 15:55:23 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ou94d-0000T8a; Mon, 1 Nov 93 15:54 PST Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: Stac Questions Answered Date: Mon, 1 Nov 1993 15:54:05 PST Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: Stac Questions Answered" on Nov 1, 12:18, Ron Stoughton writes:] > output to be low. The throughput as measured at the output of the compression > engine tells me the maximum speed of a transmission line I can support before > the effectiveness is diminished because I cannot keep it full. A better > measure would be the EFFECTIVE throughput which is the output rate times > the compression ratio. > > rms@acc.com While I don't do this exactly, you might want to check out the RGF in the comp.doc/comp.ps document on sgi.com:other/ppp-comp -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Mon Nov 1 16:03:34 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ou9CP-0000OKa; Mon, 1 Nov 93 16:02 PST Sender: owner-ppp-comp X-Path: dlr From: dlr (Dave Rand) To: ppp-comp@bungi.com Subject: Those of you at IETF Date: Mon, 1 Nov 93 16:01 PST Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk There will be a data compression dinner tommorow evening, place TBA. Check the bulletin board, or call for more info (I'm in room 2111). Dave Rand From owner-ppp-comp Mon Nov 1 22:04:28 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ouEpi-0000Fia; Mon, 1 Nov 93 22:03 PST Sender: owner-ppp-comp X-Path: mcimail.com!STAC/STAC/BobL%Stac_Electronics From: BobL To: ppp comp Subject: Re: Stac Questions Answered Date: Mon, 1 Nov 93 23:50 GMT Message-ID: <42931101235024/0005015674ND3EM@mcimail.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk >There has been a lot of discussion about the speed of one algorithm versus >another, and these comparisons seem to be focused on the speed at which a >particular algorithm can generate compressed output. Standing alone, this >is not a meaningful measure of the algorithm's performance. For example, >if I had an algorithm which compressed 1,000,000:1, I might well expect the >output to be low. The throughput as measured at the output of the compression >engine tells me the maximum speed of a transmission line I can support before >the effectiveness is diminished because I cannot keep it full. A better >measure would be the EFFECTIVE throughput which is the output rate times >the compression ratio. Ron, The speed measurements that I quote are always measured by the raw data throughput. During compression, this is the input throughput. During decompression, this is the output throughput. I believe that this is the measurement that you are asking for. Of course, with a known compression ratio, you can easily calculate the compressed data speed. Robert Lutz Stac Electronics Phone (619) 431-7474, FAX (619) 431-0947 Email:Stac/Stac/BobL%Stac_Electronics@mcimail.com From owner-ppp-comp Tue Nov 2 16:27:20 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ouW3Z-000088a; Tue, 2 Nov 93 16:26 PST Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: UNIX-compress memory requirements Date: Tue, 2 Nov 93 17:25:42 -0700 Message-ID: <9311030025.AA03432@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk If I finally understand the UNIX-compress code, the memory requirements for a receiver with hash table (the worst case) after obvious optimizations are less than N*7 bytes, where N is about 1.1*(2**B), where B=number of bits in the code. If your CPU does not have fancy X*C+Y addressing modes as the 80386 does, or if alignment is a consideration, it seems reasonable to use N*8 bytes. Using the prime numbers that make the the secondary probing in the standard code work, N is 5003 for B=12, 9001 for 13, and 18013 for B=14. That means that 12-bit LZW needs only 40KBytes of data on the receiver. Furthermore, the adaptive dictionary clearing of UNIX-compress requires only that N*2 continguous bytes be set to a recognizable value (after an obvious re-arrangement of the hash table.) I cannot see how infrequently setting 2K 32-bit store operations can matter on a reasonable system, given 1500 byte packets that require byte-at-a-time fiddling to mention perform the compression. The RFC will have to document the special action required of the receiver at packet-end to avoid spuriously assigning a code. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Fri Nov 5 09:24:50 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovUtL-0000g8a; Fri, 5 Nov 93 09:24 PST Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: clearing hash tables Date: Fri, 5 Nov 93 09:26:07 -0700 Message-ID: <9311051626.AA26351@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk I'm wrong about the cost of clearing the UNIX-compress dictionary. I think I have a scheme that costs 2 to 6 MIPS R3000 instructions per probe (almost all of the time 2 instructions) and absolutely zero when you need to clear the hash table. It uses only the obvious 7 or 8 bytes/code needed by the receiver that is also running the hash table. The idea is a simple consistency check on the hash table during probing. As far as patent nonsense goes, lisense leeches and their prey should consider the enclosed article from comp.arch, and especially the date of that well known and frequently used textbook. Vernon Schryver, vjs@sgi.com ]Newsgroups: comp.arch ]From: preston@dawn.cs.rice.edu (Preston Briggs) ]Subject: Re: Zeroing Memory in Constant Time ]Message-ID: ]Sender: news@rice.edu (News) ]Organization: Rice University, Houston ]References: <2bb5b4$pmj@avignon.daimi.aau.dk> ]Date: Fri, 5 Nov 1993 05:03:18 GMT ]Lines: 39 ] ]>edlee@chinet.chinet.com (Edward Lee) writes: ]>>The initialization of arbitrary amounts of memory cannot be ]>>"algorithmically" done (by software) in constant time if the hardware does ]>>not support this feature. If you meant something else, please clarify. ] ]and tthorn@daimi.aau.dk (Tommy Thorn) writes: ]>Ever heard of "Smart" arrays? For a reference, check Preston Briggs ]>"Register Allocation via Graph Coloring" and check for the ]>implementation of sparse sets. ] ]Someone actually noticed that section? Amazing... ] ]The idea is based on a homework problem from an algorithms class. ]Problem 2.12 in ] ] title="The Design and Analysis of Computer Algorithms", ] author="Alfred V. Aho and John E. Hopcroft and Jeffrey D. Ullman", ] publisher="Addison-Wesley", ] address="Reading, Massachusetts", ] year=1974 ] ]I won't explain it here, but they show how we can, in many cases, ]effectively initialize a data structure in constant time. ] ]Or consider a hash table. A hash table allows us to maintain a sparse ]array of any dimensionality without need to initialize (or even ]allocate) the entire array. We need to initialize the hash table and ]expand it when it overflows, but that can be accomplished in O(n log ]n) time, where n is the number of elements in the table, not the ]potential size of the whole array. ] ]Or a tree structure, where you can reinitialize it by simply clearing ]the root pointer and reclaim the storage using a mark/release protocol, ]both in constant time. ] ]Of course, none of these approaches are zeroing memory; however, they ]are all algorithmic alternatives that avoid the need to zero memory. ] ]Preston Briggs From owner-ppp-comp Fri Nov 5 09:52:57 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovVKY-00006Ha; Fri, 5 Nov 93 09:52 PST Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: clearing hash tables Date: Fri, 5 Nov 1993 12:49:33 -0500 (EST) Message-ID: <9311051749.AA10672@donut.gandalf.ca> References: <<9311051626.AA26351@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > I think I have a scheme that costs 2 to 6 MIPS R3000 instructions per > probe (almost all of the time 2 instructions) and absolutely zero when > you need to clear the hash table. It uses only the obvious 7 or 8 > bytes/code needed by the receiver that is also running the hash table. > The idea is a simple consistency check on the hash table during > probing. Do you mean crosschecking that the hash actually matches the data now in the hash table, and is not an old entry that wasn't deleted? Alternately, one could just try to optimize the clearing of the tables. One could clear the entire table in a couple of writes if you kept the tables in Video-DRAM (VRAM), less than a microsecond with RAMBus DRAMS, ... Of course, the display might look a little wierd when the compressor was running :-) -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Nov 5 10:12:08 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovVcf-0000c5a; Fri, 5 Nov 93 10:10 PST Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: List changes Date: Fri, 5 Nov 1993 10:10:39 PST Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk Effective following this message, the ppp-comp mailing list will be gatewayed to the mail ietf-ppp mailing list. If you are on the main list, you don't need to be on this one any more. Send your removal request to: ppp-comp-request@bungi.com and I'll be happy to take you off. It was good to meet most of you in person at Houston, and I'm looking forward to releasing the LAPB and compression documents next week. -- Dave Rand Internet: dlr@daver.bungi.com From owner-ppp-comp Fri Nov 5 10:54:00 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovWGl-0000kVa; Fri, 5 Nov 93 10:52 PST Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: clearing hash tables Date: Fri, 5 Nov 93 11:50:44 -0700 Message-ID: <9311051850.AA27385@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > > I think I have a scheme that costs 2 to 6 MIPS R3000 instructions per > > probe (almost all of the time 2 instructions) and absolutely zero when > > you need to clear the hash table. It uses only the obvious 7 or 8 > > bytes/code needed by the receiver that is also running the hash table. > > The idea is a simple consistency check on the hash table during > > probing. > > Do you mean crosschecking that the hash actually matches the data now > in the hash table, and is not an old entry that wasn't deleted? > > Alternately, one could just try to optimize the clearing of the tables. > One could clear the entire table in a couple of writes if you kept the > tables in Video-DRAM (VRAM), less than a microsecond with RAMBus DRAMS, > ... > > Of course, the display might look a little wierd when the compressor > was running :-) > -- > Dave Carr | dcarr@gandalf.ca | It's what you learn, I mean a pure software solution. >From your description, you notion and mine of the UNIX-compress hash table are not exactly the same, so that your description does not exactly fit what I'm talking about. Still, you obviously have the idea. It's handy that codes are always placed in the hash table starting at 257 with no gaps. If you could build a hash table with a load-factor of 1.0000, then you could clear the table even faster. vjs From owner-ppp-comp Fri Nov 5 11:13:56 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovWZ0-0000Mta; Fri, 5 Nov 93 11:11 PST Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: IETF Request/comp.compression FAQ Date: Fri, 5 Nov 1993 13:59:17 -0500 (EST) Message-ID: <9311051859.AA10775@donut.gandalf.ca> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk At the IETF, several folks expressed interest in finding public domain compressor algorithms, the Calgary Corpus, patents, etc. Here is the comp.compression FAQ file. Happy compressing :-) --- >From comp.compression Mon Aug 16 15:59:04 1993 Xref: hobbit.gandalf.ca comp.compression:7991 comp.compression.research:839 news.answers:10827 comp.answers:1539 Path: hobbit.gandalf.ca!nott!torn!howland.reston.ans.net!math.ohio-state.edu!magnus.acs.ohio-state.edu!usenet.ins.cwru.edu!agate!doc.ic.ac.uk!uknet!mcsun!julienas!chorus!chorus.fr From: jloup@chorus.fr (Jean-loup Gailly) Newsgroups: comp.compression,comp.compression.research,news.answers,comp.answers Subject: comp.compression Frequently Asked Questions (part 1/3) Summary: *** READ THIS BEFORE POSTING *** Keywords: data compression, FAQ Message-ID: Date: 10 Aug 93 20:55:05 GMT Expires: 30 Sep 93 16:17:20 GMT Sender: news@chorus.chorus.fr Reply-To: jloup@chorus.fr Followup-To: comp.compression Lines: 2497 Approved: news-answers-request@MIT.Edu Supersedes: Archive-name: compression-faq/part1 Last-modified: Aug 10th, 1993 "I've already explained this once, but repetition is the very soul of the net." (from alt.config) This file is part 1 of a set of Frequently Asked Questions (FAQ) for the groups comp.compression and comp.compression.research. (If you can't find part 2 or 3, see item 53 below.) Certain questions get asked time and again, and this is an attempt to reduce the bandwidth taken up by these posts and their associated replies. If you have a question, *please* check this file before you post. It may save a lot of peoples time. If you have not already read the overall Usenet introductory material posted to "news.announce.newusers", please do. If you don't want to see this FAQ regularly, please add the subject line to your kill file. If you have corrections or suggestions for this FAQ, send them to Jean-loup Gailly . Thank you. Part 1 is oriented towards practical usage of compression programs. Part 2 is more intended for people who want to know how compression works. Part 3 is a long list of image compression hardware. Main changes relative to the previous version: - added .sdn extension (item 2) - new versions of arj and mcvert (item 2) - fixed reference to ftp site for the Calgary corpus (item 5) - add warning about the MINC compressor (item 9) - new version of the havefun.stanford jpeg/mpeg/p64 programs (item 15) - added pointer to new mpeg encoder (item 15) - added discussion on IFS vs. Fractal Transform (item 17) - new references on fractal compression (item 17) - added ftp site for celp software (item 26) - several corrections on image compression products or companies (item 85) Contents ======== General questions: [1] What are these newsgroups about? [2] What is this .xxx file type? Where can I find the corresponding compression program? [3] What is the latest pkzip version? [4] What is an archiver? [5] What is the best general purpose compression program? [7] Which books should I read? [8] What about patents on data compression algorithms? [9] The WEB 16:1 compressor. [11] What is the V.42bis standard? [12] I need source for the winners of the Dr Dobbs compression contest [13] I need source for arithmetic coding Image and audio compression: [15] Where can I get image compression programs? [16] What is the state of the art in lossless image compression? [17] What is the state of fractal compression? [18] I need specs and source for TIFF and CCITT group 4 Fax. [19] What is JPEG? [20] I am looking for source of an H.261 codec and MPEG [25] Fast DCT (Discrete Cosine Transform) algorithms [26] Are there algorithms and standards for audio compression? Common problems: [30] My archive is corrupted! [31] pkunzip reports a CRC error! [32] VMS zip is not compatible with pkzip! [33] I have a problem with Stacker! Questions which do not really belong to comp.compression: [50] What is this 'tar' compression program? [51] I need a CRC algorithm [52] What about those people who continue to ask frequently asked questions? [53] Where are FAQ lists archived? [54] I need specs for graphics formats [55] Where can I find Lenna and other images? [56] I am looking for a message digest algorithm Part 2: (Long) introductions to data compression techniques [70] Introduction to data compression (long) Huffman and Related Compression Techniques Arithmetic Coding Substitutional Compressors The LZ78 family of compressors The LZ77 family of compressors [71] Introduction to MPEG (long) What is MPEG? Does it have anything to do with JPEG? Then what's JBIG and MHEG? What has MPEG accomplished? So how does MPEG I work? What about the audio compression? So how much does it compress? What's phase II? When will all this be finished? How do I join MPEG? How do I get the documents, like the MPEG I draft? [72] What is wavelet theory? [73] What is the theoretical compression limit? [74] Introduction to JBIG [75] Introduction to JPEG Part 3: (Long) list of image compression hardware [85] Image compression hardware [99] Acknowledgments Search for "Subject: [#]" to get to question number # quickly. Some news readers can also take advantage of the message digest format used here. If you know very little about data compression, read question 70 in part 2 first. ------------------------------------------------------------------------------ ~Subject: [1] What are these newsgroups about? comp.compression is the place to discuss about data compression, both lossless (for text or data) and lossy (for images, sound, etc..). comp.compression.research was created later to provide a forum for current research on data compression and data compression algorithms. If you are not experienced in data compression, please post in comp.compression only. If you only want to find a particular compression program for a particular operating system, please read first this FAQ and the article "How to find sources" which is regularly posted in news.answers. If you can't resist posting such a request, other groups are probably more appropriate (comp.binaries.ibm.pc.wanted, comp.sources.wanted, comp.sys.mac.wanted, comp.archives.msdos.d, alt.graphics.pixutils). Please post your request in comp.compression only as a last resource. If your question is about graphics only (no compression), please post to comp.graphics, *after* reading the comp.graphics FAQ (see item 54 below). For some unknown reason, many questions about graphics are incorrectly posted to comp.compression. Please do not post any program in binary form to comp.compression. Very short sources can be posted, but long sources should be be posted to the specialized source groups, such as comp.sources.* or alt.sources. As for any newsgroups, do not post the same message separately to comp.compression and comp.compression.research. ------------------------------------------------------------------------------ ~Subject: [2] What is this .xxx file type? Where can I find the corresponding compression program? All the programs mentioned in this section are lossless. For most programs, one US and one European ftp site are given. (wuarchive.wustl.edu: 128.152.135.4, garbo.uwasa.fi: 128.214.87.1) Many other sites (in particular wsmr-simtel20.army.mil: 192.88.110.2) have the same programs. To keep this list to a reasonable size, many programs are not mentioned here. Additional information can be found in the file ftp.cso.uiuc.edu:/doc/pcnet/compression [128.174.5.61] maintained by David Lemson (lemson@uiuc.edu). When several programs can handle the same archive format, only one of them is given. Sources for additional lossless data compressors can be found in garbo.uwasa.fi:/pc/programming/lds_10.zip and oak.oakland.edu:/pub/msdos/archivers/lz-comp2.zip. Sources in Pascal are in garbo.uwasa.fi:/pc/turbopas/preskit2.zip. For Macintosh programs, look on sumex-aim.stanford.edu:/info-mac [36.44.0.6]. For VM/CMS, look on vmd.cso.uiuc.edu:/public.477 [128.174.5.98]. For Atari, look on atari.archive.umich.edu [141.211.165.41] For Amiga, look on ftp.cso.uiuc.edu:/pub/amiga [128.174.5.59] If you don't know how to use ftp or don't have ftp access, read the article "How to find sources" which is regularly posted in news.answers. If you can't find a program given below, it is likely that a newer version exists in the same directory. (Tell me ) A very short description of the compression algorithm is given for most programs. For the meaning of LZ77, LZ78 and LZW, see question 70 in part 2 of the FAQ.) ext: produced by or read by .arc: arc, pkarc for MSDOS. (LZW algorithm) wuarchive.wustl.edu:/mirrors/msdos/starter/pk361.exe garbo.uwasa.fi:/pc/arcers/pk361.exe arc for Unix wuarchive.wustl.edu:/mirrors/misc/unix/arc521e.tar-z garbo.uwasa.fi:/unix/arcers/arc.tar.Z Contact: Howard Chu arc for VMS wuarchive.wustl.edu:/packages/compression/vax-vms/arc.exe arcmac for Mac mac.archive.umich.edu:/mac/utilities/compressionapps/arcmac.hqx arc for Amiga ftp.funet.fi:pub/amiga/fish/001-100/ff070/arc.lha .arj: arj for MSDOS (LZ77 with hashing, plus secondary static Huffman encoding on a block basis) Contact: Robert K Jung wuarchive.wustl.edu:/mirrors/msdos/archivers/arj241a.exe garbo.uwasa.fi:/pc/arcers/arj241a.exe unarj for Unix. Decompresses only. (There is no arj compressor for Unix. Don't post a request.) wuarchive.wustl.edu:/mirrors/misc/unix/unarj241.tar-z garbo.uwasa.fi:/unix/arcers/unarj241.tar.Z unarj for Mac mac.archive.umich.edu:/mac/util/compression/unarjmac.cpt.hqx unarj for Amiga ftp.funet.fi:pub/amiga/utilities/archivers/unarj-0.5.lha .cpt: Compact Pro for Mac sumex-aim.stanford.edu:/info-mac/util/compact-pro-133.hqx [36.44.0.6] For Unix: sumex-aim.stanford.edu:/info-mac/unix/macutil-20b1.shar ftp.cwi.nl:/pub/macutil2.0b3.shar.Z .exe: self-extracting MSDOS executable (creates files on disk when run) Run the file, or try unzip, lha or arj on it. .exe: compressed MSDOS executable (decompresses itself in memory then runs the decompressed code). To get the original uncompressed .exe: oak.oakland.edu:/pub/msdos/execomp/unp311.zip .gif: gif files are images compressed with the LZW algorithm. See the comp.graphics FAQ list for programs manipulating .gif files. See suffix .Z below for source of LZW. .gz, .z: gzip (or pack, see .z below). gzip uses the same algorithm as zip 1.9 (see below); it can also extract packed and compressed files. For Unix, MSDOS, OS/2, VMS, Atari, Amiga, Primos: prep.ai.mit.edu:/pub/gnu/gzip-1.2.3.tar (or .shar or .tar.z : source) prep.ai.mit.edu:/pub/gnu/gzip-msdos-1.2.3.exe (msdos, lha self-extract) garbo.uwasa.fi:/unix/arcers/gzip-1.2.3.tar.Z (source) garbo.uwasa.fi:/pc/arcers/gzip123.exe (msdos exe, self-extract) oak.oakland.edu:/pub/msdos/compress/gzip121.zip (msdos exe) ftp.spc.edu:[.macro32.savesets]gzip-1-2-2.zip (vms source & exe) sumex-aim.stanford.edu:/info-mac/tmit/gzip.hqx (Macintosh exe) .ha: ha 0.98 for MSDOS (improved PPMC - 4th order Markov modeling) garbo.uwasa.fi:/pc/arcers/ha098.exe .hqx: Macintosh BinHex format.. (BinHex is *not* a compression program, it is similar to uuencode but handles multiple forks.) for Mac: mac.archive.umich.edu:/mac/utilities/compressionapps/binhex4.0.bin for Unix: sumex-aim.stanford.edu:/info-mac/cmp/mcvert-212.shar [36.44.0.6] for MSDOS: wuarchive.wustl.edu:/mirrors/msdos/xbin23.zip .lha: .lzh: lha for MSDOS (LZ77 with a trie data structure, plus secondary static Huffman coding on a block basis) wuarchive.wustl.edu:/mirrors/msdos/archivers/lha213.exe (exe) wuarchive.wustl.edu:/mirrors/msdos/archivers/lha211sr.zip (sources) garbo.uwasa.fi:/pc/arcers/lha213.exe lharc for Unix. (LZ77 with hash table and binary trees, plus secondary Huffman coding) Warning: lharc can extract .lzh files created by lharc 1.xx but not those created by lha. See lha for Unix below. wuarchive.wustl.edu:/mirrors/misc/unix/lharc102a.tar-z garbo.uwasa.fi:/unix/arcers/lharcsrc.zoo lharc for VMS. Same warning as for Unix lharc. wuarchive.wustl.edu:/packages/compression/vax-vms/lharc.exe lha for Unix. Warning: all doc is in Japanese. wuarchive.wustl.edu:/mirrors/misc/unix/lha101u.tar-z garbo.uwasa.fi:/unix/arcers/lha-1.00.tar.Z Contact: lha-admin@oki.co.jp or oki@wbg.telcom.oki.co.jp lha for Mac mac.archive.umich.edu:/mac/utilities/compressionapps/maclha2.0.cpt.hqx lha for Amiga ftp.funet.fi:pub/amiga/utilities/archivers/LhA_e138.run .pak: pak for MSDOS (LZW algorithm) wuarchive.wustl.edu:/mirrors/msdos/archivers/pak251.exe garbo.uwasa.fi:/pc/arcers/pak251.exe .pit: PackIt (Macintosh) for Mac: sumex-aim.stanford.edu:/info-mac/util/stuffit-151.hqx [36.44.0.6] for Unix: sumex-aim.stanford.edu:/info-mac/unix/mcvert-165.shar [36.44.0.6] .pp: PowerPacker (Amiga) ftp.funet.fi:pub/amiga/fish/501-600/ff561/PPLib.lha .sea: self-extracting archive (Macintosh) Run the file to extract it. The self-extraction code can be removed with: mac.archive.umich.edu:/mac/utilities/compressionapps/desea1.11.cpt.hqx .sdn: used by the Shareware Distribution Network. Try the decompressors for .pak or .arj (see above) .shar: Shell archive. This is not a compression program. Use "sh foo.shar" to extract. .sit: Stuffit for Macintosh for Mac: sumex-aim.stanford.edu:/info-mac/util/stuffit-lite-30.hqx [36.44.0.6] for Unix: sumex-aim.stanford.edu:/info-mac/unix/unsit-15.shar [36.44.0.6] for Amiga: ftp.funet.fi:pub/amiga/utilities/archivers/unsit-1.5c2.lha for MSDOS: garbo.uwasa.fi:/pc/arcers/unsit30.zip .?q?: Squeeze for MSDOS (do not confuse with other 'squeeze' below). Static Huffman coding. oak.oakland.edu:/pub/msdos/starter/sqpc12a.com (squeeze) oak.oakland.edu:/pub/msdos/starter/nusq110.com (unsqueeze) .sqz: Squeeze for MSDOS (do not confuse with other 'squeeze' above) LZ77 with hashing. wuarchive.wustl.edu:/mirrors/msdos/archivers/sqz1083e.exe garbo.uwasa.fi:/pc/arcers/sqz1083e.exe .tar: tar is *not* a compression program. However, to be kind for you: for MSDOS wuarchive.wustl.edu:/mirrors/msdos/starter/tarread.exe garbo.uwasa.fi:/pc/unix/tar4dos.zoo for Unix tar (you have it already. To extract: tar xvf file.tar) for VMS wuarchive.wustl.edu:/packages/compression/vax-vms/tar.exe for Macintosh sumex-aim.stanford.edu:/info-mac/util/tar-30.hqx for Amiga: ftp.funet.fi:pub/amiga/fish/401-500/ff445/Tar.lha .tar.Z, .tar-z, .taz: tar + compress For Unix: zcat file.tar.Z | tar xvf - with GNU tar: tar xvzf file.tar.Z Other OS: first uncompress (see .Z below) then untar (see .tar above) .tar.z, .tgz: tar + gzip For Unix: gzip -cd file.tar.z | tar xvf - with GNU tar: tar xvzf file.tar.z Other OS: first uncompress (see .z below) then untar (see .tar above) .z: pack or gzip (see .gz above). pack uses static Huffman coding. To extract, see .gz above. .zip: pkzip 1.10 for MSDOS. (LZ77 with hashing, plus secondary static Shannon-Fano encoding on whole file) Contact: pkware.inc@mixcom.com wuarchive.wustl.edu:/mirrors/msdos/zip/pkz110eu.exe. garbo.uwasa.fi:/pc/arcers/pkz110eu.exe. Note: pkz110eu.exe is an 'export' version without encryption. zip 1.1 for Unix, MSDOS, VMS, OS/2, ... (compatible with pkzip 1.10. For corresponding unzip, see unzip 5.0p1 below). wuarchive.wustl.edu:/mirrors/misc/unix/zip11.zip arcutil 2.0 for VM/CMS (unzip only, not yet compatible with pkzip 2.04) vmd.cso.uiuc.edu:/public.477/arcutil.* [128.174.5.98]. pkzip 2.04g for MSDOS. (LZ77 with hashing, plus secondary static Huffman coding on a block basis) garbo.uwasa.fi:/pc/arcers/pkz204g.exe ftp.cica.indiana.edu:/pub/pc/starter/pkz204g.exe. zip 1.9p1 and unzip 5.0p1 for Unix, MSDOS, VMS, OS/2, Amiga, ... Compatible with pkzip 2.04g (LZ77 with hashing, plus secondary static Huffman coding on a block basis) Contact: zip-bugs@wkuvx1.bitnet oak.oakland.edu:/pub/misc/unix/zip19p1.zip (source) oak.oakland.edu:/pub/misc/unix/unz50p1.tar-z (tar.Z source) oak.oakland.edu:/pub/msdos/zip/zip19p1x.zip (MSDOS exe) oak.oakland.edu:/pub/msdos/zip/unz50p1.exe (MSDOS exe) wuarchive.wustl.edu:/mirrors/garbo.uwasa.fi/arcutil/zcrypt19.zip (encryption source. Non US residents must get it from garbo,see below) garbo.uwasa.fi:/unix/arcers/zip19p1.zip (source) garbo.uwasa.fi:/unix/arcers/unz50p1.tar.Z (source) garbo.uwasa.fi:/pc/arcers/zip19p1x.zip (MSDOS exe) garbo.uwasa.fi:/pc/arcers/unz50p1.exe (MSDOS exe) ftp.luth.se:/pub/amiga/util/arc/InfoZipI.lha (amiga exe zip+unzip) garbo.uwasa.fi:/pc/arcutil/zcrypt19.zip (encryption source) ftp.win.tue.nl:/pub/compression/zip/zcryp19x.zip (crypt MSDOS exe) for Macintosh (zip-it and unzip-201 are compatible with pkzip 2.04) sumex-aim.stanford.edu:/info-mac/util/zip-it-111.hqx sumex-aim.stanford.edu:/info-mac/util/unzip-201.hqx sumex-aim.stanford.edu:/info-mac/util/mac-zip-10.hqx .zoo: zoo 2.10 for MSDOS (algorithm copied from that of lha, see lha above) Contact: Rahul Dhesi wuarchive.wustl.edu:/mirrors/msdos/zoo/zoo210.exe garbo.uwasa.fi:/pc/arcers/zoo210.exe zoo 2.10 for Unix, VMS wsmr-simtel20.army.mil:pd8:zoo210.tar-z [192.88.110.2] garbo.uwasa.fi:/unix/arcers/zoo210.tar.Z zoo for Mac mac.archive.umich.edu:/mac/utilities/compressionapps/maczoo.sit.hqx zoo for Amiga ftp.funet.fi:pub/amiga/utilities/archivers/Zoo-2.1.lha .F: freeze for Unix (LZ77 with hashing, plus secondary dynamic Huffman encoding) wuarchive.wustl.edu:/usenet/comp.sources.misc/volume25/freeze/part0[1-2].Z ftp.inria.fr:/system/arch-compr/freeze-2.3.4.tar.Z Contact: Leonid A. Broukhis .Y: yabba for Unix, VMS, ... (Y coding, a variant of LZ78) wuarchive.wustl.edu:/usenet/comp.sources.unix/volume24/yabbawhap/part0[1-4].Z ftp.inria.fr:/system/arch-compr/yabba.tar.Z Contact: Dan Bernstein .Z: compress for Unix ('the' LZW algorithm) It is likely that your Unix system has 'compress' already. Otherwise: wuarchive.wustl.edu:/packages/compression/compress-4.1.tar (not in .Z format to avoid chicken and egg problem) compress for MSDOS wuarchive.wustl.edu:/mirrors/msdos/compress/comp430[ds].zip garbo.uwasa.fi:/pc/unix/comp430d.zip compress for Macintosh sumex-aim.stanford.edu:/info-mac/util/maccompress-32.hqx compress for Amiga ftp.funet.fi:pub/amiga/utilities/archivers/compress-4.1.lha compress for Vax/VMS wuarchive.wustl.edu:/packages/compression/vax-vms/lzcomp.exe wuarchive.wustl.edu:/packages/compression/vax-vms/lzdcmp.exe ------------------------------------------------------------------------------ ~Subject: [3] What is the latest PKZIP version? The latest official version is 2.04g. Release 2.04c had serious bugs, corrected in 2.04e and 2.04g. Be warned that there are countless bogus PKZIP 1.20, 2.0, 2.02, 3.05 and whatever scams floating around. They usually are hacks of PKZIP 1.93A beta test version. Some of them are trojans and / or carry computer virii. ------------------------------------------------------------------------------ ~Subject: [4] What is an archiver? There is a distinction between archivers and other compression programs: - an archiver takes several input files, compresses them and produces a single archive file. Examples are arc, arj, lha, zip, zoo. - other compression programs create one compressed file for each input file. Examples are freeze, yabba, compress. Such programs are often combined with tar to create compressed archives (see question 50: "What is this tar compression program?"). ------------------------------------------------------------------------------ ~Subject: [5] What is the best general purpose compression program? The answer is: it depends. (You did not expect a definitive answer, did you?) It depends whether you favor speed, compression ratio, a standard and widely used archive format, the number of features, etc... Just as for text editors, personal taste plays an important role. compress has 4 options, arj 2.30 has about 130 options; different people like different programs. *Please* do not start or continue flame wars on such matters of taste. The only objective comparisons are speed and compression ratio. Here is a short table comparing various programs on a 33Mhz Compaq 386. All programs have been run on Unix SVR4, except pkzip and arj which only run on MSDOS. (MSDOS benchmarks are available by ftp on oak.oakland.edu:/pub/msdos/info/arctst*.zip.) *Please* do not post your own benchmarks made on your own files that nobody else can access. If you think that you must absolutely post yet another benchmark, make sure that your test files are available by anonymous ftp. The programs compared here were chosen because they are the most popular or because they run on Unix and source is available. For ftp information, see above. Three programs (hpack, comp-2 and ha) have been added because they achieve better compression (at the expense of speed) and one program (lzrw3-a) has been added because it favors speed at the expense of compression: - comp-2 is in wuarchive.wustl.edu:/mirrors/msdos/ddjmag/ddj9102.zip (inner zip file nelson.zip), - hpack is in wuarchive.wustl.edu:/mirrors/misc/unix/hpack75a.tar-z and garbo.uwasa.fi:/unix/arcers/hpack78src.tar.Z - ha 0.98 is in garbo.uwasa.fi:/pc/arcers/ha098.zip - ftp.adelaide.edu.au:/pub/compression/lzrw3-a.c [129.127.40.3] The 14 files used in the comparison are from the standard Calgary Text Compression Corpus, available by ftp on ftp.cpsc.ucalgary.ca [136.159.7.18] in /pub/text.compression.corpus/text.compression.corpus.tar.Z. The whole corpus includes 18 files, but the 4 files paper[3-6] are generally omitted in benchmarks. It contains several kinds of file (ascii, binary, image, etc...) but has a bias towards large files. You may well get different ratings on the typical mix of files that you use daily, so keep in mind that the comparisons given below are only indicative. The programs are ordered by decreasing total compressed size. For a fair comparison between archivers and other programs, this size is only the size of the compressed data, not the archive size. The programs were run on an idle machine, so the elapsed time is significant and can be used to compare Unix and MSDOS programs. [Note: I did not have time to run again all benchmarks with more recent versions of freeze, arj and hpack. To be done for some future revision of this FAQ.] size lzrw3a compress lharc yabba pkzip freeze version: 4.0 1.02 1.0 1.10 2.3.5 options: -m300000 ------ ----- ------ ------ ------ ------ ------ bib 111261 49040 46528 46502 40456 41354 41515 book1 768771 416131 332056 369479 306813 350560 344793 book2 610856 274371 250759 252540 229851 232589 230861 geo 102400 84214 77777 70955 76695 76172 68626 news 377109 191291 182121 166048 168287 157326 155783 obj1 21504 12647 14048 10748 13859 10546 10453 obj2 246814 108040 128659 90848 114323 90130 85500 paper1 53161 24522 25077 21748 22453 20041 20021 paper2 82199 39479 36161 35275 32733 32867 32693 pic 513216 111000 62215 61394 65377 63805 53291 progc 39611 17919 19143 15399 17064 14164 14143 progl 71646 24358 27148 18760 23512 17255 17064 progp 49379 16801 19209 12792 16617 11877 11686 trans 93695 30292 38240 28092 31300 23135 22861 3,141,622 1,400,105 1,259,141 1,200,580 1,159,340 1,141,821 1,109,290 real 0m35s 0m59s 5m03s 2m40s 5m27s user 0m25s 0m29s 4m29s 1m46s 4m58s sys 0m05s 0m10s 0m07s 0m18s 0m08s MSDOS: 1m39s zoo lha arj pkzip zip hpack comp-2 ha 2.10 1.0(Unix) 2.30 2.04g 1.9 0.75a 0.98 ah 2.13(MSDOS) -jm -ex -6 a2 ------ ------ ------ ------ ------- ------ ------ ------ bib 40742 40740 36090 35186 34950 35619 29840 26927 book1 339076 339074 318382 313566 312619 306876 237380 235733 book2 228444 228442 210521 207204 206306 208486 174085 163535 geo 68576 68574 69209 68698 68418 58976 64590 59356 news 155086 155084 146855 144954 144395 141608 128047 123335 obj1 10312 10310 10333 10307 10295 10572 10819 9799 obj2 84983 84981 82052 81213 81336 80806 85465 80381 paper1 19678 19676 18710 18519 18525 18607 16895 15675 paper2 32098 32096 30034 29566 29674 29825 25453 23956 pic 52223 52221 53578 52777 55051 51778 55461 51639 progc 13943 13941 13408 13363 13238 13475 12896 11795 progl 16916 16914 16408 16148 16175 16586 17354 15298 progp 11509 11507 11308 11214 11182 11647 11668 10498 trans 22580 22578 20046 19808 18879 20506 21023 17927 1,096,166 1,096,138 1,036,934 1,019,451 1,021,043 1,005,367 890,976 845,854 real 4m07s 6m03s 1m49s 1h22m17s 27m05s user 3m47s 4m23s 1m43s 1h20m46s 19m27s sys 0m04s 0m08s 0m02s 0m12s 2m03s MSDOS: 1m49s 2m41s 1m43s 14m43s Notes: - the compressed data for 'zoo ah' is always two bytes longer than for lha. This is simply because both programs are derived from the same source (ar002, written by Haruhiko Okumura, available by ftp in wuarchive.wustl.edu:/mirrors/msdos/archivers/ar002.zip). - hpack 0.75a gives slightly different results on SunOS. (To be checked with latest version of hpack). - the MSDOS versions are all optimized with assembler code and were run on a RAM disk. So it is not surprising that they often go faster than their Unix equivalent. ------------------------------------------------------------------------------ ~Subject: [7] Which books should I read? [BWC 1989] Bell, T.C, Cleary, J.G. and Witten, I.H, "Text Compression", Prentice-Hall 1989. ISBN: 0-13-911991-4. Price: approx. US$40 The reference on text data compression. [Nel 1991] Mark Nelson, "The Data Compression Book" M&T Books, Redwood City, CA, 1991. ISBN 1-55851-216-0. Price $36.95 including two 5" PC-compatible disks bearing all the source code printed in the book. A practical introduction to data compression. The book is targeted at a person who is comfortable reading C code but doesn't know anything about data compression. Its stated goal is to get you up to the point where you are competent to program standard compression algorithms. [Will 1990] Williams, R. "Adaptive Data Compression", Kluwer Books, 1990. ISBN: 0-7923-9085-7. Price: US$75. Reviews the field of text data compression and then addresses the problem of compressing rapidly changing data streams. [Stor 1988] Storer, J.A. "Data Compression: Methods and Theory", Computer Science Press, Rockville, MD. ISBN: 0-88175-161-8. A survey of various compression techniques, mainly statistical non-arithmetic compression and LZSS compression. Includes complete Pascal code for a series of LZ78 variants. [ACG 1991] Advances in Speech Coding, edited by Atal, Cuperman, and Gersho, Kluwer Academic Press, 1991. [GG 1991] Vector Quantization and Signal Compression, by Gersho and Gray, Kluwer Acad. Press, 1991 [CT 1991] Elements of Information Theory, by T.M.Cover and J.A.Thomas John Wiley & Sons, 1991. ISBN 0-471-06259-6. Review papers: [BWC 1989] Bell, T.C, Witten, I.H, and Cleary, J.G. "Modeling for Text Compression", ACM Computing Surveys, Vol.21, No.4 (December 1989), p.557 A good general overview of compression techniques (as well as modeling for text compression); the condensed version of "Text Compression". [Lele 1987] Lelewer, D.A, and Hirschberg, D.S. "Data Compression", ACM Computing Surveys, Vol.19, No.3 (September 1987), p.261. A survey of data compression techniques which concentrates on Huffman compression and makes only passing mention of other techniques. ------------------------------------------------------------------------------ ~Subject: [8] What about patents on data compression algorithms? [Note: the appropriate group for discussing software patents is comp.patents (or misc.legal.computing), not comp.compression.] All patents mentioned here are US patents, and thus probably not applicable outside the US. See item 70, "Introduction to data compression" for the meaning of LZ77, LZ78 or LZW. (a) Run length encoding - Tsukiyama has two patents on run length encoding: 4,586,027 and 4,872,009 granted in 1986 and 1989 respectively. The first one covers run length encoding in its most primitive form: a length byte followed by the repeated byte. The second patent covers the 'invention' of limiting the run length to 16 bytes and thus the encoding of the length on 4 bits. Here is the start of claim 1 of patent 4,872,009, just for pleasure: 1. A method of transforming an input data string comprising a plurality of data bytes, said plurality including portions of a plurality of consecutive data bytes identical to one another, wherein said data bytes may be of a plurality of types, each type representing different information, said method comprising the steps of: [...] - O'Brien has patented (4,988,998) run length encoding followed by LZ77. (b) LZ77 - Waterworth patented (4,701,745) the algorithm now known as LZRW1, because Ross Williams reinvented it later and posted it on comp.compression on April 22, 1991. (See item 5 for the ftp site with all LZRW derivatives.) The *same* algorithm has later been patented by Gibson & Graybill (see below). The patent office failed to recognize that the same algorithm was patented twice, even though the wording used in the two patents is very similar. The Waterworth patent is now owned by Stac Inc, and used as the basis of a lawsuit against Microsoft, concerning the compression feature of MSDOS 6.0. - Fiala and Greene obtained in 1990 a patent (4,906,991) on all implementations of LZ77 using a tree data structure. Claim 1 of the patent is much broader than the algorithms published by Fiala and Greene in Comm.ACM, April 89. The patent covers the algorithm published by Rodeh and Pratt in 1981 (J. of the ACM, vol 28, no 1, pp 16-24). It also covers the algorithm previously patented by Eastman-Lempel-Ziv (4,464,650), and the algorithms used in lharc, lha and zoo. - Notenboom (from Microsoft) 4,955,066 uses three levels of compression, starting with run length encoding. - The Gibson & Graybill patent 5,049,881 covers the LZRW1 algorithm previously patented by Waterworth and reinvented by Ross Williams. Claims 4 and 12 are very general and could be interpreted as applying to any LZ algorithm using hashing (including all variants of LZ78): 4. A compression method for compressing a stream of input data into a compressed stream of output data based on a minimum number of characters in each input data string to be compressed, said compression method comprising the creation of a hash table, hashing each occurrence of a string of input data and subsequently searching for identical strings of input data and if such an identical string of input data is located whose string size is at least equal to the minimum compression size selected, compressing the second and all subsequent occurrences of such identical string of data, if a string of data is located which does not match to a previously compressed string of data, storing such data as uncompressed data, and for each input strings after each hash is used to find a possible previous match location of the string, the location of the string is stored in the hash table, thereby using the previously processed data to act as a compression dictionary. Claim 12 is identical, with 'method' replaced with 'apparatus'. Since the 'minimal compression size' can be as small as 2, the claim could cover any dictionary technique of the LZ family. However the text of the patent and the other claims make clear that the patent should cover the LZRW1 algorithm only. (In any case the Gibson & Graybill patent is likely to be invalid because of the prior art in the Waterworth patent.) - Phil Katz, author of pkzip, also has a patent on LZ77 (5,051,745) but the claims only apply to sorted hash tables, and when the hash table is substantially smaller than the window size. - IBM patented (5,001,478) the idea of combining a history buffer (the LZ77 technique) and a lexicon (as in LZ78). - Stac Inc patented (5,016,009 and 5,126,739) yet another variation of LZ77 with hashing. The '009 patent is used in the lawsuit against Microsoft (see above). Stac also has patents on LZ77 with parallel lookup in hardware (4,841,092 and 5,003,307). - Robert Jung, author of 'arj', has been granted patent 5,140,321 for one variation of LZ77 with hashing. This patent covers the LZRW3-A algorithm, also previously discovered by Ross Williams. LZRW3-A was posted on comp.compression on July 15, 1991. The patent was filed two months later on Sept 4, 1991. (The US patent system allows this because of the 'invention date' rule.) - Chambers 5,155,484 is yet another variation of LZ77 with hashing. The hash function is just the juxtaposition of two input bytes, this is the 'invention' being patented. The hash table is named 'direct lookup table'. (c) LZ78 - One form of the original LZ78 algorithm was patented (4,464,650) by its authors Lempel, Ziv, Cohn and Eastman. - The LZW algorithm used in 'compress'is patented by IBM (4,814,746) and Unisys (4,558,302). It is also used in the V.42bis compression standard (see question 11 on V.42bis below) and in Postscript Level 2. (Unisys sells the license to modem manufacturers for a onetime $25,000 fee.) The IBM patent application was filed three weeks before that of Unisys, but the US patent office failed to recognize that they covered the same algorithm. (The IBM patent is more general, but its claim 7 is exactly LZW.) - AP coding is patented by Storer (4,876,541). (Get the yabba package for source code, see question 2 above, file type .Y) (d) arithmetic coding - IBM holds many patents on arithmetic coding (4,286,256 4,295,125 4,463,342 4,467,317 4,633,490 4,652,856 4,891,643 4,905,297 4,935,882). It has patented in particular the Q-coder implementation of arithmetic coding. The arithmetic coding option of the JPEG standard requires use of the patented algorithm. No JPEG-compatible method is possible without infringing the patent, because what IBM actually claims rights to is the underlying probability model (the heart of an arithmetic coder). (See the JPEG FAQ for details.) AT&T has 3 patents on arithmetic coding (4,973,961, 5,023,611, 5,025,258). As can be seen from the above list, some of the most popular compression programs (compress, pkzip, zoo, lha, arj) are now covered by patents. (This says nothing about the validity of these patents.) Here are some references on data compression patents. A number of them are taken from the list prep.ai.mit.edu:/pub/lpf/patent-list. 3,914,586 Data compression method and apparatus filed 10/25/73, granted 10/21/75 General Motors Corporation, Detroit MI Duane E. McIntosh, Santa Ynez CA Data compression apparatus is disclosed is operable in either a bit pair coding mode of a word coding mode depending on the degree of redundancy of the data to be encoded. 3,976,844 Data communication system for transmitting data in compressed form filed Apr. 4, 1975, granted Aug. 24, 1976 inventor Bernard K. Betz, assignee Honeywell Information Systems, Inc. [encode differences with previous line] 4,021,782 Data compaction system and apparatus inventor Hoerning filed 04/30/1975, granted 05/03/1977 [A primitive form of LZ77 with implicit offsets (compare with previous record)] 4,054,951 Data expansion apparatus inventor R.D. Jackson, assignee IBM filed Jun. 30, 1976, granted Oct. 18, 1977 [Covers only decompression of data compressed with a variant of LZ77.] 4,087,788 Data compression system filed 1/14/77, granted 5/2/78 NCR Canada LTD - NCR Canada Ltee, Mississauga CA Brian J. Johannesson, Waterloo CA A data compression system is disclosed in which the left hand boundary of a character is developed in the form of a sequence of Freeman direction codes, the codes being stored in digital form within a processor. 4,286,256 Method and means for arithmetic coding using a reduced number of operations. granted Aug 25, 1981 assignee IBM 4,295,125 A method and means for pipeline decoding of the high to low order pairwise combined digits of a decodable set of relatively shifted finite number of strings granted Oct 13, 1981 assignee IBM 4,412,306 System for minimizing space requirements for storage and transmission of digital signals filed May 14, 1981, granted Oct. 25, 1983 inventor Edward W. Moll 4,463,342 A method and means for carry-over control in a high order to low order combining of digits of a decodable set of relatively shifted finite number strings. granted Jul 31, 1984 assignee IBM 4,491,934 Data compression process filed May 12, 1982, granted Jan. 1, 1985 inventor Karl E. Heinz 4,464,650 Apparatus and method for compressing data signals and restoring the compressed data signals inventors Lempel, Ziv, Cohn, Eastman assignees Sperry Corporation and At&T Bell Laboratories filed 8/10/81, granted 8/7/84 A compressor parses the input data stream into segments where each segment comprises a prefix and the next symbol in the data stream following the prefix. 4,467,317 High-speed arithmetic compression using using concurrent value updating. granted Aug 21, 1984 assignee IBM 4,494,108 Adaptive source modeling for data file compression within bounded memory filed Jun. 5, 1984, granted Jan. 15, 1985 invntors Glen G. Langdon, Jorma J. Rissanen assignee IBM order 1 Markov modeling 4,558,302 High speed data compression and decompression apparatus and method inventor Welch assignee Sperry Corporation (now Unisys) filed 6/20/83, granted 12/10/85 The text for this patent can be ftped from rusmv1.rus.uni-stuttgart.de (129.69.1.12) in /info/comp.patents/US4558302.Z. 4,560,976 Data compression filed 6/5/84, granted 12/24/85 Codex Corporation, Mansfield MA Steven G. Finn, Framingham, MA A stream of source characters, which occur with varying relative frequencies, is encoded into a compressed stream of codewords, each having one, two or three subwords, by ranking the source characters by their current frequency of appearance, encoding the source characters having ranks no higher than a first number as one subword codewords, source characters having ranks higher than the first number but no higher than a second number as two subword codewords, and the remaining source characters as three subword codewords. 4,586,027 Method and system for data compression and restoration inventor Tsukimaya et al. assignee Hitachi filed 08/07/84, granted 04/29/86 4,597,057 System for compressed storate of 8-bit ascii bytes using coded strings of 4-bit nibbles. inventor Snow, assignee System Development corporation. filed 12/31/1981, granted 06/24/1986. Compression using static dictionary of common words, prefixes and suffixes. 4,612,532 Data compression apparatus and method inventor Bacon, assignee Telebyte Corportion filed Jun. 19, 1984, granted Sep. 16, 1986 [Uses followsets as in the pkzip 0.92 'reduce' algorithm, but the followsets are dynamically updated. This is in effect a sort of order-1 Markov modeling.] 4,622,545 Method and apparatus for image compression and Manipulation inventor William D. Atkinson assignee Apple computer Inc. filed 9/30/82 granted 11/11/86 4,633,490 Symmetrical adaptive data compression/decompression system. granted Dec 30, 1985 assignee IBM 4,652,856 A multiplication-free multi-alphabet arithmetic code. granted Feb 4, 1986 assignee IBM 4,667,649 Data receiving apparatus filed 4/18/84, granted 6/30/87 inventors Kunishi et al. assignee Canon Kabushiki Kaisha, Tokyo Japan compression of Fax images. 4,682,150 Data compression method and apparatus inventors Mathes and Protheroe, assignee NCR Corporation, Dayton OH A system and apparatus for compressing redundant and nonredundant binary data generated as part of an operation of a time and attendance terminal in which the data represents the time an employee is present during working hours. 4,701,745 Data compression system inventor Waterworth John R assignee Ferranti PLC GB, patent rights now acquired by Stac Inc. filed 03/03/1986 (03/06/1985 in GB), granted 10/20/1987 Algorithm now known as LZRW1 (see above) I claim: 1. A data compression system comprising an input store for receiving and storing a plurality of bytes of uncompressed data from an outside source, and data processing means for processing successive bytes of data from the input store; the data processing means including circuit means operable to check whether a sequence of successive bytes to be processed identical with a sequence of bytes already processed, and including hash generating means responsive to the application of a predetermined number of bytes in sequence to derive a hash code appropriate to those bytes, a temporary store in which the hash code may represent the address of a storage location, and a pointer counter operable to store in the temporary store at said address a pointer indicative of the position in the input store of one of the predetermined number of bytes; output means operable to apply to a transfer medium each byte of data not forming part of such an identical sequence; and encoding means responsive to the identification of such a sequence to apply to the transfer medium an identification signal which identifies both the location in the input store of the previous occurrence of the sequence of bytes and the number of bytes contained in the sequence. 4,730,348 Adaptive data compression system inventor MacCrisken, assignee Adaptive Computer Technologies filed Sep. 19, 1986, granted Mar. 8, 1988 [order-1 Markov modeling + Huffman coding + LZ77] 4,758,899 Data compression control device inventor Tsukiyama, assignee Hitachi filed 11/20/1985, granted 07/19/1988 Limits compression to ensure that tape drive stays busy. 4,809,350 Data compression system filed Jan. 30, 1987, granted Feb. 28, 1989 inventor Yair Shimoni & Ron Niv assignee Elscint Ltd., Haifa, Israel [Image compression via variable length encoding of differences with predicted data.] 4,814,746 Data compression method inventors Victor S. Miller, Mark N. Wegman assignee IBM filed 8/11/86, granted 3/21/89 A previous application was filed on 6/1/83, three weeks before the application by Welch (4,558,302) Communications between a Host Computing System and a number of remote terminals is enhanced by a data compression method which modifies the data compression method of Lempel and Ziv by addition of new character and new string extensions to improve the compression ratio, and deletion of a least recently used routine to limit the encoding tables to a fixed size to significantly improve data transmission efficiency. 4,853,696 Code converter for data compression/decompression filed 4/13/87, granted 8/1/89 inventor Amar Mukherjee, Maitland FL assignee University of Central Florida, Orlando FL Another hardware Huffman encoder: A code converter has a network of logic circuits connected in reverse binary tree fashion with logic paths between leaf nodes and a common root node. 4,872,009 Method and apparatus for data compression and restoration inventor Tsukimaya et al. assignee Hitachi filed 12/07/87, granted 10/03/89 This patent on run length encoding covers the 'invention' of limiting the run length to 16 bytes and thus the encoding of the length on 4 bits. 4,876,541 Stem [sic] for dynamically compressing and decompressing electronic data filed 10/15/87, granted 10/24/89 inventor James A. Storer assignee Data Compression Corporation A data compression system for encoding and decoding textual data, including an encoder for encoding the data and for a decoder for decoding the encoded data. 4,891,643 Arithmetic coding data compression/de-compression by selectively employed, diverse arithmetic encoders and decoders. granted Jan 2, 1990 assignee IBM 4,905,297 granted Feb 27, 1990 assignee IBM Arithmetic coding encoder and decoder system. 4,906,991 Textual substitution data compression with finite length search window filed 4/29/1988, granted 3/6/1990 inventors Fiala,E.R., and Greene,D.H. assignee Xerox Corporation 4,935,882 Probability adaptation for arithmetic coders. granted Jun 19, 1990 assignee IBM 4,941,193 Barnsley, fractal compression. 4,943,869 Compression Method for Dot Image Data filed 1988-05-04, granted 1990-07-24 assignee Fuji Photo Film Co. Lossy and lossless image compression schemes. 4,955,066 Compressing and Decompressing Text Files filed 10/13/89, granted 09/04/90 inventor Notenboom, L.A. assignee Microsoft Now extended as 5,109,433 [Noted in signon screen of Word 5.5 and on the outside of the MS-DOS 5.0 Upgrade.] A method of compressing a text file in digital form is disclosed. A full text file having characters formed into phrases is provided by an author. The characters are digitally represented by bytes. A first pass compression is sequentially followed by a second pass compression of the text which has previously been compressed. A third or fourth level of compression is serially performed on the compressed text. For example, in a first pass, the text is run-length compressed. In a second pass, the compressed text is further compressed with key phrase compression. In a third pass, the compressed text is further compressed with Huffman compression. The compressed text is stored in a text file having a Huffman decode tree, a key phrase table, and a topic index. The data is decompressed in a single pass and provided one line at a time as an output. Sequential compressing of the text minimizes the storage space required for the file. Decompressing of the text is performed in a single pass. As a complete line is decompressed, it is output rapidly, providing full text to the user. 4,973,961 Method and apparatus for carry-over control in arithmetic coding. granted Nov 27, 1990 assignee AT&T 4,988,998 Data compression system for successively applying at least two data compression methods to an input data stream. inventor O'Brien assignee Storage Technology Corporation, Louisville, Colorado filed Sep 5, 1989, granted Jan 29, 1991. Run length encoding followed by LZ77. 5,001,478 Method of Encoding Compressed Data filed 12/28/89, granted 03/19/91 inventor Michael E. Nagy assignee IBM 1. A method of encoding a compressed data stream made up of a sequence of literal references, lexicon references and history references, which comprises the steps of: assigning to each literal reference a literal identifier; assigning to each history reference a history identifier; assigning to each lexicon reference a lexicon identifier; and emitting a data stream with said identifiers assigned to said references. Gordon Irlam says: The invention can probably be best understood by considering the decompressor. It consists of a history buffer, and a lexicon buffer, both of which are initially empty. The history buffer contains the last n symbols emitted. Whenever a history buffer reference is to be output the string so referenced is subsequently moved to the lexicon buffer for future reference. Thus the history buffer keeps track of strings that may be repeated on a very short term basis, while the lexicon buffer stores items for a longer time. Furthermore a history reference involves specifying both the offset and length within the history buffer, whereas a lexicon reference simply specifies a number denoting the string. Both buffers have a finite size. 5,003,307 Data compression apparatus with shift register search means filed Oct. 6, 1989, granted Mar. 26, 1991 inventors George Glen A, Ivey Glen E, Whiting Douglas L assignee Stac Inc continuation of 4,841,092 5,016,009 Data compression apparatus and method filed 01/13/1989, granted 05/14/1991 inventors George Glen A, Ivey Glen E, Whiting Douglas L assignee Stac Inc LZ77 with offset hash table (extended in 5,126,739) 5,023,611 Entropy encoder/decoder including a context extractor. granted Jun 11, 1991 assignee AT&T 5,025,258 Adaptive probability estimator for entropy encoder/decoder. granted Jun 18, 1991 assignee AT&T 5,049,881 Apparatus and method for very high data rate-compression incorporating lossless data compression and expansion utilizing a hashing technique inventors Dean K. Gibson, Mark D. Graybill assignee Intersecting Concepts, Inc. filed 6/18/90, granted 9/17/91 [covers lzrw1, almost identical with Waterworth 4,701,745] 5,051,745 String searcher, and compressor using same filed 8/21/90, granted 9/24/91 inventor Phillip W. Katz (author of pkzip) In the string search method and apparatus pointers to the string to be searched are indexed via a hashing function and organized according to the hashing values of the string elements pointed to. The hashing function is also run on the string desired to be found, and the resulting hashing value is used to access the index. If the resulting hashing value is not in the index, it is known that the target string does not appear in the string being searched. Otherwise the index is used to determine the pointers which correspond to the target hashing value, these pointers pointing to likely candidates for matching the target string. The pointers are then used to sequentially compare each of the locations in the string being searched to the target string, to determine whether each location contains a match to the target string. In the method and apparatus for compressing a stream of data symbols, a fixed length search window, comprising a predetermined contiguous portion of the symbol stream, is selected as the string to be searched by the string searcher. If a string to be compressed is found in the symbol stream, a code is output designating the location within the search window of the matching string and the length of the matching string. 5,065,447 Barnsley, fractal compression 5,109,433 Compressing and decompressing text files inventor Notenboom assignee Microsoft extension of 4,955,066 5,126,739 Data Compression Apparatus and Method filed Nov. 27, 1990, granted June 30, 1992. inventor Whiting et. al assignee Stac Inc LZ77 with offset hash table (extension of 5,016,009) 5,140,321 Data compression/decompression method and apparatus filed 9/4/91, granted 8/18/92 inventor Robert Jung assignee Prime Computer 5,155,484 Fast data compressor with direct lookup table indexing into history buffer filed 9/13/1991, granted 10/13/1992 inventor Chambers, IV, Lloyd L., Menlo Park, California assignee Salient Software, Inc., Palo Alto, California (02) Uses a 64K hash table indexed by the first two characters of the input string. Includes several claims on the LZ77 file format (literal or offset,length). 5,179,378 file Jul. 30, 1991, granted Jan. 12, 1993 inventor Ranganathan assignee University of South Florida Method and apparatus for the compression and decompression of data using Lempel-Ziv based techniques. [This covers LZ77 hardware compression with a systolic array of processors working in parallel.] Japan 2-46275 Coding system granted Feb 26, 1990 [Patents one form of arithmetic coding.] ------------------------------------------------------------------------------ ~Subject: [9] The WEB 16:1 compressor. [WARNING: this topic has generated the greatest volume of news in the history of comp.compression. Read this before posting on this subject.] [WARNING 2: it is quite possible that the story is repeating itself with a compressor called MINC by Premier Research Corporation, Ltd. They claim a breakthrough in lossless data compression using a recursive method, that is, applying the compressor to the compressed output of the previous run. More details in the next revision of the FAQ.] (a) What the press says April 20, 1992 Byte Week Vol 4. No. 25: "In an announcement that has generated high interest - and more than a bit of skepticism - WEB Technologies (Smyrna, GA) says it has developed a utility that will compress files of greater than 64KB in size to about 1/16th their original length. Furthermore, WEB says its DataFiles/16 program can shrink files it has already compressed." [...] "A week after our preliminary test, WEB showed us the program successfully compressing a file without losing any data. But we have not been able to test this latest beta release ourselves." [...] "WEB, in fact, says that virtually any amount of data can be squeezed to under 1024 bytes by using DataFiles/16 to compress its own output multiple times." June 1992 Byte, Vol 17 No 6: [...] According to Earl Bradley, WEB Technologies' vice president of sales and marketing, the compression algorithm used by DataFiles/16 is not subject to the laws of information theory. [...] (b) First details, by John Wallace : I called WEB at (404)514-8000 and they sent me some product literature as well as chatting for a few minutes with me on the phone. Their product is called DataFiles/16, and their claims for it are roughly those heard on the net. According to their flier: "DataFiles/16 will compress all types of binary files to approximately one-sixteenth of their original size ... regardless of the type of file (word processing document, spreadsheet file, image file, executable file, etc.), NO DATA WILL BE LOST by DataFiles/16." (Their capitalizations; 16:1 compression only promised for files >64K bytes in length.) "Performed on a 386/25 machine, the program can complete a compression/decompression cycle on one megabyte of data in less than thirty seconds" "The compressed output file created by DataFiles/16 can be used as the input file to subsequent executions of the program. This feature of the utility is known as recursive or iterative compression, and will enable you to compress your data files to a tiny fraction of the original size. In fact, virtually any amount of computer data can be compressed to under 1024 bytes using DataFiles/16 to compress its own output files muliple times. Then, by repeating in reverse the steps taken to perform the recusive compression, all original data can be decompressed to its original form without the loss of a single bit." Their flier also claims: "Constant levels of compression across ALL TYPES of FILES" "Convenient, single floppy DATA TRANSPORTATION" >From my telephone conversation, I was was assured that this is an actual compression program. Decompression is done by using only the data in the compressed file; there are no hidden or extra files. (c) More information, by Rafael Ramirez : Today (Tuesday, 28th) I got a call from Earl Bradley of Web who now says that they have put off releasing a software version of the algorithm because they are close to signing a major contract with a big company to put the algorithm in silicon. He said he could not name the company due to non-disclosure agreements, but that they had run extensive independent tests of their own and verified that the algorithm works. [...] He said the algorithm is so simple that he doesn't want anybody getting their hands on it and copying it even though he said they have filed a patent on it. [...] Mr. Bradley said the silicon version would hold up much better to patent enforcement and be harder to copy. He claimed that the algorithm takes up about 4K of code, uses only integer math, and the current software implementation only uses a 65K buffer. He said the silicon version would likely use a parallel version and work in real-time. [...] (d) The impossiblity proofs. It is impossible for a given program to compress without loss *all* files greater than a certain size by at least one bit. This can be proven by a simple counting argument. (Many other proofs have been posted on comp.compression, *please* do not post yet another one.) Assume that the program can compress without loss all files of size >= N bits. Compress with this program all the 2^N files which have exactly N bits. All compressed files have at most N-1 bits, so there are at most (2^N)-1 different compressed files [2^(N-1) files of size N-1, 2^(N-2) of size N-2, and so on, down to 1 file of size 0]. So at least two different input files must compress to the same output file. Hence the compression program cannot be lossless. (Stronger results about the number of incompressible files can be obtained, but the proofs are a little more complex.) This argument applies of course to WEB's case (take N = 64K*8 bits). Note that no assumption is made about the compression algorithm. The proof applies to *any* algorithm, including those using an external dictionary, or repeated application of another algorithm, or combination of different algorithms, or representation of the data as formulas, etc... All schemes are subject to the counting argument. There is no need to use information theory to provide a proof, just basic mathematics. This assumes of course that the information available to the decompressor is only the bit sequence of the compressed data. If external information such as a file name or a number of iterations is necessary to decompress the data, the bits providing the extra information must be included in the bit count of the compressed data. (Otherwise, it would be sufficient to consider any input data as a number, use this as the iteration count or file name, and pretend that the compressed size is zero.) [See also question 73 "What is the theoretical compression limit?" in part 2 of this FAQ.] (e) No software version Appeared on BIX, reposted by Bruce Hoult : tojerry/chaos #673, from abailey, 562 chars, Tue Jun 16 20:40:34 1992 Comment(s). ---------- TITLE: WEB Technology I promised everyone a report when I finally got the poop on WEB's 16:1 data compression. After talking back and forth for a year and being put off for the past month by un-returned phone calls, I finally got hold of Marc Spindler who is their sales manager. _No_ software product is forth coming, period! He began talking about hardware they are designing for delivery at the end of the year. [...] (f) Product cancelled Posted by John Toebes on Aug 10th, 1992: [Long story omitted, confirming the reports made above about the original WEB claims.] 10JUL92 - Called to Check Status. Was told that testing had uncovered a new problem where 'four numbers in a matrix were the same value' and that the programmers were off attempting to code a preprocessor to eliminate this rare case. I indicated that he had told me this story before. He told me that the programmers were still working on the problem. 31JUL92 - Final Call to Check Status. Called Earl in the morning and was told that he still had not heard from the programmers. [...] Stated that if they could not resolve the problem then there would probably not be a product. 03AUG92 - Final Call. Earl claims that the programmers are unable to resolve the problem. I asked if this meant that there would not be a product as a result and he said yes. (g) Conclusion The last report given above should put an end to the WEB story. [Note from the FAQ maintainer: I intended to remove this story from the FAQ, but the recent announcement of the MINC compressor has some similarities with the WEB story so it is useful to keep it a little longer.] ------------------------------------------------------------------------------ ~Subject: [11] What is the V.42bis standard? A description of the V.42bis standard is given in "The V.42bis standard for data-compressing modems," by Clark Thomborson , IEEE Micro, Oct 1992, pp. 41-53. Short introduction, by Alejo Hausner : The V.42bis Compression Standard was proposed by the International Consultative Committee on Telephony and Telegraphy (CCITT) as an addition to the v.42 error-correction protocol for modems. Its purpose is to increase data throughput, and uses a variant of the Lempel-Ziv-Welch (LZW) compression method. It is meant to be implemented in the modem hardware, but can also be built into the software that interfaces to an ordinary non-compressing modem. V.42bis can send data compressed or not, depending on the data. There are some types of data that cannot be compressed. For example, if a file was compressed first, and then sent through a V.42bis modem, the modem would not likely reduce the number of bits sent. Indeed it is likely that the amount of data would increase somewhat. To avoid this problem, the algorithm constantly monitors the compressibility of the data, and if it finds fewer bits would be necessary to send it uncompressed, it switches to transparent mode. The sender informs the receiver of this transition through a reserved escape code. Henceforth the data is passed as plain bytes. The choice of escape code is clever. Initially, it is a zero byte. Any occurrence of the escape code is replaced, as is customary, by two escape codes. In order to prevent a string of escape codes from temporarily cutting throughput in half, the escape code is redefined by adding 51 mod 256 each time it is used. While transmitting in transparent mode, the sender maintains the LZW trees of strings, and expects the receiver to do likewise. If it finds an advantage in returning to compressed mode, it will do so, first informing the receiver by a special control code. Thus the method allows the hardware to adapt to the compressibility of the data. The CCITT standards documents used to be available by ftp on ftp.uu.net in /doc/standards/ccitt, but this service has been discontinued. If you ftp to digital.resource.org, in directory pub/standards there is a file that says that making the standards available in the first place was just an experiment. The documents are now on src.doc.ic.ac.uk, in directory /computing/ccitt/standards/ccitt (they used to be in directory doc/ccitt-standards/ccitt, so look around if you don't find them). The v42bis standard is in /computing/ccitt/standards/ccitt/1992/v/v42bis.asc.Z. A mail server for CCITT documents is available at teledoc@itu.arcom.ch. For more information, contact Robert Shaw or Antoinette Bautista . Warning by John Levine : This teledoc thing is much less than meets the eye. What it actually has is one-page abstracts of some but not all CCITT recommendations, along with junk like lists of the national representatives to CCITT. If you want the actual text of a recommendation, you have to send large amounts of money to Switzerland, same as ever. However, a closer reading of the Teledoc announcement shows that they say they're planning to make the actual text of some CCITT recommendations available on-line sometime in 1993. See also the Standards FAQ posted to news.answers or get it by ftp in rtfm.mit.edu:/pub/usenet/news.answers/standards-faq. ------------------------------------------------------------------------------ ~Subject: [12] I need source for the winners of the Dr Dobbs compression contest The source of the top 6 programs of the Feb 91 Dr Dobbs data compression contest are available by ftp on wsmr-simtel20.army.mil in pd1:ddjcompr.zip. [192.88.110.2] garbo.uwasa.fi:/pc/source/ddjcompr.zip [128.214.87.1] The sources are in MSDOS end-of-line format, one directory per program. Unix or VMS users, use "unzip -a ddjcompr" to get correct end-of-lines (add -d to recreate the directory structure if you are using an obsolete version of unzip such as 4.1). Three of the 6 programs are not portable and only run on MSDOS. compact and urban work on Unix, sixpack only requires minor modifications. ------------------------------------------------------------------------------ ~Subject: [13] I need source for arithmetic coding (See question 70 for an introduction to arithmetic coding.) The source for the arithmetic coder described in Chap.5 of Bell, Cleary, and Witten's book "Text Compression" (see question 7 above) (or, equivalently, in: Witten, Neal, and Cleary's article "Arithmetic Coding for data Compression" from Communications of the Association for Computing Machinery, 30 (6), pp.520-540, June, 1987) is available via anonymous ftp from fsa.cpsc.ucalgary.ca (136.159.2.1) in directory /pub/arithmetic.coding. It only comes with a simple order-0 model but it's set up so that adding your own more sophisticated one is straightforward. A low precision arithmetic coding implementation avoiding hardware division is available on the same site (fsa.cpsc.ucalgary.ca) in /pub/arithmetic.coding/low.precision.version/low.precision.version.shar. Kris Popat has worked on "Scalar Quantization with Arithmetic Coding." It describes an arithmetic coding technique which is quite general and computationally inexpensive. The documentation and example C code are available via anonymous ftp from media-lab.media.mit.edu (18.85.0.2), in /pub/k-arith-code. The program 'urban' in ddjcompr.zip (see item 12 above) is a high order arithmetic coder working at the bit level. It is written by Urban Koistinen . ------------------------------------------------------------------------------ ~Subject: [15] Where can I get image compression programs? JPEG: Source code for most any machine: ftp.uu.net:/graphics/jpeg/jpegsrc.v4.tar.Z [137.39.1.9] nic.funet.fi:/pub/graphics/packages/jpeg/jpegsrc.v4.tar.Z [128.214.6.100] Contact: jpeg-info@uunet.uu.net (Independent JPEG Group) havefun.stanford.edu:pub/jpeg/JPEGv1.2.beta.tar.Z (supports lossless mode) Contact: Andy Hung (see item 20 below) xv, an image viewer which can read JPEG pictures, is available in export.lcs.mit.edu: contrib/xv-2.21.tar.Z [18.24.0.12] MPEG: havefun.stanford.edu:/pub/mpeg/MPEGv1.2.alpha.tar.Z Contact: Andy Hung (see item 20 below) toe.cs.berkeley.edu:/pub/multimedia/mpeg/mpeg_play-2.0.tar.Z toe.cs.berkeley.edu:/pub/multimedia/mpeg/mpeg_encode-1.0.tar.Z. Contact: mpeg-bugs@cs.berkeley.edu nvr.com:/pub/NVR-software/Product-1.0.4.tar.Z (192.82.231.50) (free demo copy of NVR's software toolkit for SPARCstations) Contact: Todd Brunhoff H.261(P*64): havefun.stanford.edu:pub/p64/P64v1.2.alpha.tar.Z Contact: Andy Hung (see item 20 below) ftp.inria.fr:/INRIA/ivs-2.1.tar.Z (Inria videoconference system) epic: (pyramid wavelet coder, see item 72) whitechapel.media.mit.edu:/pub/epic.tar.Z [18.85.0.125] Contact: Eero P. Simoncelli The "Lenna" test image is available as part of the EPIC package, where it is named "test_image". hcompress: (wavelet impage compression, see item 72) stsci.edu:/software/hcompress/hcompress.tar.Z wavethresh: (wavelet software for the language S) gdr.bath.ac.uk:/pub/masgpn/wavethresh2.2.Z Contact: gpn@maths.bath.ac.uk compfits: uwila.cfht.hawaii.edu:/pub/compfits/compfits.tar.Z [128.171.80.50] Contact: Jim Wright fitspress: cfata4.harvard.edu:/pub/fitspress08.tar.Z [128.103.40.79] tiff: For source and sample images, see question 18 below. For image compression hardware, see item 85 in part 3 of this FAQ. ------------------------------------------------------------------------------ ~Subject: [16] What is the state of the art in lossless image compression? The current state-of-the-art is the JBIG algorithm. For an introduction to JBIG, see question 74 in part 2. JBIG works best on bi-level images (like faxes) and also works well on Gray-coded grey scale images up to about six or so bits per pixel. You just apply JBIG to the bit planes individually. For more bits/pixel, lossless JPEG provides better performance, sometimes. (For JPEG, see question 19 below.) You can find a description of JBIG in ISO/IEC CD 11544, contained in document ISO/IEC JTC1/SC2/N2285. The only way to get it is to ask your National Standards Body for a copy. In the USA, call ANSI at (212) 642-4900. ------------------------------------------------------------------------------ ~Subject: [17] What is the state of fractal compression? from Tal Kubo : According to Barnsley's book 'Fractals Everywhere', this method is based on a measure of deviation between a given image and its approximation by an IFS code. The Collage Theorem states that there is a convergent process to minimize this deviation. Unfortunately, according to an article Barnsley wrote for BYTE a few years ago, this convergence was rather slow, about 100 hours on a Cray, unless assisted by a person. Barnsley et al are not divulging any technical information beyond the meager bit in 'Fractals Everywhere'. The book explains the idea of IFS codes at length, but is vague about the application of the Collage theorem to specific compression problems. There is reason to believe that Barnsley's company has *no algorithm* which takes a given reasonable image and achieves the compression ratios initially claimed for their fractal methods. The 1000-to-1 compression advertised was achieved only for a 'rigged' class of images, with human assistance. The best unaided performance I've heard of is good lossy compression of about 80-1. Steve Tate confirms: Compression ratios (unzoomed) seem to range from 20:1 to 60:1... The quality is considerably worse than wavelets or JPEG on most of the non-contrived images I have seen. But Yuval Fisher disagrees: Their performance has improved dramatically beyond what they were talking about in BYTE a few years ago. Human assistance to the compression is no longer needed and the compression time is reasonable, although the more time and compute power you throw at the compression, the smaller the resulting file for the same level of quality. Geoffrey A Stephenson adds: Iterated systems are shipping a general purpose compressor at about 300 Pounds in the UK that claims "640x480 24 bit colour compression of about 1 min at 922k -> 10k on a 486/50 software only, decomp. to 8 bits in 3 secs, etc." At a recent multimedia conference in London they handed out free demo disks that show the decomp. in action. The package runs under both DOS anf WIN (DLLs provided for use in applications). They also sell a board to speed up compression and offer versions supporting full motion video (but not apparently at all SVGA sizes like the static picture version). I have not yet got my hands on a full version to test different types of pictures, but friends have a and claim it looks good. Thomas W. Colthurst clarifies the distinction between IFS and the Fractal Transform: It is time, once and for all, to put to death the Barnsley myth that IFSs are good for image compression. They are not. Various algorithms have been proposed for this "inverse problem" ranging from the trendy (genetic algorithms) to the deep (moment methods) to the ad hoc (the hungry algorithm) to the absurd (the so-called "graduate student algorithm", consisting of locking up a grad student in a tiny office with a SGI workstation and not letting them out until they come up with a good IFS for your image). They are all useless for practical image compression. In fact, there are even good theoretical reasons for believing that IFSs will never be useful for image compression. For example, even if you have an IFS for object A and an IFS for object B, there is no way to combine these IFSs to get an IFS for object A union B or object A intersect B. Even Barnsley himself admits, in his latest book, that he doesn't use IFS image compression. Instead, he uses the so-called "fractal transform," which is really just a variant of vector quantization where you use the image itself, sampled at a higher scale, as the VQ codebook. To be fair, the fractal transform can be analyzed using local IFSs, but local IFSs are immensely more complicated and general than normal IFSs, to the point where one feels suspect even using the word "IFS" to describe them. It should be emphasized that the fractal transform is a real, working method that performs about as well as other existing methods like VQ or the discrete cosine transform. The fractal transform will probably never beat vector quantization (VQ) as for size of the compressed image, but does have the advantage that you don't need to carry your codebook around. The latest results have it slightly winning over the discrete cosine transform; only time and more research will tell if this advantage persists. Just like VQ, the fractal transform takes a while to compress, but is quick at decompression (Barnsley's company has hardware to do this in realtime). In short, IFSs are good for just about everything fractals are (and more!), but are absolutely horrid for image compression. Programs: A fractal image compression program is available by ftp in lyapunov.ucsd.edu:/pub/young-fractal/unifs10.zip. (Unix users, See item 2 above for unzip on Unix.) Note the file size before you ftp it: 1.2 MB. The package contains source for compression and decompression, source for X-windows decompression, MSDOS executables and images. A newer version of the program is in yuvpak20.zip. A fractal image decompression program (note: decompression only) is available in /pub/inls-ucsd/fractal-2.0.tar on on the same ftp site (lyapunov.ucsd.edu). Note the file size before you ftp it: 1.3 MB. This file also contains a paper by Yuval Fisher (see reference below), and some executables and sample images. Reading this paper is required to understand how the Young compression programs (see above) works. ~References: A. Jacquin, 'Fractal image coding based on a theory of iterated contractive image transformations', Visual Comm. and Image Processing, vol SPIE-1360, 1990. (The best paper that explains the concept in a simple way.) A. Jacquin, "A Fractal Theory of Iterated Markov Operators with Applications to Digital Image Coding", PhD Thesis, Georgia Tech, 1989. It can be obtained from university microfilms for $35, phone 1-800-521-0600. M. Barnsley, L. Anson, "Graphics Compression Technology, SunWorld, October 1991, pp. 42-52. M.F. Barnsley, A. Jacquin, F. Malassenet, L. Reuter & A.D. Sloan, 'Harnessing chaos for image synthesis', Computer Graphics, vol 22 no 4 pp 131-140, 1988. M.F. Barnsley, A.E. Jacquin, 'Application of recurrent iterated function systems to images', Visual Comm. and Image Processing, vol SPIE-1001, 1988. A. Jacquin, "Image Coding Based on a Fractal Theory of Iterated Contractive Image Transformations" p.18, January 1992 (Vol 1 Issue 1) of IEEE Trans on Image Processing. A.E. Jacquin, 'A novel fractal block-coding technique for digital images', Proc. ICASSP 1990. G.E. Oien, S. Lepsoy & T.A. Ramstad, 'An inner product space approach to image coding by contractive transformations', Proc. ICASSP 1991, pp 2773-2776. D.S. Mazel, Fractal Modeling of Time-Series Data, PhD Thesis, Georgia Tech, 1991. (One dimensional, not pictures) S. A. Hollatz, "Digital image compression with two-dimensional affine fractal interpolation functions", Department of Mathematics and Statistics, University of Minnesota-Duluth, Technical Report 91-2. (a nuts-and-bolts how-to-do-it paper on the technique) Stark, J., "Iterated function systems as neural networks", Neural Networks, Vol 4, pp 679-690, Pergamon Press, 1991. Monro D M and Dudbridge F, "Fractal block coding of images", Electronics Letters 28(11):1053-1054 (1992) Beaumont J M, "Image data compression using fractal techniques", British Telecom Technological Journal 9(4):93-108 (1991) Fisher Y, "Fractal image compression", Siggraph 92 Graf S, "Barnsley's Scheme for the Fractal Encoding of Images", Journal Of Complexity, V8, 72-78 (1992). Monro D.M. 'A hybrid fractal transform', Proc ICASSP 93, pp. V: 169-72 Monro D.M. & Dudbridge F. 'Fractal approximation of image blocks', Proc ICASSP 92, pp. III: 485-488 Monro D.M., Wilson D., Nicholls J.A. 'High speed image coding with the Bath Fractal Transform', IEEE International Symposium on Multimedia Technologies Southampton, April 1993 Jacobs, E.W., Y. Fisher and R.D. Boss. "Image Compression: A study of the Iterated Transform Method." _Signal Processing 29_ (1992) 25-263 Vrscay, Edward R. "Iterated Function Systems: Theory, Applications, and the Inverse Problem." _Fractal Geometry and Analysis_, J. Belair and S. Dubuc (eds.) Kluwer Academic, 1991. 405-468. Books: The Fractal Transform, Michael F. Barnsley and Louisa F. Anson ISBN 0-86720-218-1, ca. 250 pp, $49.95 Fractal Image Compression Michael F. Barnsley and Lyman P. Hurd ISBN 0-86720-457-5, ca. 250 pp., $49.95 Copies can be ordered directly from the publisher by sending a message to kpeters@math.harvard.edu with name, address and a Mastercard or Visa card number with expiration date. Barnsley's company is: Iterated Systems, Inc. 5550A Peachtree Parkway, Suite 650 Norcross, GA 30092 tel: 404-840-0310 or 1-800-4FRACTL fax: 404-840-0806 ------------------------------------------------------------------------------ ~Subject: [18] I need specs and source for TIFF and CCITT group 4 Fax Specs for Group 3 and 4 image coding (group 3 is very similar to group 4) are in CCITT (1988) volume VII fascicle VII.3. They are recommendations T.4 and T.6 respectively. There is also an updated spec contained in 1992 recommendations T.1 to T.6. CCITT specs are available by anonymous ftp (see above answer on V.42bis). The T.4 and T.6 specs are in the directory /computing/ccitt/standards/ccitt/1988/ascii, files 7_3_01.txt.Z and 7_3_02.txt.Z respectively. The following paper covers T.4, T.6 and JBIG: "Review of standards for electronic imaging for facsimile systems" in Journal of Electronic Imaging, Vol. 1, No. 1, pp. 5-21, January 1992. Source code can be obtained as part of a TIFF toolkit - TIFF image compression techniques for binary images include CCITT T.4 and T.6: sgi.com:/graphics/tiff/v3.2.tar.Z [192.48.153.1] Contact: sam@sgi.com There is also a companion compressed tar file (v3.0pics.tar.Z) that has sample TIFF image files. A draft of TIFF 6.0 is in TIFF6.ps.Z. Concerning JPEG compression in TIFF 6.0, Tom Lane adds: The TIFF document won't do you much good unless you also have the official JPEG standard. That, you have to buy from ANSI or your national ISO member organization (DIN over there, I suppose). Worse, the TIFF 6.0 spec has a number of serious problems in its JPEG features. A clarification note will probably be needed to ensure that TIFF JPEG files are compatible across different implementations. I can't in good faith recommend that anyone use TIFF-JPEG until these problems are resolved. Software for reading and writing CCITT Group 3 and 4 images is also available in directory merry.cs.monash.edu.au:/pub/alanf/TIFF_FAX (130.194.67.101). Contact: Alan Finlay . See also question 54 below. ------------------------------------------------------------------------------ ~Subject: [19] What is JPEG? JPEG (pronounced "jay-peg") is a standardized image compression mechanism. JPEG stands for Joint Photographic Experts Group, the original name of the committee that wrote the standard. JPEG is designed for compressing either full-color or gray-scale digital images of "natural", real-world scenes. It does not work so well on non-realistic images, such as cartoons or line drawings. JPEG does not handle black-and-white (1-bit-per-pixel) images, nor does it handle motion picture compression. Standards for compressing those types of images are being worked on by other committees, named JBIG and MPEG respectively. Regular JPEG is "lossy", meaning that the image you get out of decompression isn't quite identical to what you originally put in. The algorithm achieves much of its compression by exploiting known limitations of the human eye, notably the fact that small color details aren't perceived as well as small details of light-and-dark. Thus, JPEG is intended for compressing images that will be looked at by humans. If you plan to machine-analyze your images, the small errors introduced by JPEG may be a problem for you, even if they are invisible to the eye. The JPEG standard includes a separate lossless mode, but it is not widely used and does not give nearly as much compression as the lossy mode. Question 75 "Introduction to JPEG" (in part 2 of this FAQ) gives an overview of how JPEG works and provides references for further reading. Also see the JPEG FAQ article, which covers JPEG software and usage hints. The JPEG FAQ is posted regularly in news.answers by Tom Lane . (See question 53 "Where are FAQ lists archived" if this posting has expired at your site.) For JPEG software, see item 15 above. For JPEG hardware, see item 85 in part 3 of this FAQ. ------------------------------------------------------------------------------ ~Subject: [20] I am looking for source of an H.261 codec and MPEG The H.261 spec is available on src.doc.ic.ac.uk in /computing/ccitt/standards/ccitt/1992/h/h261.doc.Z (or h261.rtf.Z). For H.261 hardware, see item 85 in part 3 of this FAQ. from Thierry TURLETTI : We have implemented a software version of H.261 codec. It runs on top of UNIX and X-Windows. The coder uses the simple video capture board "VideoPix" provided by SUN for the SparcStation. The output is directed towards a standard TCP connection, instead of the leased lines or switched circuits for which regular H.261 codecs are designed. This enable us to test video conferences over regular internet connections. We have to polish it a bit, but the first release is now available by anonymous ftp from avahi.inria.fr, in "/pub/h261.tar.Z". from Andy Hung : Public domain UNIX C source code to do both image and image sequence compression and decompression is available by anonymous ftp: MPEG-I havefun.stanford.edu:pub/mpeg/MPEGv1.2.alpha.tar.Z CCITT H.261(P*64) havefun.stanford.edu:pub/p64/P64v1.2.alpha.tar.Z JPEG havefun.stanford.edu:pub/jpeg/JPEGv1.2.beta.tar.Z These codecs operate on raw raster scanned images. A software program to display raw raster-scanned YUV images and image sequences on X grayscale or color monitors is provided by a program in the anonymous ftp directory havefun.stanford.edu pub/cv/CVv1.1.tar.Z. If you are using the codecs above, we recommend that you ftp this file over as well. The source code has been compiled on DEC and SUN workstations. Caution: the P64 codec has not been tested compliant (any available p64 video streams would be much appreciated - please let us know at achung@cs.stanford.edu). The other codecs have been tested with streams from other encoders. We also have some IPB MPEG-I video coded streams in pub/mpeg/*.mpg; and P64 video streams in pub/p64/*.p64 that we have generated using our codecs. For a more complete description see the file havefun.stanford.edu:pub/README. ------------------------------------------------------------------------------ ~Subject: [25] Fast DCT (Discrete Cosine Transform) algorithms Many image compression methods, including the JPEG, MPEG, and H.261 standards, are based on the discrete cosine transform. A good overall introduction to DCT is the book "Discrete Cosine Transform---Algorithms, Advantages, Applications" by K.R. Rao and P. Yip (Academic Press, London, 1990). This has an extensive, though already dated, bibliography. Here are some newer references provided by Tom Lane . Most of these are in IEEE journals or conference proceedings, notably ICASSP = IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing. ICCAS = IEEE Intl. Conf. on Circuits and Systems. DCC = Data Compression Conference. Polynomial Transform Computation of the 2-D DCT, Duhamel & Guillemot, ICASSP '90 p. 1515. A Forward-Mapping Realization of the Inverse DCT, McMillan & Westover, DCC '92 p. 219. A Fast Algorithm for 2-D DCT, Cho, Yun & Lee, ICASSP '91 p. 2197. Fast Algorithm and Implementation of 2-D DCT, Cho & Lee, Tr. CAS v38 p. 297. A DCT Chip based on a new Structured and Computationally Efficient DCT Algorithm, Duhamel, Guillemot & Carlach, ICCAS '90 p. 77. Trade-offs in the Computation of Mono- and Multi-dimensional DCTs, Vetterli, Duhamel & Guillemot, ICASSP '89 p. 999. Practical Fast 1-D DCT Algorithms with 11 Multiplications, Loeffler, Ligtenberg & Moschytz, ICASSP '89 p. 988. New Scaled DCT Algorithms for Fused Multiply/Add Architectures, Linzer & Feig, ICASSP '91 p. 2201. Fast Algorithms for the 2-D Discrete Cosine Transform, Kamangar & Rao, IEEE Tr. Computers, v C-31 p. 899. Fast 2-D Discrete Cosine Transform, Vetterli, ICASSP '85 p. 1538. A Two-Dimensional Fast Cosine Transform, Haque, Tr. ASSP v ASSP-33 p. 1532. Real-Time Parallel and Fully Pipelined 2-D DCT Lattice Structures with Application to HDTV Systems, Chiu & Liu, Tr. CAS for Video Tech, v 2 p. 25. The free JPEG code (jpegsrc.v4.tar.Z) has one of the fastest implementations of the DCT code. It's all in the files jfwddct.c and jrevdct.c (which do the dct and idct, respectively). See item 15 for ftp locations. ------------------------------------------------------------------------------ ~Subject: [26] Are there algorithms and standards for audio compression? Yes. See the introduction to MPEG given in part 2 of this FAQ. A lossless compressor for 8bit and 16bit audio data (.au) is available by anonymous ftp at svr-ftp.eng.cam.ac.uk:/pub/misc/shorten-1.08.tar.Z. It works by using Huffman coding of prediction residuals. Compression is generally better than that obtained by applying general purpose compression utilities to audio files. Copied from the comp.dsp FAQ posted by guido@cwi.nl (Guido van Rossum): Strange though it seems, audio data is remarkably hard to compress effectively. For 8-bit data, a Huffman encoding of the deltas between successive samples is relatively successful. For 16-bit data, companies like Sony and Philips have spent millions to develop proprietary schemes. Public standards for voice compression are slowly gaining popularity, e.g. CCITT G.721 and G.723 (ADPCM at 32 and 24 kbits/sec). (ADPCM == Adaptive Delta Pulse Code Modulation.) Free source code for a *fast* 32 kbits/sec ADPCM (lossy) algorithm is available by ftp from ftp.cwi.nl as /pub/adpcm.shar. (** NOTE: if you are using v1.0, you should get v1.1, released 17-Dec-1992, which fixes a serious bug -- the quality of v1.1 is claimed to be better than uLAW **) (Note that U-LAW and silence detection can also be considered compression schemes.) You can get a G.721/722/723 package by email to teledoc@itu.arcom.ch, with GET ITU-3022 as the *only* line in the body of the message. A note on u-law from Markus Kuhn : u-law (more precisely (greek mu)-law or 5-law if you have an 8-bit ISO terminal) is more an encoding then a compression method, although a 12 to 8 bit reduction is normally part of the encoding. The official definition is CCITT recommendation G.711. If you want to know how to get CCITT documents, check the Standards FAQ posted to news.answers or get the file standards-faq by ftp in directory rtfm.mit.edu:/pub/usenet/news.answers. See also the comp.dsp FAQ for more information on: - The U.S. DoD's Federal-Standard-1016 based 4800 bps code excited linear prediction voice coder version 3.2a (CELP 3.2a) - The U.S. DoD's Federal-Standard-1015/NATO-STANAG-4198 based 2400 bps linear prediction coder version 53 (LPC-10e v53) - Realtime DSP code and hardware for FS-1015 and FS-1016 You can find the comp.dsp FAQ in comp.dsp or news.answers with subject: "FAQ: Audio File Formats" or by ftp on rtfm.mit.edu in /pub/usenet/news.answers/audio-fmts/part1. CELP C code for Sun SPARCs is available for anonymous ftp at furmint.nectar.cs.cmu.edu, in directory celp.audio.compression. Version 3.2a is also in super.org:/pub/celp_3.2a.tar.Z. Recommended reading: Digital Coding of Waveforms: Principles and Applications to Speech and Video. N. S. Jayant and Peter Noll. Prentice-Hall, 1984, ISBN 0-13-211913-7. from Markus Kuhn : One highest quality sound compression format is called ASPEC and has been developped by a team at the Frauenhofer Institut in Erlangen (Germany) and others. ASPEC produces CD like quality and offers several bitrates, one is 128 kbit/s. It is a lossy algorithm that throws away frequencys that aren't registered in the human cochlea in addition to sophisticated entropy coding. The 64 kbit/s ASPEC variant might soon bring hifi quality ISDN phone connections. It has been implemented on standard DSPs. The Layer 3 MPEG audio compression standard now contains what is officially called the best parts of the ASPEC and MUSICAM algorithms. A reference is: K.Brandenburg, G.Stoll, Y.F.Dehery, J.D.Johnston, L.v.d.Kerkhof, E.F.Schroeder: "The ISO/MPEG-Audio Codec: A Generic Standard for Coding of High Quality Digital Audio", 92nd. AES-convention, Vienna 1992, preprint 3336 from Jutta Degener and Carsten Bormann : GSM 06.10 13 kbit/s RPE/LTP speech compression available -------------------------------------------------------- The Communications and Operating Systems Research Group (KBS) at the Technische Universitaet Berlin is currently working on a set of UNIX-based tools for computer-mediated telecooperation that will be made freely available. As part of this effort we are publishing an implementation of the European GSM 06.10 provisional standard for full-rate speech transcoding, prI-ETS 300 036, which uses RPE/LTP (residual pulse excitation/long term prediction) coding at 13 kbit/s. GSM 06.10 compresses frames of 160 13-bit samples (8 kHz sampling rate, i.e. a frame rate of 50 Hz) into 260 bits; for compatibility with typical UNIX applications, our implementation turns frames of 160 16-bit linear samples into 33-byte frames (1650 Bytes/s). The quality of the algorithm is good enough for reliable speaker recognition; even music often survives transcoding in recognizable form (given the bandwidth limitations of 8 kHz sampling rate). Version 1.0 of the implementation is available per anonymous ftp from tub.cs.tu-berlin.de as /pub/tubmik/gsm-1.0.tar.Z. Questions and bug reports should be directed to toast@tub.cs.tu-berlin.de. Note that the distribution is not available via E-mail (please use one of the ftp-via-E-mail servers). from Bob Kimball : I work for Qualcomm Inc. and we are designing a digital cellular telephone system. Our phone uses our variable rate vocoder (QCELP) which is designed for speach and compresses 64Kb/s speach to 8Kb/s through 1Kb/s with 8Kb/s being full rate and 1Kb/s for 1/8 rate speach. It works great for speach. The QCELP process is documented in our Common Air Interface (CAI) which is available for anonymous ftp from lorien.qualcomm.com in /pub/cdma each chapter is a postscript file. The vocoder is described in appendix A. The whole document is quite large. This is the document which is currently going through the TIA standard committee so it is not a final version. The appendix on the vocoder should be almost identical to the final version... whenever that comes out. from Nicola Ferioli : On wsmr-simtel20.army.mil, pd1: VOCPAK10.ZIP Lossless 8-bit sound file compressor VOCPACK is a compressor/decompressor for 8-bit digital sound using a lossless algorithm; it is useful to save disk space without degrading sound quality. It can compress signed and unsigned data, sampled at any rate, mono or stereo. Since the method used is not lossy, it isn't necessary to strip file headers before compressing. VOCPACK was developed for use with .VOC (SoundBlaster) and .WAV (Windows) files, but any 8-bit sound can be compressed since the program takes no assumptions about the file structure. The typical compression ratio obtained goes from 0,8 for files sampled at 11 KHz to 0,4 for 44 Khz files. The best results are obtained with 44 KHz sounds (mono or stereo): general-purpose archivers create files that can be twice longer than the output of VOCPACK. You can obtain smaller values using lossy compressors but if your goal is to keep the sound quality unaltered you should use a lossless program like VOCPACK. ------------------------------------------------------------------------------ ~Subject: [30] My archive is corrupted! The two most common reasons for this are (1) failing to use the magic word "tenex" (when connected to SIMTEL20 and other TOPS20 systems) or "binary" (when connected to UNIX systems) when transferring the file from an ftp site to your host machine. The reasons for this are technical and boring. A synonym for "tenex" is "type L 8", in case your ftp doesn't know what "tenex" means. (2) failing to use an eight-bit binary transfer protocol when transferring the file from the host to your PC. Make sure to set the transfer type to "binary" on both your host machine and your PC. ------------------------------------------------------------------------------ ~Subject: [31] pkunzip reports a CRC error! The portable zip 1.1 contains many workarounds for undocumented restrictions in pkunzip. Compatibility is ensured for pkunzip 1.10 only. All previous versions (pkunzip 1.0x) have too many bugs and cannot be supported. This includes Borland unzip. So if your pkunzip reports a CRC error, check that you are not using an obsolete version. Get either pkzip 2.04g or unzip 5.0p1 (see question 2 above for ftp sites). To generate zip files compatible with pkunzip 1.10, use zip 1.1 (see item 2 above for ftp site). ------------------------------------------------------------------------------ ~Subject: [32] VMS zip is not compatible with pkzip! The problem is most likely in the file transfer program. Many use kermit to transfer zipped files between PC and VMS VAX. The following VMS kermit settings make VMS-ZIP compatible with PKZIP: VMS kermit PC kermit --------------- -------------- Uploading PKZIPped file to be UNZIPped: set fi ty fixed set fi ty bi Downloading ZIPped file to be PKUNZIPped: set fi ty block set fi ty bi If you are not using kermit, transfer a file created by pkzip on MSDOS to VMS, transfer it back to your PC and check that pkunzip can extract it. ------------------------------------------------------------------------------ ~Subject: [33] I have a problem with Stacker! The newsgroup comp.compression is *not* the appropriate place to discuss about one specific program on one specific operating system. Since you have bought a legal copy of Stacker, you have the documentation of your product; please read it. If you can't find the answer in the documentation, please report the problem to the Stac customer support. If you really feel that the net has to know about your problem, please post in one of the MSDOS newsgroups, such as comp.os.msdos.apps or comp.binaries.ibm.pc.d. ------------------------------------------------------------------------------ ~Subject: [50] What is this 'tar' compression program? tar is not a compression program. It just combines several files into one, without compressing them. tar file are often compressed with 'compress', resulting in a .tar.Z file. See question 2, file type .tar.Z. GNU tar has the capability to (de)compress files as well. When you have to archive a lot of very small files, it is often preferable to create a single .tar file and compress it, than to compress the individual files separately. The compression program can thus take advantage of redundancy between separate files. The disadvantage is that you must uncompress the whole .tar file to extract any member. ------------------------------------------------------------------------------ ~Subject: [51] I need a CRC algorithm As its name implies (Cyclic Redundancy Check) a crc adds redundancy whereas the topic of this group is to remove it. But since this question comes up often, here is some code (by Rob Warnock ). The following C code does CRC-32 in BigEndian/BigEndian byte/bit order. That is, the data is sent most significant byte first, and each of the bits within a byte is sent most significant bit first, as in FDDI. You will need to twiddle with it to do Ethernet CRC, i.e., BigEndian/LittleEndian byte/bit order. [Left as an exercise for the reader.] The CRCs this code generates agree with the vendor-supplied Verilog models of several of the popular FDDI "MAC" chips. u_long crc32_table[256]; /* Initialized first time "crc32()" is called. If you prefer, you can * statically initialize it at compile time. [Another exercise.] */ u_long crc32(u_char *buf, int len) { u_char *p; u_long crc; if (!crc32_table[1]) /* if not already done, */ init_crc32(); /* build table */ crc = 0xffffffff; /* preload shift register, per CRC-32 spec */ for (p = buf; len > 0; ++p, --len) crc = (crc << 8) ^ crc32_table[(crc >> 24) ^ *p]; return ~crc; /* transmit complement, per CRC-32 spec */ } /* * Build auxiliary table for parallel byte-at-a-time CRC-32. */ #define CRC32_POLY 0x04c11db7 /* AUTODIN II, Ethernet, & FDDI */ init_crc32() { int i, j; u_long c; for (i = 0; i < 256; ++i) { for (c = i << 24, j = 8; j > 0; --j) c = c & 0x80000000 ? (c << 1) ^ CRC32_POLY : (c << 1); crc32_table[i] = c; } } See also ftp.uni-erlangen.de:/pub/doc/ISO/english/async-HDLC, and the source of all archivers, such as the file makecrc.c in the sources of zip 1.9 (see item 2). ------------------------------------------------------------------------------ ~Subject: [52] What about those people who continue to ask frequently asked questions in spite of the frequently asked questions document? Just send them a polite mail message, referring them to this document. There is no need to flame them on comp.compression. That would just add more noise to this group. Posted answers that are in the FAQ are just as annoying as posted questions that are in the FAQ. ------------------------------------------------------------------------------ ~Subject: [53] Where are FAQ lists archived? Many are crossposted to news.answers. That newsgroup should have a long expiry time at your site; if not, talk to your sysadmin. FAQ lists are available by anonymous FTP from rtfm.mit.edu. The comp.compression FAQ that you are reading is in directory /pub/usenet/news.answers/compression-faq If you don't have FTP access, you can access the archives by mail server. Send an email message to mail-server@rtfm.mit.edu containing the commands send usenet/news.answers/compression-faq/part1 send usenet/news.answers/compression-faq/part2 send usenet/news.answers/compression-faq/part3 For instructions, send an email message to the same address with the words "help" and "index" (no quotes) on separate lines. If you don't get a reply, check your return address, or add a line such as path myname@foo.edu ------------------------------------------------------------------------------ ~Subject: [54] I need specs for graphics formats Have a look in directory /pub/graphics.formats on zamenhof.cs.rice.edu. It contains descriptions of gif, tiff, fits, etc... See also the FAQ list for comp.graphics. See item 53 for an ftp site. ------------------------------------------------------------------------------ ~Subject: [55] Where can I find Lenna and other images? A bunch of standard images (lenna, baboon, cameraman, crowd, moon etc..) are on ftp site eedsp.gatech.edu (130.207.226.2) in directory /database/images. The images are in 256-level grayshades (256x256 pixels, 256 "colors"). [Note: the site ipl.rpi.edu mentioned below keeps changing. Images stay there for a while then disappear. The information is kept in this FAQ in case the image directories become available again.] The site ipl.rpi.edu (128.113.14.50) has standard images in two directories: ipl.rpi.edu:/pub/image/still/usc ipl.rpi.edu:/pub/image/still/canon (The directory /pub/image/sequence has been taken offline because of possible copyright problems.) In each of those directories are the following directories: bgr - 24 bit blue, green, red color - 24 bit red, green, blue gray - 8 bit grayscale uniform weighted gray601 - 8 bit grayscale CCIR-601 weighted And in these directories are the actual images. For example, the popular lena image is in ipl.rpi.edu:/pub/image/still/usc/color/lena # 24 bit RGB ipl.rpi.edu:/pub/image/still/usc/bgr/lena # 24 bit BGR ipl.rpi.edu:/pub/image/still/usc/gray/lena # 8 bit gray All of the images are in Sun rasterfile format. You can use the pbm utilities to convert them to whatever format is most convenient. [pbm is available in ftp.ee.lbl.gov:/pbmplus*.tar.Z]. Questions about the ipl archive should be sent to help@ipl.rpi.edu. There are few gray-scale still images and some raw data of test results available in directory nic.funet.fi:/pub/pics/misc/test-images. Rodney Peck is interested in some method of establishing a canonical ftp database of images but does not have the resources to provide an ftp site for that database. Send suggestions to rodney@balltown.cma.com. Beware: the same image often comes in many different forms, at different resolutions, etc... The original lenna image is 512 wide, 512 high, 8 bits per pel, red, green and blue fields. Gray-scale versions of Lenna have been obtained in two different ways from the original: (1) Using the green field as a gray-scale image, and (2) Doing an RGB->YUV transformation and saving the Y component. Method (1) makes it easier to compare different people's results since everyone's version should be the same using that method. Method (2) produces a more correct image. For the curious: 'lena' or 'lenna' is a digitized Playboy centerfold, from November 1972. (Lenna is the spelling in Playboy, Lena is the Swedish spelling of the name.) Lena Soderberg (ne Sjooblom) was last reported living in her native Sweden, happily married with three kids and a job with the state liquor monopoly. In 1988, she was interviewed by some Swedish computer related publication, and she was pleasantly amused by what had happened to her picture. That was the first she knew of the use of that picture in the computer business. The editorial in the January 1992 issue of Optical Engineering (v. 31 no. 1) details how Playboy has finally caught on to the fact that their copyright on Lenna Sjooblom's photo is being widely infringed. It sounds as if you will have to get permission from Playboy to publish it in the future. Note on the CCITT test images, by Robert Estes : The ccitt files are in ipl.rpi.edu:/image-archive/bitmap/ccitt (128.113.14.50). They are named ccitt-n.ras.Z where n goes from 1 to 8. Each file has an accompanying doc file called ccitt-n.ras.doc which describes the image file. Here's the doc file for ccitt-1.ras: Name ccitt-1.ras Size 1728 x 2376 x 1 Type 1 bit standard format sun rasterfile Keywords binary standard image 1 bit fax Description One of eight images from the standard binary CCITT test image set. This set is commonly used to compare binary image compression techniques. The images are are 1728x2376 pixels. ------------------------------------------------------------------------------ ~Subject: [56] I am looking for a message digest algorithm Look on the ftp site rsa.com, in directory /pub. MD4 and MD5 are there. This question would be more appropriate on sci.crypt. End of part 1 of the comp.compression faq. >From comp.compression Mon Aug 16 15:59:15 1993 Xref: hobbit.gandalf.ca comp.compression:7992 comp.compression.research:840 news.answers:10828 comp.answers:1540 Path: hobbit.gandalf.ca!nott!torn!howland.reston.ans.net!math.ohio-state.edu!magnus.acs.ohio-state.edu!usenet.ins.cwru.edu!agate!doc.ic.ac.uk!uknet!mcsun!julienas!chorus!chorus.fr From: jloup@chorus.fr (Jean-loup Gailly) Newsgroups: comp.compression,comp.compression.research,news.answers,comp.answers Subject: comp.compression Frequently Asked Questions (part 2/3) Summary: *** READ THIS BEFORE POSTING *** Keywords: data compression, FAQ Message-ID: Date: 10 Aug 93 20:55:14 GMT Expires: 30 Sep 93 16:17:20 GMT References: Sender: news@chorus.chorus.fr Reply-To: jloup@chorus.fr Followup-To: comp.compression Lines: 1106 Approved: news-answers-request@MIT.Edu Supersedes: Archive-name: compression-faq/part2 Last-modified: June 21st, 1993 This file is part 2 of a set of Frequently Asked Questions for the groups comp.compression and comp.compression.research. If you did not get part 1 or 3, you can get them by ftp on rtfm.mit.edu in directory /pub/usenet/news.answers/compression-faq If you don't want to see this FAQ regularly, please add the subject line to your kill file. If you have corrections or suggestions for this FAQ, send them to Jean-loup Gailly . Thank you. Contents ======== Part 2: (Long) introductions to data compression techniques [70] Introduction to data compression (long) Huffman and Related Compression Techniques Arithmetic Coding Substitutional Compressors The LZ78 family of compressors The LZ77 family of compressors [71] Introduction to MPEG (long) What is MPEG? Does it have anything to do with JPEG? Then what's JBIG and MHEG? What has MPEG accomplished? So how does MPEG I work? What about the audio compression? So how much does it compress? What's phase II? When will all this be finished? How do I join MPEG? How do I get the documents, like the MPEG I draft? [72] What is wavelet theory? [73] What is the theoretical compression limit? [74] Introduction to JBIG [75] Introduction to JPEG Part 3: (Long) list of image compression hardware [85] Image compression hardware [99] Acknowledgments Search for "Subject: [#]" to get to question number # quickly. Some news readers can also take advantage of the message digest format used here. ------------------------------------------------------------------------------ ~Subject: [70] Introduction to data compression (long) Written by Peter Gutmann . Huffman and Related Compression Techniques ------------------------------------------ *Huffman compression* is a statistical data compression technique which gives a reduction in the average code length used to represent the symbols of a alphabet. The Huffman code is an example of a code which is optimal in the case where all symbols probabilities are integral powers of 1/2. A Huffman code can be built in the following manner: (1) Rank all symbols in order of probability of occurrence. (2) Successively combine the two symbols of the lowest probability to form a new composite symbol; eventually we will build a binary tree where each node is the probability of all nodes beneath it. (3) Trace a path to each leaf, noticing the direction at each node. For a given frequency distribution, there are many possible Huffman codes, but the total compressed length will be the same. It is possible to define a 'canonical' Huffman tree, that is, pick one of these alternative trees. Such a canonical tree can then be represented very compactly, by transmitting only the bit length of each code. This technique is used in most archivers (pkzip, lha, zoo, arj, ...). A technique related to Huffman coding is *Shannon-Fano coding*, which works as follows: (1) Divide the set of symbols into two equal or almost equal subsets based on the probability of occurrence of characters in each subset. The first subset is assigned a binary zero, the second a binary one. (2) Repeat step (1) until all subsets have a single element. The algorithm used to create the Huffman codes is bottom-up, and the one for the Shannon-Fano codes is top-down. Huffman encoding always generates optimal codes, Shannon-Fano sometimes uses a few more bits. Arithmetic Coding ----------------- It would appear that Huffman or Shannon-Fano coding is the perfect means of compressing data. However, this is *not* the case. As mentioned above, these coding methods are optimal when and only when the symbol probabilities are integral powers of 1/2, which is usually not the case. The technique of *arithmetic coding* does not have this restriction: It achieves the same effect as treating the message as one single unit (a technique which would, for Huffman coding, require enumeration of every single possible message), and thus attains the theoretical entropy bound to compression efficiency for any source. Arithmetic coding works by representing a number by an interval of real numbers between 0 and 1. As the message becomes longer, the interval needed to represent it becomes smaller and smaller, and the number of bits needed to specify that interval increases. Successive symbols in the message reduce this interval in accordance with the probability of that symbol. The more likely symbols reduce the range by less, and thus add fewer bits to the message. 1 Codewords +-----------+-----------+-----------+ /-----\ | |8/9 YY | Detail |<- 31/32 .11111 | +-----------+-----------+<- 15/16 .1111 | Y | | too small |<- 14/16 .1110 |2/3 | YX | for text |<- 6/8 .110 +-----------+-----------+-----------+ | | |16/27 XYY |<- 10/16 .1010 | | +-----------+ | | XY | | | | | XYX |<- 4/8 .100 | |4/9 | | | +-----------+-----------+ | | | | | X | | XXY |<- 3/8 .011 | | |8/27 | | | +-----------+ | | XX | | | | | |<- 1/4 .01 | | | XXX | | | | | |0 | | | +-----------+-----------+-----------+ As an example of arithmetic coding, lets consider the example of two symbols X and Y, of probabilities 0.66 and 0.33. To encode this message, we examine the first symbol: If it is a X, we choose the lower partition; if it is a Y, we choose the upper partition. Continuing in this manner for three symbols, we get the codewords shown to the right of the diagram above - they can be found by simply taking an appropriate location in the interval for that particular set of symbols and turning it into a binary fraction. In practice, it is also necessary to add a special end-of-data symbol, which is not represented in this simpe example. In this case the arithmetic code is not completely efficient, which is due to the shortness of the message - with longer messages the coding efficiency does indeed approach 100%. Now that we have an efficient encoding technique, what can we do with it? What we need is a technique for building a model of the data which we can then use with the encoder. The simplest model is a fixed one, for example a table of standard letter frequencies for English text which we can then use to get letter probabilities. An improvement on this technique is to use an *adaptive model*, in other words a model which adjusts itself to the data which is being compressed as the data is compressed. We can convert the fixed model into an adaptive one by adjusting the symbol frequencies after each new symbol is encoded, allowing the model to track the data being transmitted. However, we can do much better than that. Using the symbol probabilities by themselves is not a particularly good estimate of the true entropy of the data: We can take into account intersymbol probabilities as well. The best compressors available today take this approach: DMC (Dynamic Markov Coding) starts with a zero-order Markov model and gradually extends this initial model as compression progresses; PPM (Prediction by Partial Matching) looks for a match of the text to be compressed in an order-n context. If no match is found, it drops to an order n-1 context, until it reaches order 0. Both these techniques thus obtain a much better model of the data to be compressed, which, combined with the use of arithmetic coding, results in superior compression performance. So if arithmetic coding-based compressors are so powerful, why are they not used universally? Apart from the fact that they are relatively new and haven't come into general use too much yet, there is also one major concern: The fact that they consume rather large amounts of computing resources, both in terms of CPU power and memory. The building of sophisticated models for the compression can chew through a fair amount of memory (especially in the case of DMC, where the model can grow without bounds); and the arithmetic coding itself involves a fair amount of number crunching. There is however an alternative approach, a class of compressors generally referred to as *substitutional* or *dictionary-based compressors*. Substitutional Compressors -------------------------- The basic idea behind a substitutional compressor is to replace an occurrence of a particular phrase or group of bytes in a piece of data with a reference to a previous occurrence of that phrase. There are two main classes of schemes, named after Jakob Ziv and Abraham Lempel, who first proposed them in 1977 and 1978. LZ78-based schemes work by entering phrases into a *dictionary* and then, when a repeat occurrence of that particular phrase is found, outputting the dictionary index instead of the phrase. There exist several compression algorithms based on this principle, differing mainly in the manner in which they manage the dictionary. The most well-known scheme (in fact the most well-known of all the Lempel-Ziv compressors, the one which is generally (and mistakenly) referred to as "Lempel-Ziv Compression"), is Terry Welch's LZW scheme, which he designed in 1984 for implementation in hardware for high- performance disk controllers. Input string: /WED/WE/WEE/WEB Character input: Code output: New code value and associated string: /W / 256 = /W E W 257 = WE D E 258 = ED / D 259 = D/ WE 256 260 = /WE / E 261 = E/ WEE 260 262 = /WEE /W 261 263 = E/W EB 257 264 = WEB B LZW starts with a 4K dictionary, of which entries 0-255 refer to individual bytes, and entries 256-4095 refer to substrings. Each time a new code is generated it means a new string has been parsed. New strings are generated by appending the current character K to the end of an existing string w. The algorithm for LZW compression is as follows: set w = NIL loop read a character K if wK exists is in the dictionary w = wK else output the code for w add wK to the string table w = K endloop A sample run of LZW over a (highly redundant) input string can be seen in the diagram above. The strings are built up character-by-character starting with a code value of 256. LZW decompression takes the stream of codes and uses it to exactly recreate the original input data. Just like the compression algorithm, the decompressor adds a new string to the dictionary each time it reads in a new code. All it needs to do in addition is to translate each incoming code into a string and send it to the output. A sample run of the LZW decompressor is shown in below. Input code: /WED<256>E<260><261><257>B Input code: Output string: New code value and associated string: / / W W 256 = /W E E 257 = WE D D 258 = ED 256 /W 259 = D/ E E 260 = /WE 260 /WE 261 = E/ 261 E/ 262 = /WEE 257 WE 263 = E/W B B 264 = WEB The most remarkable feature of this type of compression is that the entire dictionary has been transmitted to the decoder without actually explicitly transmitting the dictionary. At the end of the run, the decoder will have a dictionary identical to the one the encoder has, built up entirely as part of the decoding process. LZW is more commonly encountered today in a variant known as LZC, after its use in the UNIX "compress" program. In this variant, pointers do not have a fixed length. Rather, they start with a length of 9 bits, and then slowly grow to their maximum possible length once all the pointers of a particular size have been used up. Furthermore, the dictionary is not frozen once it is full as for LZW - the program continually monitors compression performance, and once this starts decreasing the entire dictionary is discarded and rebuilt from scratch. More recent schemes use some sort of least-recently-used algorithm to discard little-used phrases once the dictionary becomes full rather than throwing away the entire dictionary. Finally, not all schemes build up the dictionary by adding a single new character to the end of the current phrase. An alternative technique is to concatenate the previous two phrases (LZMW), which results in a faster buildup of longer phrases than the character-by-character buildup of the other methods. The disadvantage of this method is that a more sophisticated data structure is needed to handle the dictionary. [A good introduction to LZW, MW, AP and Y coding is given in the yabba package. For ftp information, see question 2 in part one, file type .Y] LZ77-based schemes keep track of the last n bytes of data seen, and when a phrase is encountered that has already been seen, they output a pair of values corresponding to the position of the phrase in the previously-seen buffer of data, and the length of the phrase. In effect the compressor moves a fixed-size *window* over the data (generally referred to as a *sliding window*), with the position part of the (position, length) pair referring to the position of the phrase within the window. The most commonly used algorithms are derived from the LZSS scheme described by James Storer and Thomas Szymanski in 1982. In this the compressor maintains a window of size N bytes and a *lookahead buffer* the contents of which it tries to find a match for in the window: while( lookAheadBuffer not empty ) { get a pointer ( position, match ) to the longest match in the window for the lookahead buffer; if( length > MINIMUM_MATCH_LENGTH ) { output a ( position, length ) pair; shift the window length characters along; } else { output the first character in the lookahead buffer; shift the window 1 character along; } } Decompression is simple and fast: Whenever a ( position, length ) pair is encountered, go to that ( position ) in the window and copy ( length ) bytes to the output. Sliding-window-based schemes can be simplified by numbering the input text characters mod N, in effect creating a circular buffer. The sliding window approach automatically creates the LRU effect which must be done explicitly in LZ78 schemes. Variants of this method apply additional compression to the output of the LZSS compressor, which include a simple variable-length code (LZB), dynamic Huffman coding (LZH), and Shannon-Fano coding (ZIP 1.x)), all of which result in a certain degree of improvement over the basic scheme, especially when the data are rather random and the LZSS compressor has little effect. Recently an algorithm was developed which combines the ideas behind LZ77 and LZ78 to produce a hybrid called LZFG. LZFG uses the standard sliding window, but stores the data in a modified trie data structure and produces as output the position of the text in the trie. Since LZFG only inserts complete *phrases* into the dictionary, it should run faster than other LZ77-based compressors. All popular archivers (arj, lha, zip, zoo) are variations on the LZ77 theme. ------------------------------------------------------------------------------ ~Subject: [71] Introduction to MPEG (long) For MPEG players, see item 15 in part 1 of the FAQ. Frank Gadegast also posts a FAQ specialized in MPEG. Introduction written by Mark Adler around January 1992 (and hence wildly out of date in this fast moving area--any volunteers to update this section are welcome): Q. What is MPEG? A. MPEG is a group of people that meet under ISO (the International Standards Organization) to generate standards for digital video (sequences of images in time) and audio compression. In particular, they define a compressed bit stream, which implicitly defines a decompressor. However, the compression algorithms are up to the individual manufacturers, and that is where proprietary advantage is obtained within the scope of a publicly available international standard. MPEG meets roughly four times a year for roughly a week each time. In between meetings, a great deal of work is done by the members, so it doesn't all happen at the meetings. The work is organized and planned at the meetings. Q. So what does MPEG stand for? A. Moving Pictures Experts Group. Q. Does it have anything to do with JPEG? A. Well, it sounds the same, and they are part of the same subcommittee of ISO along with JBIG and MHEG, and they usually meet at the same place at the same time. However, they are different sets of people with few or no common individual members, and they have different charters and requirements. JPEG is for still image compression. Q. Then what's JBIG and MHEG? A. Sorry I mentioned them. Ok, I'll simply say that JBIG is for binary image compression (like faxes), and MHEG is for multi-media data standards (like integrating stills, video, audio, text, etc.). For an introduction to JBIG, see question 74 below. Q. Ok, I'll stick to MPEG. What has MPEG accomplished? A. So far (as of January 1992), they have completed the "Committee Draft" of MPEG phase I, colloquially called MPEG I. It defines a bit stream for compressed video and audio optimized to fit into a bandwidth (data rate) of 1.5 Mbits/s. This rate is special because it is the data rate of (uncompressed) audio CD's and DAT's. The draft is in three parts, video, audio, and systems, where the last part gives the integration of the audio and video streams with the proper timestamping to allow synchronization of the two. They have also gotten well into MPEG phase II, whose task is to define a bitstream for video and audio coded at around 3 to 10 Mbits/s. Q. So how does MPEG I work? A. First off, it starts with a relatively low resolution video sequence (possibly decimated from the original) of about 352 by 240 frames by 30 frames/s (US--different numbers for Europe), but original high (CD) quality audio. The images are in color, but converted to YUV space, and the two chrominance channels (U and V) are decimated further to 176 by 120 pixels. It turns out that you can get away with a lot less resolution in those channels and not notice it, at least in "natural" (not computer generated) images. The basic scheme is to predict motion from frame to frame in the temporal direction, and then to use DCT's (discrete cosine transforms) to organize the redundancy in the spatial directions. The DCT's are done on 8x8 blocks, and the motion prediction is done in the luminance (Y) channel on 16x16 blocks. In other words, given the 16x16 block in the current frame that you are trying to code, you look for a close match to that block in a previous or future frame (there are backward prediction modes where later frames are sent first to allow interpolating between frames). The DCT coefficients (of either the actual data, or the difference between this block and the close match) are "quantized", which means that you divide them by some value to drop bits off the bottom end. Hopefully, many of the coefficients will then end up being zero. The quantization can change for every "macroblock" (a macroblock is 16x16 of Y and the corresponding 8x8's in both U and V). The results of all of this, which include the DCT coefficients, the motion vectors, and the quantization parameters (and other stuff) is Huffman coded using fixed tables. The DCT coefficients have a special Huffman table that is "two-dimensional" in that one code specifies a run-length of zeros and the non-zero value that ended the run. Also, the motion vectors and the DC DCT components are DPCM (subtracted from the last one) coded. Q. So is each frame predicted from the last frame? A. No. The scheme is a little more complicated than that. There are three types of coded frames. There are "I" or intra frames. They are simply a frame coded as a still image, not using any past history. You have to start somewhere. Then there are "P" or predicted frames. They are predicted from the most recently reconstructed I or P frame. (I'm describing this from the point of view of the decompressor.) Each macroblock in a P frame can either come with a vector and difference DCT coefficients for a close match in the last I or P, or it can just be "intra" coded (like in the I frames) if there was no good match. Lastly, there are "B" or bidirectional frames. They are predicted from the closest two I or P frames, one in the past and one in the future. You search for matching blocks in those frames, and try three different things to see which works best. (Now I have the point of view of the compressor, just to confuse you.) You try using the forward vector, the backward vector, and you try averaging the two blocks from the future and past frames, and subtracting that from the block being coded. If none of those work well, you can intra- code the block. The sequence of decoded frames usually goes like: IBBPBBPBBPBBIBBPBBPB... Where there are 12 frames from I to I (for US and Japan anyway.) This is based on a random access requirement that you need a starting point at least once every 0.4 seconds or so. The ratio of P's to B's is based on experience. Of course, for the decoder to work, you have to send that first P *before* the first two B's, so the compressed data stream ends up looking like: 0xx312645... where those are frame numbers. xx might be nothing (if this is the true starting point), or it might be the B's of frames -2 and -1 if we're in the middle of the stream somewhere. You have to decode the I, then decode the P, keep both of those in memory, and then decode the two B's. You probably display the I while you're decoding the P, and display the B's as you're decoding them, and then display the P as you're decoding the next P, and so on. Q. You've got to be kidding. A. No, really! Q. Hmm. Where did they get 352x240? A. That derives from the CCIR-601 digital television standard which is used by professional digital video equipment. It is (in the US) 720 by 243 by 60 fields (not frames) per second, where the fields are interlaced when displayed. (It is important to note though that fields are actually acquired and displayed a 60th of a second apart.) The chrominance channels are 360 by 243 by 60 fields a second, again interlaced. This degree of chrominance decimation (2:1 in the horizontal direction) is called 4:2:2. The source input format for MPEG I, called SIF, is CCIR-601 decimated by 2:1 in the horizontal direction, 2:1 in the time direction, and an additional 2:1 in the chrominance vertical direction. And some lines are cut off to make sure things divide by 8 or 16 where needed. Q. What if I'm in Europe? A. For 50 Hz display standards (PAL, SECAM) change the number of lines in a field from 243 or 240 to 288, and change the display rate to 50 fields/s or 25 frames/s. Similarly, change the 120 lines in the decimated chrominance channels to 144 lines. Since 288*50 is exactly equal to 240*60, the two formats have the same source data rate. Q. You didn't mention anything about the audio compression. A. Oh, right. Well, I don't know as much about the audio compression. Basically they use very carefully developed psychoacoustic models derived from experiments with the best obtainable listeners to pick out pieces of the sound that you can't hear. There are what are called "masking" effects where, for example, a large component at one frequency will prevent you from hearing lower energy parts at nearby frequencies, where the relative energy vs. frequency that is masked is described by some empirical curve. There are similar temporal masking effects, as well as some more complicated interactions where a temporal effect can unmask a frequency, and vice-versa. The sound is broken up into spectral chunks with a hybrid scheme that combines sine transforms with subband transforms, and the psychoacoustic model written in terms of those chunks. Whatever can be removed or reduced in precision is, and the remainder is sent. It's a little more complicated than that, since the bits have to be allocated across the bands. And, of course, what is sent is entropy coded. Q. So how much does it compress? A. As I mentioned before, audio CD data rates are about 1.5 Mbits/s. You can compress the same stereo program down to 256 Kbits/s with no loss in discernable quality. (So they say. For the most part it's true, but every once in a while a weird thing might happen that you'll notice. However the effect is very small, and it takes a listener trained to notice these particular types of effects.) That's about 6:1 compression. So, a CD MPEG I stream would have about 1.25 MBits/s left for video. The number I usually see though is 1.15 MBits/s (maybe you need the rest for the system data stream). You can then calculate the video compression ratio from the numbers here to be about 26:1. If you step back and think about that, it's little short of a miracle. Of course, it's lossy compression, but it can be pretty hard sometimes to see the loss, if you're comparing the SIF original to the SIF decompressed. There is, however, a very noticeable loss if you're coming from CCIR-601 and have to decimate to SIF, but that's another matter. I'm not counting that in the 26:1. The standard also provides for other bit rates ranging from 32Kbits/s for a single channel, up to 448 Kbits/s for stereo. Q. What's phase II? A. As I said, there is a considerable loss of quality in going from CCIR-601 to SIF resolution. For entertainment video, it's simply not acceptable. You want to use more bits and code all or almost all the CCIR-601 data. From subjective testing at the Japan meeting in November 1991, it seems that 4 MBits/s can give very good quality compared to the original CCIR-601 material. The objective of phase II is to define a bit stream optimized for these resolutions and bit rates. Q. Why not just scale up what you're doing with MPEG I? A. The main difficulty is the interlacing. The simplest way to extend MPEG I to interlaced material is to put the fields together into frames (720x486x30/s). This results in bad motion artifacts that stem from the fact that moving objects are in different places in the two fields, and so don't line up in the frames. Compressing and decompressing without taking that into account somehow tends to muddle the objects in the two different fields. The other thing you might try is to code the even and odd field streams separately. This avoids the motion artifacts, but as you might imagine, doesn't get very good compression since you are not using the redundancy between the even and odd fields where there is not much motion (which is typically most of image). Or you can code it as a single stream of fields. Or you can interpolate lines. Or, etc. etc. There are many things you can try, and the point of MPEG II is to figure out what works well. MPEG II is not limited to consider only derivations of MPEG I. There were several non-MPEG I-like schemes in the competition in November, and some aspects of those algorithms may or may not make it into the final standard for entertainment video compression. Q. So what works? A. Basically, derivations of MPEG I worked quite well, with one that used wavelet subband coding instead of DCT's that also worked very well. Also among the worked-very-well's was a scheme that did not use B frames at all, just I and P's. All of them, except maybe one, did some sort of adaptive frame/field coding, where a decision is made on a macroblock basis as to whether to code that one as one frame macroblock or as two field macroblocks. Some other aspects are how to code I-frames--some suggest predicting the even field from the odd field. Or you can predict evens from evens and odds or odds from evens and odds or any field from any other field, etc. Q. So what works? A. Ok, we're not really sure what works best yet. The next step is to define a "test model" to start from, that incorporates most of the salient features of the worked-very-well proposals in a simple way. Then experiments will be done on that test model, making a mod at a time, and seeing what makes it better and what makes it worse. Example experiments are, B's or no B's, DCT vs. wavelets, various field prediction modes, etc. The requirements, such as implementation cost, quality, random access, etc. will all feed into this process as well. Q. When will all this be finished? A. I don't know. I'd have to hope in about a year or less. Q. How do I join MPEG? A. You don't join MPEG. You have to participate in ISO as part of a national delegation. How you get to be part of the national delegation is up to each nation. I only know the U.S., where you have to attend the corresponding ANSI meetings to be able to attend the ISO meetings. Your company or institution has to be willing to sink some bucks into travel since, naturally, these meetings are held all over the world. (For example, Paris, Santa Clara, Kurihama Japan, Singapore, Haifa Israel, Rio de Janeiro, London, etc.) Q. Well, then how do I get the documents, like the MPEG I draft? A. MPEG is a draft ISO standard. It's exact name is ISO CD 11172. The draft consists of three parts: System, Video, and Audio. The System part (11172-1) deals with synchronization and multiplexing of audio-visual information, while the Video (11172-2) and Audio part (11172-3) address the video and the audio compression techniques respectively. You may order it from your national standards body (e.g. ANSI in the USA) or buy it from companies like OMNICOM phone +44 438 742424 FAX +44 438 740154 ------------------------------------------------------------------------------ ~Subject: [72] What is wavelet theory? Preprints and software are available by anonymous ftp from the Yale Mathematics Department computer ceres.math.yale.edu[130.132.23.22], in pub/wavelets and pub/software. epic and hcompress are wavelet coders. (For source code, see item 15 in part one). Bill Press of Harvard/CfA has made some things available for anonymous ftp on cfata4.harvard.edu [128.103.40.79] in directory /pub. There is a short TeX article on wavelet theory (wavelet.tex, to be included in a future edition of Numerical Recipes), some sample wavelet code (wavelet.f, in FORTRAN - sigh), and a beta version of an astronomical image compression program which he is currently developing (FITS format data files only, in fitspress08.tar.Z). A mailing list dedicated to research on wavelets has been set up at the University of South Carolina. To subscribe to this mailing list, send a message with "subscribe" as the subject to wavelet@math.scarolina.edu. A 5 minute course in wavelet transforms, by Richard Kirk : Do you know what a Haar transform is? Its a transform to another orthonormal space (like the DFT), but the basis functions are a set of square wave bursts like this... +--+ +------+ + | +------------------ + | +-------------- +--+ +------+ +--+ +------+ ------+ | +------------ --------------+ | + +--+ +------+ +--+ +-------------+ ------------+ | +------ + | + +--+ +-------------+ +--+ +---------------------------+ ------------------+ | + + + +--+ This is the set of functions for an 8-element 1-D Haar transform. You can probably see how to extend this to higher orders and higher dimensions yourself. This is dead easy to calculate, but it is not what is usually understood by a wavelet transform. If you look at the eight Haar functions you see we have four functions that code the highest resolution detail, two functions that code the coarser detail, one function that codes the coarser detail still, and the top function that codes the average value for the whole `image'. Haar function can be used to code images instead of the DFT. With bilevel images (such as text) the result can look better, and it is quicker to code. Flattish regions, textures, and soft edges in scanned images get a nasty `blocking' feel to them. This is obvious on hardcopy, but can be disguised on color CRTs by the effects of the shadow mask. The DCT gives more consistent results. This connects up with another bit of maths sometimes called Multispectral Image Analysis, sometimes called Image Pyramids. Suppose you want to produce a discretely sampled image from a continuous function. You would do this by effectively `scanning' the function using a sinc function [ sin(x)/x ] `aperture'. This was proved by Shannon in the `forties. You can do the same thing starting with a high resolution discretely sampled image. You can then get a whole set of images showing the edges at different resolutions by differencing the image at one resolution with another version at another resolution. If you have made this set of images properly they ought to all add together to give the original image. This is an expansion of data. Suppose you started off with a 1K*1K image. You now may have a 64*64 low resolution image plus difference images at 128*128 256*256, 512*512 and 1K*1K. Where has this extra data come from? If you look at the difference images you will see there is obviously some redundancy as most of the values are near zero. From the way we constructed the levels we know that locally the average must approach zero in all levels but the top. We could then construct a set of functions out of the sync functions at any level so that their total value at all higher levels is zero. This gives us an orthonormal set of basis functions for a transform. The transform resembles the Haar transform a bit, but has symmetric wave pulses that decay away continuously in either direction rather than square waves that cut off sharply. This transform is the wavelet transform ( got to the point at last!! ). These wavelet functions have been likened to the edge detecting functions believed to be present in the human retina. Loren I. Petrich adds that order 2 or 3 Daubechies discrete wavelet transforms have a speed comparable to DCT's, and usually achieve compression a factor of 2 better for the same image quality than the JPEG 8*8 DCT. (See item 25 in part 1 of this FAQ for references on fast DCT algorithms.) ------------------------------------------------------------------------------ ~Subject: [73] What is the theoretical compression limit? There is no compressor that is guaranteed to compress all possible input files. If it compresses some files, then it must enlarge some others. This can be proven by a simple counting argument (see question 9). As an extreme example, the following algorithm achieves 100% compression for one special input file and enlarges all other files by only one bit: - if the input data is , output an empty file. - otherwise output one bit (zero or one) followed by the input data. The concept of theoretical compression limit is meaningful only if you have a model for your input data. See question 70 above for some examples of data models. ------------------------------------------------------------------------------ ~Subject: [74] Introduction to JBIG A short introduction, written by Mark Adler : JBIG losslessly compresses binary (one-bit/pixel) images. (The B stands for bi-level.) Basically it models the redundancy in the image as the correlations of the pixel currently being coded with a set of nearby pixels called the template. An example template might be the two pixels preceding this one on the same line, and the five pixels centered above this pixel on the previous line. Note that this choice only involves pixels that have already been seen from a scanner. The current pixel is then arithmetically coded based on the eight-bit (including the pixel being coded) state so formed. So there are (in this case) 256 contexts to be coded. The arithmetic coder and probability estimator for the contexts are actually IBM's (patented) Q-coder. The Q-coder uses low precision, rapidly adaptable (those two are related) probability estimation combined with a multiply-less arithmetic coder. The probability estimation is intimately tied to the interval calculations necessary for the arithmetic coding. JBIG actually goes beyond this and has adaptive templates, and probably some other bells and whistles I don't know about. You can find a description of the Q-coder as well as the ancestor of JBIG in the Nov 88 issue of the IBM Journal of Research and Development. This is a very complete and well written set of five articles that describe the Q-coder and a bi-level image coder that uses the Q-coder. You can use JBIG on grey-scale or even color images by simply applying the algorithm one bit-plane at a time. You would want to recode the grey or color levels first though, so that adjacent levels differ in only one bit (called Gray-coding). I hear that this works well up to about six bits per pixel, beyond which JPEG's lossless mode works better. You need to use the Q-coder with JPEG also to get this performance. Actually no lossless mode works well beyond six bits per pixel, since those low bits tend to be noise, which doesn't compress at all. Anyway, the intent of JBIG is to replace the current, less effective group 3 and 4 fax algorithms. Another introduction to JBIG, written by Hank van Bekkem : The following description of the JBIG algorithm is derived from experiences with a software implementation I wrote following the specifications in the revision 4.1 draft of September 16, 1991. The source will not be made available in the public domain, as parts of JBIG are patented. JBIG (Joint Bi-level Image Experts Group) is an experts group of ISO, IEC and CCITT (JTC1/SC2/WG9 and SGVIII). Its job is to define a compression standard for lossless image coding ([1]). The main characteristics of the proposed algorithm are: - Compatible progressive/sequential coding. This means that a progressively coded image can be decoded sequentially, and the other way around. - JBIG will be a lossless image compression standard: all bits in your images before and after compression and decompression will be exactly the same. In the rest of this text I will first describe the JBIG algorithm in a short abstract of the draft. I will conclude by saying something about the value of JBIG. JBIG algorithm. -------------- JBIG parameter P specifies the number of bits per pixel in the image. Its allowable range is 1 through 255, but starting at P=8 or so, compression will be more efficient using other algorithms. On the other hand, medical images such as chest X-rays are often stored with 12 bits per pixel, while no distorsion is allowed, so JBIG can certainly be of use in this area. To limit the number of bit changes between adjacent decimal values (e.g. 127 and 128), it is wise to use Gray coding before compressing multi-level images with JBIG. JBIG then compresses the image on a bitplane basis, so the rest of this text assumes bi-level pixels. Progressive coding is a way to send an image gradually to a receiver instead of all at once. During sending, more detail is sent, and the receiver can build the image from low to high detail. JBIG uses discrete steps of detail by successively doubling the resolution. The sender computes a number of resolution layers D, and transmits these starting at the lowest resolution Dl. Resolution reduction uses pixels in the high resolution layer and some already computed low resolution pixels as an index into a lookup table. The contents of this table can be specified by the user. Compatibility between progressive and sequential coding is achieved by dividing an image into stripes. Each stripe is a horizontal bar with a user definable height. Each stripe is separately coded and transmitted, and the user can define in which order stripes, resolutions and bitplanes (if P>1) are intermixed in the coded data. A progressive coded image can be decoded sequentially by decoding each stripe, beginning by the one at the top of the image, to its full resolution, and then proceeding to the next stripe. Progressive decoding can be done by decoding only a specific resolution layer from all stripes. After dividing an image into bitplanes, resolution layers and stripes, eventually a number of small bi-level bitmaps are left to compress. Compression is done using a Q-coder. Reference [2] contains a full description, I will only outline the basic principles here. The Q-coder codes bi-level pixels as symbols using the probability of occurrence of these symbols in a certain context. JBIG defines two kinds of context, one for the lowest resolution layer (the base layer), and one for all other layers (differential layers). Differential layer contexts contain pixels in the layer to be coded, and in the corresponding lower resolution layer. For each combination of pixel values in a context, the probability distribution of black and white pixels can be different. In an all white context, the probability of coding a white pixel will be much greater than that of coding a black pixel. The Q-coder assigns, just like a Huffman coder, more bits to less probable symbols, and so achieves compression. The Q-coder can, unlike a Huffmann coder, assign one output codebit to more than one input symbol, and thus is able to compress bi-level pixels without explicit clustering, as would be necessary using a Huffman coder. Maximum compression will be achieved when all probabilities (one set for each combination of pixel values in the context) follow the probabilities of the pixels. The Q-coder therefore continuously adapts these probabilities to the symbols it sees. JBIG value. ---------- In my opinion, JBIG can be regarded as two combined devices: - Providing the user the service of sending or storing multiple representations of images at different resolutions without any extra cost in storage. Differential layer contexts contain pixels in two resolution layers, and so enable the Q-coder to effectively code the difference in information between the two layers, instead of the information contained in every layer. This means that, within a margin of approximately 5%, the number of resolution layers doesn't effect the compression ratio. - Providing the user a very efficient compression algorithm, mainly for use with bi-level images. Compared to CCITT Group 4, JBIG is approximately 10% to 50% better on text and line art, and even better on halftones. JBIG is however, just like Group 4, somewhat sensitive to noise in images. This means that the compression ratio decreases when the amount of noise in your images increases. An example of an application would be browsing through an image database, e.g. an EDMS (engineering document management system). Large A0 size drawings at 300 dpi or so would be stored using five resolution layers. The lowest resolution layer would fit on a computer screen. Base layer compressed data would be stored at the beginning of the compressed file, thus making browsing through large numbers of compressed drawings possible by reading and decompressing just the first small part of all files. When the user stops browsing, the system could automatically start decompressing all remaining detail for printing at high resolution. [1] "Progressive Bi-level Image Compression, Revision 4.1", ISO/IEC JTC1/SC2/WG9, CD 11544, September 16, 1991 [2] "An overview of the basic principles of the Q-coder adaptive binary arithmetic coder", W.B. Pennebaker, J.L. Mitchell, G.G. Langdon, R.B. Arps, IBM Journal of research and development, Vol.32, No.6, November 1988, pp. 771-726 (See also the other articles about the Q-coder in this issue) ------------------------------------------------------------------------------ ~Subject: [75] Introduction to JPEG Here is a brief overview of the inner workings of JPEG, plus some references for more detailed information, written by Tom Lane . Please read item 19 in part 1 first. JPEG works on either full-color or gray-scale images; it does not handle bilevel (black and white) images, at least not efficiently. It doesn't handle colormapped images either; you have to pre-expand those into an unmapped full-color representation. JPEG works best on "continuous tone" images; images with many sudden jumps in color values will not compress well. There are a lot of parameters to the JPEG compression process. By adjusting the parameters, you can trade off compressed image size against reconstructed image quality over a *very* wide range. You can get image quality ranging from op-art (at 100x smaller than the original 24-bit image) to quite indistinguishable from the source (at about 3x smaller). Usually the threshold of visible difference from the source image is somewhere around 10x to 20x smaller than the original, ie, 1 to 2 bits per pixel for color images. Grayscale requires a little bit less space. JPEG defines a "baseline" lossy algorithm, plus optional extensions for progressive and hierarchical coding. There is also a separate lossless compression mode; this typically gives about 2:1 compression, ie about 12 bits per color pixel. Most currently available JPEG hardware and software handles only the baseline mode. Here's the outline of the baseline compression algorithm: 1. Transform the image into a suitable color space. This is a no-op for grayscale, but for color images you generally want to transform RGB into a luminance/chrominance color space (YCbCr, YUV, etc). The luminance component is grayscale and the other two axes are color information. The reason for doing this is that you can afford to lose a lot more information in the chrominance components than you can in the luminance component; the human eye is not as sensitive to high-frequency color info as it is to high-frequency luminance. (See any TV system for precedents.) You don't have to change the color space if you don't want to, as the remainder of the algorithm works on each color component independently, and doesn't care just what the data is. However, compression will be less since you will have to code all the components at luminance quality. 2. (Optional) Downsample each component by averaging together groups of pixels. The luminance component is left at full resolution, while the color components are usually reduced 2:1 horizontally and either 2:1 or 1:1 (no change) vertically. In JPEG-speak these alternatives are usually called 2h2v and 2h1v sampling, but you may also see the terms "411" and "422" sampling. This step immediately reduces the data volume by one-half or one-third, while having almost no impact on perceived quality. (Obviously this would not be true if you tried it in RGB color space...) Note that downsampling is not applicable to gray-scale data. 3. Group the pixel values for each component into 8x8 blocks. Transform each 8x8 block through a discrete cosine transform (DCT); this is a relative of the Fourier transform and likewise gives a frequency map, with 8x8 components. Thus you now have numbers representing the average value in each block and successively higher-frequency changes within the block. The motivation for doing this is that you can now throw away high-frequency information without affecting low-frequency information. (The DCT transform itself is reversible except for roundoff error.) See question 25 for fast DCT algorithms. 4. In each block, divide each of the 64 frequency components by a separate "quantization coefficient", and round the results to integers. This is the fundamental information-losing step. A Q.C. of 1 loses no information; larger Q.C.s lose successively more info. The higher frequencies are normally reduced much more than the lower. (All 64 Q.C.s are parameters to the compression process; tuning them for best results is a black art. It seems likely that the best values are yet to be discovered. Most existing coders use simple multiples of the example tables given in the JPEG standard.) 5. Encode the reduced coefficients using either Huffman or arithmetic coding. (Strictly speaking, baseline JPEG only allows Huffman coding; arithmetic coding is an optional extension.) Notice that this step is lossless, so it doesn't affect image quality. The arithmetic coding option uses Q-coding; it is identical to the coder used in JBIG (see question 74). Be aware that Q-coding is patented. Most existing implementations support only the Huffman mode, so as to avoid license fees. The arithmetic mode offers maybe 5 or 10% better compression, which isn't enough to justify paying fees. 6. Tack on appropriate headers, etc, and output the result. In an "interchange" JPEG file, all of the compression parameters are included in the headers so that the decompressor can reverse the process. For specialized applications, the spec permits the parameters to be omitted from the file; this saves several hundred bytes of overhead, but it means that the decompressor must know what parameters the compressor used. The decompression algorithm reverses this process, and typically adds some smoothing steps to reduce pixel-to-pixel discontinuities. Extensions: The progressive mode is intended to support real-time transmission of images. It allows the DCT coefficients to be sent incrementally in multiple "scans" of the image. With each scan, the decoder can produce a higher-quality rendition of the image. Thus a low-quality preview can be sent very quickly, then refined as time allows. Notice that the decoder must do essentially a full JPEG decode cycle for each scan, so this scheme is useful only with fast decoders (meaning dedicated hardware, at least at present). However, the total number of bits sent can actually be somewhat less than is necessary in the baseline mode, especially if arithmetic coding is used. So progressive coding might be useful even if the decoder will simply save up the bits and make only one output pass. The hierarchical mode represents an image at multiple resolutions. For example, one could provide 512x512, 1024x1024, and 2048x2048 versions of the image. The higher-resolution images are coded as differences from the next smaller image, and thus require many fewer bits than they would if stored independently. (However, the total number of bits will be greater than that needed to store just the highest-resolution frame.) Note that the individual frames in a hierarchical sequence may be coded progressively if desired. Lossless JPEG: The separate lossless mode does not use DCT, since roundoff errors prevent a DCT calculation from being lossless. For the same reason, one would not normally use colorspace conversion or downsampling, although these are permitted by the standard. The lossless mode simply codes the difference between each pixel and the "predicted" value for the pixel. The predicted value is a simple function of the already-transmitted pixels just above and to the left of the current one (eg, their average; 8 different predictor functions are permitted). The sequence of differences is encoded using the same back end (Huffman or arithmetic) used in the lossy mode. The main reason for providing a lossless option is that it makes a good adjunct to the hierarchical mode: the final scan in a hierarchical sequence can be a lossless coding of the remaining differences, to achieve overall losslessness. This isn't quite as useful as it may at first appear, because exact losslessness is not guaranteed unless the encoder and decoder have identical IDCT implementations (ie identical roundoff errors). ~References: For a good technical introduction to JPEG, see: Wallace, Gregory K. "The JPEG Still Picture Compression Standard", Communications of the ACM, April 1991 (vol. 34 no. 4), pp. 30-44. (Adjacent articles in that issue discuss MPEG motion picture compression, applications of JPEG, and related topics.) If you don't have the CACM issue handy, a PostScript file containing a revised version of this article is available at ftp.uu.net, graphics/jpeg/wallace.ps.Z. The file (actually a preprint for an article to appear in IEEE Trans. Consum. Elect.) omits the sample images that appeared in CACM, but it includes corrections and some added material. Note: the Wallace article is copyright ACM and IEEE, and it may not be used for commercial purposes. An alternative, more leisurely explanation of JPEG can be found in "The Data Compression Book" by Mark Nelson ([Nel 1991], see question 7). This book provides excellent introductions to many data compression methods including JPEG, plus sample source code in C. The JPEG-related source code is far from industrial-strength, but it's a pretty good learning tool. An excellent textbook about JPEG is "JPEG Still Image Data Compression Standard" by William B. Pennebaker and Joan L. Mitchell. Published by Van Nostrand Reinhold, 1993, ISBN 0-442-01272-1. 650 pages, price US$59.95. (VNR will accept credit card orders at 800/842-3636, or get your local bookstore to order it.) This book includes the complete text of the ISO JPEG standards, DIS 10918-1 and draft DIS 10918-2. Review by Tom Lane: "This is by far the most complete exposition of JPEG in existence. It's written by two people who know what they are talking about: both serve on the ISO JPEG standards committee. If you want to know how JPEG works or why it works that way, this is the book to have." There are a number of errors in the first printing of the Pennebaker & Mitchell book. An errata list is available at ftp.uu.net: graphics/jpeg/pm.errata. At last report, all were fixed in the second printing. The official specification of JPEG is not currently available on-line. I hear that CCITT specs may be on-line sometime soon, which would change this. At the moment, your best bet is to buy the Pennebaker and Mitchell textbook. >From comp.compression Mon Aug 16 15:59:24 1993 Xref: hobbit.gandalf.ca comp.compression:7993 comp.compression.research:841 news.answers:10829 comp.answers:1541 Path: hobbit.gandalf.ca!nott!torn!howland.reston.ans.net!math.ohio-state.edu!magnus.acs.ohio-state.edu!usenet.ins.cwru.edu!agate!doc.ic.ac.uk!uknet!mcsun!julienas!chorus!chorus.fr From: jloup@chorus.fr (Jean-loup Gailly) Newsgroups: comp.compression,comp.compression.research,news.answers,comp.answers Subject: comp.compression Frequently Asked Questions (part 3/3) Summary: *** READ THIS BEFORE POSTING *** Keywords: data compression, FAQ Message-ID: Date: 10 Aug 93 20:55:21 GMT Expires: 30 Sep 93 16:17:20 GMT References: Sender: news@chorus.chorus.fr Reply-To: jloup@chorus.fr Followup-To: comp.compression Lines: 556 Approved: news-answers-request@MIT.Edu Supersedes: Archive-name: compression-faq/part3 Last-modified: Aug 10th, 1993 This file is part 3 of a set of Frequently Asked Questions for the groups comp.compression and comp.compression.research. If you did not get part 1 or 2, you can get them by ftp on rtfm.mit.edu in directory /pub/usenet/news.answers/compression-faq If you don't want to see this FAQ regularly, please add the subject line to your kill file. If you have corrections or suggestions for this FAQ, send them to Jean-loup Gailly . Thank you. Contents ======== Part 3: (Long) list of image compression hardware [85] Image compression hardware [99] Acknowledgments Search for "Subject: [#]" to get to question number # quickly. Some news readers can also take advantage of the message digest format used here. ------------------------------------------------------------------------------ ~Subject: [85] Image compression hardware Here is a list of sources of image compression hardware (JPEG, MPEG, H.261 and others), reposted with the author's permission. The list is probably a little dated already, but it is a good starting point for seeking compression chips. (Please send corrections/additions to jloup@chorus.fr). References are taken from: VIDEO COMPRESSION OPTIONS, IEEE CICC 6-May-92 John J. Bloomer, jbloomer@crd.ge.com, Fathy F. Yassa, Aiman A. Abdel-Malek General Electric Corporate R&D, KWC317 Signals and Systems Laboratory PO Box 8, Schenectady NY, 12301 (Too many people have sent comments, corrections or additions so I am just making a common acknowledgment here.) Pipelined Processors, Building Blocks (Chip Sets) ------------------------------------------------- STI3200, IMSA121, STI3208 - SGS-Thompson DCT processors. 602-867-6279 - 3200 has multiple block size options, DC to 13.5 MHz - A121 8x8 fixed blocks, DC to 20MHz, add/sub loop, CCITT compatible - 3208 8x8 fixed blocks, DC to 40MHz, CCITT compatible at 20MHz STI3220 - SGS-Thompson motion estimator (H.261, MPEG). 602-867-6279 - 8-bit input pixels, 4-bit H and V vectors out - adjustable block size matcher (8x8, 8x16, 16x16) - +7/-8 search window - 5V, 2W at 18MHz (max), 68 pin PLCC L64765 , L64735 , L64745 - 3-chip LSI Logic JPEG set. 408-433-4383 - L64765 raster-to-block and color-space converter, jointly developed with Rapid Tech. - L64735 block DCT processor - L64745 JPEG coder support, stand-alone lossless DPCM codec, dynamic Huffman - 27 MB/s on CCIR601 frames - minimal support logic, color and gray scale - 68-pin PGA or PLCC, 27 and 20 MHz versions L647*0 and L6471* families - LSI Logic H.621/MPEG pieces. 408-433-8000 - L64720 motion estimator, 30/40MHz, 8x8, 16x16 blocks, 32x32 or 16x16 search window, 68-pin CPGA or PPGA - L64730 & 735 8x8 DCT processors (12 & 8-9 bits) - L64740 8x8 block quantization - L64760 intra/inter-frame coding decision - L64715 BCH error correction - L64750/L64751 variable length encode/decode (H.261-specific) ZR36020 and ZR36031 - Zoran DCT processor & quantization/encoding. 408-986-1314 - JPEG-like scheme using 16-bit, two's complement fixed point arithmetic - includes bit-rate controls for constant # of pictures per card - 7.4 MHz, < 1W, 20mW in standby mode, 7.5 frames/s (f/s) - 36020 - 44-pin plastic quad flatpack (PQFP) or 48-pin ceramic DIP - 36031 - 100-pin PQFP or 85-pin PGA. - co-developments with Fuji Photo Film Co. Ltd. digital IC-card camera market Does 2-passes of image: generate histogram for optimum Huffman tables and quantization compute step size (ala H.261 and MPEG-I) for each macroblock or minimum coded unit (MCU). JPEG-compatible codec expected soon. LDM3104 - Olympus DCT coefficient encoder - constant rate, digital IC-card camera market - 750 mW, 25 mW standby, 100-poin QFP TMC2312 - TRW quantizer/Huffman encoder, TMC2313 Huffman decoder/dequantizer TMC2311 - TRW CMOS Fast Cosine Transform Processor. - 12 Bits, 15 M pixels/s - complies with the CCITT SGXV ( e.g. JPEG, H.261 and MPEG ) - includes an adder-subtractor for linear predictive coding MN195901 - Matsushita Electric Industrial Co. See ISSCC 1992 - 16-bit, 60 MIP video signal processor - 25 uS instruction processing - on-board DCT and absolute differencing - Philips Signetics US fab. HGCT - Ricoh CRC, Generalized Chen Transform demonstration chip. 408-281-1436 - 2D JPEG/MPEG/H.261 compatible DCT - includes quantization - 30MHz, 15K gates - licensing possible GCTX64000 - Graphic Communication Technology Corp. chipset - provides CCITT H.261 - VLSI Technology and Hitachi supply H.261 codec core. 1 micron CMOS. BT - British Telecommunications plc., Martlesham labs designed - H.261 codec chipset, Motorola fab. - 13 chips total for codec. Pipelined Processors, Monolithic, Programmable ---------------------------------------------- Vision Processor - Integrated Information Technology Inc. 408-727-1885 - generic DCT, motion compensated & entropy coding codec - microcode for still- and motion-video compression (JPEG, H.261 and MPEG1) - 1 micron CMOS, 20 MHz and 33 MHz, PGA and 84-pin QFP - JPEG only and JPEG/H/261/MPEG versions available, H.261 at 30 f/s. - used by Compression Labs, Inc. CDV teleconferencing system - rumored to be the heart of the AT&T picture phone MN195901 - Matsushita Electric Industrial Corp - 40 MHz DSP, built-in DCT - 16-bit fixed-point AVP1000 - AT&T JPEG, MPEG and H.261 codec chipset. 800-372-2447 - 1400D decoder, 1400C system controller - 1300E H.261 (CIF, QCIF, CIF240) at 30 f/s, I-frame only MPEG. - 1400E is superset of 1300E, motion with 1/2 pixel resolution over +/- 32 pixels - YCbCr video or digital input, on-board rate FIFOs, external RAM required - 0.75 micron, 50 MHz CMOS AVP1000 is from AT&T Microelectronics. The AT&T chip set handles MPEG-1, H.261, and JPEG. 1400D has on board color space convertor. Limited to 4Mb/s coded rate. The DSP does the MUSICAM decoding (up to layer II ?) 82750PB, 82750DB - Intel DVI pixel and display YUV color space processors. - proprietary machine code employed for compression - usable for other algorithms (e.g., JPEG, H.261 or MPEG1 at reduced data rates) Pipelined Processors, Monolithic, Fixed Lossless - Entropy Coders, DPCM, VQ --------------------------------------------------------------------------- DCP - Integrated Information Tech. Inc. Data Compressor Processor 408-727-1885 - LZ codec with on-chip dictionary store - on-chip buffers supporting block moves - targeting disk drives and network controller markets - 3.3V, 84-pin PQFP Mystic - HP's DC-LZ codec. 408-749-9500 AHA3210 - Advanced Hardware Architectures DC-LZ codec. 208-883-8000 - two independent DMA ports for 10 MB/s compress, decompress & pass-thru - addressing allows up to 16 MB record compression - 20 MHz internal clock, 200 mW, 100-pin PQFP - interface to AHA5101/5121 QIC tape controller/formatter - HP licensee AHA3xxx/xxy - Rice (UNC) algorithm, 20M samples/sec, 4 to 14 bits. 208-883-8000 CRM1000 - CERAM Inc. entropy codec, proprietary algorithm. 719-540-8500 Rice - UNC algorithm prototype, 180 Mb/s. See IEEE CICC 1992 - other CICC 1992 papers: +JS.E. Kerneny et.al. differential read, pyramidal output CCD + A. Aggoun et.al. DPCM processing DCD - Philips Data Compressor Decompressor IC. 914-945-6000 - See CICC 1990 proceedings, H. Blume, et.al. - LZ codec, 20 MHz clock - Internal FIFOs, separate input/output buses, max 10 Mword/s data in - 5 V CMOS, 175-pin PGA 9705 - Second generation Stac Electronics accelerator chip. 619-431-7474 - Stacker LZA compression scheme(LZ-based) - compress at approx. 2.5 MB/s, decompress at 6 MB/s (39+ faster than 9704) - standby mode 300uA - embedded in tapes and disks (e.g., QIC-122 Ten X Technology 512-346-8360) - file compression board & software: + for the PC/AT - from Stac + for the Macintosh - from Sigma Design 415-770-0100 (40 MHz 9703) - InfoChip Systems Inc. - proprietary string-matching technology 408-727-0514 VCEP or OTI95C71/Am95C71 - Oak Technology Inc. 408-737-0888 - AMD CCITT B&W fax image compression Pipelined Processors, Monolithic, Fixed Lossy --------------------------------------------- MB86356B - Fujitsu LTD. - JPEG DIS 10918-1 baseline codec - on-chip quantizer tables - 2.5M pixel/sec input, up to 10MB/sec output - supports progressive and DPCM lossless modes - 135 pin PGA. CL550-30 - C-Cube Microsystems 408-944-6300 - JPEG-8-R2 compliant baseline codec - 350-level pipeline, on-chip Huffman and quantizer table - 44.1 MB/sec (15 MB/sec for -10) - RGB, YUV, CMYK supported, CCIR 601 in real-time - 16/32-bit host interface - 144 pin PGA or QFP, 2.5W at 29.41 MHz Limited to 2MB/sec (15Mb/s) coded rate. 35MHz PGA version available. 2:1 horizontal filter, on board programmable color space convertor. Allows on pair of quantization tables to be loaded while other pair is used to code or decode data stream. Needs maintanence by host. STI140 - SGS-Thompson JPEG baseline codec. 617-259-0300 [** Now cancelled **] - see CICC 1991 proceedings, M. Bolton. - 20 Mpixel/sec input, up to 20 MB/sec output - supports 24-bit color, 8-bit grey and 12-bit extended pixels - on chip Huffman and quantizer tables - 144 pin PQFP, 5V, < 2W., 10mW power-down mode - 1.2 micron, 3-layer metal CMOS, 20 MHz. ` UVC7710 - UVC Corp. Integrated Multimedia Processor. Was 714-261-5336, out of business now. - proprietary, patented intra-frame compression, on-chip code tables - 20-35:1, 12.5 Mpixels/sec., compressed audio - includes much of the PC-AT (16-bit ISA) bus interface logic - 128 pin PJQFP plastic CL950 - C-Cube/JVC implementation of the MPEG-JVC or extended mode MPEG2 announced. 6-9 Mb/sec. JVC mode is not MPEG-II compliant (there isn't an MPEG2 standard yet) but is an extension of MPEG1 at a higher rate plus interlace video handling. CL450 - Announced June 1992. Scaled down version of CL950, with 3Mb/sec limit. Does not code or decode JPEG, only MPEG-I decoding. CD-I - ASICs planned for CD-ROM, Compact Disk-Interactive defacto standards - CD-ROM XA - Sony-Philips-Motorola-Microsoft - CDTV - Commodore. YUV processing. - audio ADPCM encode/decode PC/AT boards available from Sony 408-432-0190 Codecs Chips Under Development ------------------------------ MPEG1 codec chips due from - TI, Brooktree, Cypress Semiconductor, Motorola (successor to the DSP96002 Multimedia Engine), Xing Technology/Analog Devices, Sony and C-Cube Windbond Electronics Corp. is developing a DSP chip for CD-I, MPEG and JPEG Using these Chips: Board Level Compression Hardware ------------------------------------------------------ + JPEG Using CL550 + JPEG Using Other Chip Sets + DSP Chip Based JPEG/MPEG Solutions + Integrated Compressed Digital Video Boards JPEG Using CL550 --------------- C-Cube - 408-944-6300 ISA and NuBus boards - for development and limited time-constraint applications - 1-2.5 MB/sec host bus constraints - Image Compression Interface (ISI) software for 3rd party CL550 integration VideoSpigot/SuperSqueeze - SuperMac Technology 408-245-2202 - a CL550A on a NuBus board - 24 frame/s with CD-quality audio - reads from Winchester and magneto-optic drives Fluency VSA-1000 - Fluent Machines, Inc. AT board set. 508 626-2144 - compress/decompress real-time synced audio & video to a i386 PC Winchester - NTSC or PAL input, 320x240 pixels saved - uses i960 chip, no additional boards needed - M/S Windows support, 3rd party S/W (e.g., AimTech 603-883-0220) Super Motion Compression - New Media Graphics PC/AT board. 800 288-2207 - 8Khz, 8-bit compressed audio - 30 f/s JPEG to & from disk - earlier reports: still-frame compression in several seconds per MB Leadview - Lead Tech Inc. AT board uses the CL550 to compress/decompress JFIF or JTIF format files Monalisa - Opta Inc. AT board uses the CL550 Squeeze - Rapid Technology AT board - Integrated by a number of vendors into 3rd party multimedia, video-editing PC stations Parallax Graphics - SBus, VME and PC-AT boards. 408-727-2220 or info@parallax.com Chips and Technologies - JPEG development kit due. JPEG Using Other Chipsets ------------------------- Visionary - Rapid Technology JPEG AT board. 716-833-8534 - LSI Logic JPEG chips L647-35, -45 & -65 - 30 f/s motion JPEG - 256x240 pixel compression and display from CCIR-601 input - private codec-frame buffer bus - also integrated with TrueVision multimedia hardware Media 100 - Data Translation nonlinear video production system for the Macintosh (QuickTime). 22 MB/s (PAL) and 18MB/s (NTSC) throughput. Alice - Telephoto Communications Inc. 619-452-0903 - Alice-H350 (PC/AT) and -H365 (PS/2) codec boards - use a 40 MHz TMS320C51 DSP and a IMSA121 DCT processor chip - JPEG (lossy and lossless), CCITT G3/G4, color and grey-scale images Xing Technology - Hardware accelerator. 805-473-0145 - compatible with their VT-Express JPEG Turbo Accelerator Software Video/1 - PsiTech Inc. 714-968-7818 - includes a 6U VME/VSB JPEG Processing Card - compresses RS-170, NTSC, PAL or Secam video into 8 MB of on-board RAM DSP Chip Based JPEG/MPEG Solutions ---------------------------------- Optipac - Optivision Inc. PC/AT, ISA & VME codecs. 800-562-8934 - JPEG (lossless and lossy), CCITT III/IV - 1 to 5 TMS32C025s - 512x400x16-bit images in < 1 sec. XCeed ICDP-II - Micron Technology Inc. NuBus card - uses two AT&T Microelectronics DSP-16 DSP chips - driven by Storm Technologies PicturePress software - executes an enhanced JPEG algorithm at near-realtime. PicturePress Accelerator - Storm Technology 415-691-1111 (see above) - also has a line of VME compression boards - Micro Dynamics Ltd. imaging systems use Storm accelerator 301-589-6300 Picture Packer Accelerator - Video & Image Compression Corp. - AT and NuBus boards use the JPEG Open Standard and a TMS320C25 VideoPix - Software JPEG boards are offered by Sun Microsystems (S-Bus). Phoenix System - T/one Inc. uses an Optivision Optipac 3250 to talk to a Storm Technologies NuBus PicturePress Accelerator to talk JPEG over analog phone lines. Nextdimension - NeXt Computer Inc. 415-780-3912 - 24+8-bit alpha, 640x480, 30 f/s decompression - CL550 version not shipping as announced. Spirit-40 - Sonitech International Inc. ISA card. 617-235-6824 - two TMS320C40 DSPs for 80 MFLOPS - connect 16 boards in a hypercube for up to 1280 MFLOPS - JPEG, MPEG-1 audio and other voice coding applications included HardPak - CERAM Inc., ISA and EISA file compression board. 719-540-8500 - 3.4 x 1.8 inch footprint (notebook, laptops) - 32KB on-board write-thru file compression cache - CERAM also has an SBus compressive swap-space accelerator for Suns macDSP - Spectral Innovations, AT&T DSPC32-based accelerator. 408-727-1314 - JPEG functions available - 30 MFLOPS on the NuBus VCA-1 - Video compression accelerator for Sun workstations. Tel: (310)829-7733, FAX: (310)829-1694, Internet: spacecc@cerf.net Special-Purpose Hardware for Motion Estimation and DCTs Performs 8x8 DCTs in 21 microsec after first DCT at 52 microsec.* Performs 32x32 cross search for 16x16 block in 239 microsec.* (*Stated times are for a 25-MHz SBus.) Mounts in a single SBus slot. Included software allows user-transparent access. Price: $2,900 (subject to change without notice). Integrated Digital Video Boards - Miscellaneous Multimedia, Video Conferencing ------------------------------------------------------------------------------ VCI/oem - Vista Communication Instruments, Inc. +358 0 460 099 - two AT-board H.261 video codec, PAL or NTSC cameras and monitors -56 kbps (64 kbps) to 2 Mbps, 64 kbps increments - H.221 framing and synchronizing - H.241 network signalling - H.200/AV.254 forthcoming standard for compressed audio - network interface boards available MediaStation- VideoLogic Inc., JPEG compression board for ISA bus. 617-494-0530 - works with VideoLogic DVA-4000/ISA motion video board, custom bus - CL-550 plus ADPCM and PCM audio support - Inmos Transputer for I/O scheduling - Microsoft Windows Multimedia Extensions and proprietary interfaces DECspin - Digital Equipment CorpSound/Picture Information Network 508-493-5111 - full motion, true-color (24-bit) and greyscale (8-bit black & white) - variable frame size and rate up to 640 x480 x30 NTSC true-color - Internet or DECnet transmission and disk I/O of live synchronized video/audio - video teleconferencing using standard network protocols - create and edit of audio and video sequences - voice grade live audio sequences - DECmedia DECvideo and DECaudio hardware and software required ActionMedia II - Intel/IBM DVI PS/2 and PC/AT boards. 914-642-5472 - i750 processor boards for capture and delivery systems - Microsoft programming support libraries - proprietary RTV and PLV compression algorithms resident, time and time/space VQ - contact Intel Literature Center for #B4P-05 algorithm structure - Real Time Video (RTV) algorithm 1.5 , effective 128x120 pixel sequence at 30 f/s. - RTV 1.0 is 128x240 at 10 f/s. - Presentation Level Video (PLV) - extensive off-line processing, exploits inter-frame coherence. - i750 processor capable of playing-back PLV-compressed 256x240 sequences at 30 f/s. DVI Board - Fast Electronic U.S. Inc. laptop board. 508-655-3278 - uses Intel i750 chipset - compress or decompress video at up to 30 f/s EyeQ - New Video Corp. DVI boards for the Macintosh. 213-396-0282 - uses Intel i750 chipset - 150 KB/s full-motion compressed video - T1 and Winchester integration paths Copernicus 1000 & 2000 - DesignTech, 408-453-9510 - DVI-based presentation and authoring systems Spectrum Signal Processing - DSP96002-based PC-AT board - up to four boards in cascade - other TI, Analog Devices and AT&T-based DSP offerings Ariel Corp. - Dual DSP96002 PC-AT board with compression support. 201-429-2900 Capture I - UVC Corp., 16-bit ISA bus board. was 714-261-5336, out of business now. - 30 f/s of 640/480 interlace capture and record (uses UVC7710) - NTSC or PAL input - VPC200/201 development board set - proprietary NTSC video codec (audio card required). Leadview - Lead Technologies, Inc. accelerates an enhanced JPEG algorithm on ISA IBM - near-term availability: (1) IBM United Kingdom and British Telecommunications plc. - PC or PS/2 add-on boards by end of 1993 - interface to ISDN 2 service (one or two 64kb/s channels) - BT also planning residential videophone product with GEC Marconi Ltd. (2) IBM Japan PS/2 board - uses GCTX64000 for H.261 - ISDN (narrowband 64kb/s ) and IEEE 802.5 LAN interfaces Optibase 100 - Optibase, Inc. DSP-based compression/expansion boards. 818-719-6566 - supports JPEG - supports CCITT G.721 and ANSI T1.301 & T1.303 drafts (voice and music) - and proprietary compression (AADCT, lossless) Motorola - DSP56002 (fixed-point 40MHz version of the 56001) AT&T JPEG coder (George Warner ) - runs on a DSP3210 under the VCOS operating system. The coder can be used to simultaneously compress/decompress multiple images and/or be used in conjunction with other DSP modules to preprocess or postprocess the image data. Other modules available for the DSP3210 include audio coders (such as MPEG, SBC, CDXA, and G.722), modem/fax data pumps (V.32bis, V.22bis, and V.29), DTMF, call progress detection, sample rate conversion, and more. MWave - TI, IBM, Intermetrics multimedia system, due from IBM in 1993. Misc. NuBus boards - RasterOps , Radius, Mass Microsystems, Orange Micro, IBM M - - Motion. P.OEM - Interated Systems Inc. fractal compression boards for the PC. 404-840-0310 two desktop video conferencing products for Sparc's with the Parallax XVIDEO board: Communique! - desktop video conferencing products for Sparcs with the Parallax XVIDEO board: InSoft, Inc., 4718 Old Gettsburg Road, Executive Park West I, Suite 307 Mechanicsburg, PA 17055, USA. email: info@insoft.com phone: 717-766-6290, fax: 717-766-8866 PSVC - desktop video conferencing products for Sparcs with the Parallax XVIDEO board: Paradise Software, Inc., 55 Princeton Heightstown Rd, Suite 109 Princeton, NJ 08550, USA. email: support@paradise.com phone: 609-275-4475, fax: 609-275-4702 North Valley Research - video and other time-based media in a UNIX environment North Valley Research; 15262 NW Greenbriar Pkwy; Beaverton, OR 97006 Phone (503) 531-5707, Fax (503) 690-2320. Todd Brunhoff Boards Under Development ------------------------ Matrox - Matrox Studio line of PC boards will include a 64-bit MOVIE bus and JPEG compression. ------------------------------------------------------------------------------ ~Subject: [99] Acknowledgments There are too many people to cite. Thanks to all people who directly or indirectly contributed to this FAQ. -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Nov 5 14:16:16 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovZQb-0000YSa; Fri, 5 Nov 93 14:14 PST Sender: owner-ppp-comp X-Path: donut.gandalf.ca!dcarr From: dcarr@gandalf.ca (Dave Carr) To: ppp-comp@bungi.com Subject: Re: clearing hash tables Date: Fri, 5 Nov 1993 16:11:18 -0500 (EST) Message-ID: <9311052111.AA10844@donut.gandalf.ca> References: <<9311051850.AA27385@rhyolite.wpd.sgi.com>> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk You didn't answer my question :-) > > Do you mean crosschecking that the hash actually matches the data now > > in the hash table, and is not an old entry that wasn't deleted? > >From your description, you notion and mine of the UNIX-compress hash > table are not exactly the same, so that your description does not > exactly fit what I'm talking about. Still, you obviously have the > idea. Yes. It's been a while since I looked at Unix Compress. Open address chaining or something like that. Damn fancy data structures again. Had to look that one up in Knuth if I remember. Obviously wasn't written by an engineer :-) > If you could build a hash table with a load-factor of 1.0000, then > you could clear the table even faster. Wouldn't you lose more time when compressing to save this time while clearing? -- Dave Carr | dcarr@gandalf.ca | It's what you learn, Principal Designer | TEL (613) 723-6500 | after you know it all, Gandalf Data Limited | FAX (613) 226-1717 | that counts. From owner-ppp-comp Fri Nov 5 16:08:47 1993 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0ovbBp-0000esa; Fri, 5 Nov 93 16:07 PST Sender: owner-ppp-comp X-Path: rhyolite.wpd.sgi.com!vjs From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) To: ppp-comp@bungi.com Subject: Re: clearing hash tables Date: Fri, 5 Nov 93 17:05:18 -0700 Message-ID: <9311060005.AA29204@rhyolite.wpd.sgi.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk > Dave Carr | dcarr@gandalf.ca > > You didn't answer my question :-) Oh? Sorry. I tried to answer affirmatively without being inaccurate. > ... > Yes. It's been a while since I looked at Unix Compress. Open address > chaining or something like that. Damn fancy data structures again. > Had to look that one up in Knuth if I remember. Obviously wasn't > written by an engineer :-) The UNIX-compress source is rather nasty. The comments and general shape look as if it has been heavily hacked in about two dozen 2-hour sessions by a dozen different people over a dozen years. Exactly how it works was hard for my aging neurons to figure out, although it is clear once you understand it. The source is due for an overhaul and cleanup; it assumes int's may be short's and that only long's 32 bits and that 32 bits is a big deal. However, the data structures including the hash table are plain and simple. The source even includes pointers to the standard references. > > If you could build a hash table with a load-factor of 1.0000, then > > you could clear the table even faster. > > Wouldn't you lose more time when compressing to save this time while > clearing? That was a joke. A load factor of 1 affects the expected value of the number of probes per look up. Until I remembered the implications of the minor detail that the load facter is always < 1, I had a really neat scheme for clearing the table involving toggling a single bit in a single variable once for each clearing. I wish I knew someone with the time and inclination now to work one of the public domain PPP implementations or the BSDI BSD/386 1.0 PPP code. I think I could provide Predictor 1 and UNIX-compress source for STREAMS or mbufs to someone with the time and inclination to add the CCP state machine and user-option parsing to one of those. I know I could provided source for mixed STREAMS and mbufs such as used in the SGI kernel, but no one else needs such. The obvious, nicely aligned, 8-byte version of the LZW hash table+dictionary needed for the receiver has one byte of padding. If the longest string encoded could be limited to 257 bytes, then the structure could include with each code the length of its string. That would save a byte-copy during decompression, since you could know where to start laying down the decoded bytes in reverse order. It would also remove a consistency test from the main loop of decoding. (Unlike the user-code utility, in the kernel you may not want to depend on a segmentation violation to detect bogus codes from the transmitter.) Could people be convinced to warp UNIX-compress that much? Without profiling, it looks to me that most cycles in LZW are spent getting and putting the fractional-byte codes. There are faster and cleaner ways to code in C the getcode() and output() functions, but they're always going to be nasty. Vernon Schryver, vjs@sgi.com From owner-ppp-comp Mon Jan 3 07:44:21 1994 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0pGrQY-0000Wxa; Mon, 3 Jan 94 07:42 PST Sender: owner-ppp-comp X-Path: tri-flow.ftp.com!craig From: craig@tri-flow.ftp.com (Craig Fox) To: ppp-comp@bungi.com Subject: Re: List changes Date: Mon, 3 Jan 94 10:42:43 EST Message-ID: <9401031542.AA26446@tri-flow.ftp.com.ftp.com> Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk Were these discussions archived anywhere safe? I want to know if I can delete all of my message copies. Craig > From owner-ppp-comp@daver.bungi.com Fri Nov 5 13:21:39 1993 > X-Path: dlr > From: dlr@daver.bungi.com (Dave Rand) > To: ppp-comp@bungi.com > Subject: List changes > Date: Fri, 5 Nov 1993 10:10:39 PST > Reply-To: ppp-comp@bungi.com > > Effective following this message, the ppp-comp mailing list will be gatewayed > to the mail ietf-ppp mailing list. If you are on the main list, you don't > need to be on this one any more. Send your removal request to: > > ppp-comp-request@bungi.com > > and I'll be happy to take you off. It was good to meet most of you > in person at Houston, and I'm looking forward to releasing the > LAPB and compression documents next week. > > > -- > Dave Rand > Internet: dlr@daver.bungi.com From owner-ppp-comp Wed Jan 5 08:34:01 1994 Return-Path: Received: by daver.bungi.com (Smail3.1.28.1 #1) id m0pHbAL-0000WXa; Wed, 5 Jan 94 08:33 PST Sender: owner-ppp-comp X-Path: dlr From: dlr@daver.bungi.com (Dave Rand) To: ppp-comp@bungi.com Subject: Re: List changes Date: Wed, 5 Jan 1994 08:32:42 PST Message-ID: Sender: owner-ppp-comp@daver Reply-To: ppp-comp@bungi.com Precedence: bulk [In the message entitled "Re: List changes" on Jan 3, 10:42, Craig Fox writes:] > Were these discussions archived anywhere safe? I want to know if I can > delete all of my message copies. I have them on bungi.com, and they are available for anonymous ftp from sgi.com:other/ppp-comp -- Dave Rand Internet: dlr@daver.bungi.com