Performance of gzip compared to node-zopfli

[csvn]

Hello!

Thanks for creating this project. I did read the line (below) from the README.md, and feel free to close this issue if this is not in the scope.

This library is a JavaScript binding to zopfli with WebAssembly. This is slower than native extensions for zopfli, but because wasm is a portable binary format, its installation is much easier than native extensions.

I was using node-zopfli to gzip the files in our dist folder. Due to the hassle of using node-gyp, I decided to switch to this package. I noticed though that for our artifacts, it took roughly 4 times longer to gzip all the files (~6s to ~22s).

Input files	gtag.js (65KB)	Medium1 (578 KB)	Large1 (3.1MB)	8 files2 (3.5MB)	8 build files3 (3.5MB)
node-zopfli (s)	0.55	2.1	8.8	4.5	6
@gfx-zopfli (s)	0.48	2.4	10.0	15.5	22

¹ The gtag script was copied until it reached the target size
² The gtag script was copied into 8 files with sizes varying from 100KB to 1MB
³ Our project has 4 javascript files with their respective .map files, with sizes between 100KB and 1MB

Scripts

Below is the script used to transform files (executed with node --experimental-modules <file>). I also tried using @gfx/zopfli with a stream transform (inpsired by node-zopfli), but it did not seem to affect the speed (I had a theory that streams due to backpressure could be more effective).

node-zopfli

import fs from 'fs';
import glob from 'glob';
import zopfli from 'node-zopfli';
import { join } from 'path';
import { promisify } from 'util';
import { pipeline } from 'stream';


main()
  .then(() => console.log('Files have been successfully gzipped.'))
  .catch(err => console.error(err));

async function main() {
  const distFolder = join(process.cwd(), 'dist');
  const gzipFolder = join(distFolder, 'gzip');

  try {
    await fs.promises.mkdir(gzipFolder, { recursive: true });
  } catch (err) {
    if (err.code !== 'EEXIST') throw err;
  }

  const pipe = promisify(pipeline);
  const files = await promisify(glob)('*!(.d.ts)', { nodir: true, cwd: distFolder });

  await Promise.all(files.map(p => pipe(
    fs.createReadStream(join(distFolder, p)),
    zopfli.createGzip(),
    fs.createWriteStream(join(gzipFolder, p))
  )));
}

@gfx/zopfli

import fs from 'fs';
import glob from 'glob';
import zopfli from '@gfx/zopfli';
import { join } from 'path';
import { promisify } from 'util';


main()
  .then(() => console.log('Files have been successfully gzipped.'))
  .catch(err => console.log(err));

async function main() {
  const distFolder = join(process.cwd(), 'dist');
  const gzipFolder = join(distFolder, 'gzip');

  try {
    await fs.promises.mkdir(gzipFolder, { recursive: true });
  } catch (err) {
    if (err.code !== 'EEXIST') throw err;
  }

  const files = await promisify(glob)('*!(.d.ts)', { nodir: true, cwd: distFolder });

  await Promise.all(files.map(async p => {
    const buffer = await fs.promises.readFile(join(distFolder, p));
    const gzipped = await zopfli.gzipAsync(buffer, {});
    await fs.promises.writeFile(join(gzipFolder, p), gzipped);
  }));
}

@gfx/zopfli (transform stream)

import fs from 'fs';
import glob from 'glob';
import zopfli from '@gfx/zopfli';
import { join } from 'path';
import { promisify } from 'util';
import { Transform, pipeline } from 'stream';


main()
  .then(() => console.log('Files have been successfully gzipped.'))
  .catch(err => console.error(err));

async function main() {
  const distFolder = join(process.cwd(), 'dist');
  const gzipFolder = join(distFolder, 'gzip');

  try {
    await fs.promises.mkdir(gzipFolder, { recursive: true });
  } catch (err) {
    if (err.code !== 'EEXIST') throw err;
  }

  const pipe = promisify(pipeline);
  const files = await promisify(glob)('*!(.d.ts)', { nodir: true, cwd: distFolder });

  await Promise.all(files.map(p => pipe(
    fs.createReadStream(join(distFolder, p)),
    new Gzip(),
    fs.createWriteStream(join(gzipFolder, p))
  )));
}

class Gzip extends Transform {
  constructor() {
    super();
    this.in = new Buffer.alloc(0);
  }

  _transform(chunk, encoding, done) {
    this.in = Buffer.concat([this.in, chunk]);
    done();
  }

  _flush(done) {
    zopfli.gzip(this.in, {}, (err, out) => {
      if (err) {
        done(err);
      } else {
        this.push(out);
        done();
      }
    });
  }
}

Is .wasm the reason for the slowdown?
Is it within reason that @gfx/zopfli is sometimes ~4x slower than using node-zopfli?
Or maybe I've done something weird in the script?

@gfx/zopfli version: 1.0.13

Edit: Added version info

[gfx]

Thanks to the report. This library may be much slower than node-zopfli, but I have not investigated it deeply. It could be wasm / emscripten issues; maybe it's time to investigate the performance.

BTW I use this library in production, and because node-zopfli is sometimes failed to build and I don't want to relay on node-gyp, I have no choice except for @gfx/zopfli.

[mvasilkov]

Hi @gfx,

Great library! I'm so looking forward to using this in my projects.

With relation to Emscripten's runtime performance: from my limited experience, the usual suspect for a bottleneck in V8 is cross-boundary calls (JS to WASM, or WASM to JS), including memory management, copying return values, things like that.

[csvn]

@gfx I agree, I just upgraded to Node v12, and node-pre-gyp was missing pre-builts for that version, and I spent way too long to try to make node-gyp build without errors (without succeeding). I was just hoping for having the best of both worlds (no node-gyp and around the same performance) 😉