Pako encode live diagram urls via Python

[dvulanov]

Hello,

I really appreciate the functionality of the mermaid live editor. For me, I'm generating my urls via python. I can encode64 graphs, but would also like to be able to encode larger graphs using pako.

Does anyone know the python implementation of pako (zlib) that's compatible with mermaid live?

Many thanks for any help,
Dan

[dvulanov]

I managed to figure it out with help from stackoverflow and debugging in the browser. Hopefully this helps someone else:

import base64
import json
import zlib

def js_string_to_byte(data):
    return bytes(data, 'ascii')

def js_bytes_to_string(data):
    return data.decode('ascii')

def js_btoa(data):
    return base64.b64encode(data)

def pako_deflate(data):
    compress = zlib.compressobj(9, zlib.DEFLATED, 15, 8,zlib.Z_DEFAULT_STRATEGY)
    compressed_data = compress.compress(data)
    compressed_data += compress.flush()
    return compressed_data

def genPakoLink(graphMarkdown: str):
    jGraph = {
            "code": graphMarkdown,
            "mermaid": {"theme": "default"}
        }
    byteStr = js_string_to_byte(json.dumps(jGraph))
    deflated = pako_deflate(byteStr)
    dEncode = js_btoa(deflated)
    link = 'http://mermaid.live/view#pako:' + js_bytes_to_string(dEncode).replace('+', '-').replace('/', '_')
    return link

[tamsanh]

Very valuable, thank you!

[78wesley]

This worked! Thank you!

[78wesley]

I've shorten the code and also added a decoder.

import base64
import json
import zlib


def encodePako(code: str, config: dict = {}) -> str:
    graph = {"code": code, "mermaid": config}
    compress = zlib.compressobj(9, zlib.DEFLATED, 15, 8, zlib.Z_DEFAULT_STRATEGY)
    compressed_data = compress.compress(bytes(json.dumps(graph), "ascii")) + compress.flush()
    return "#pako:" + base64.b64encode(compressed_data).decode("ascii").replace("+", "-").replace("/", "_")


def decodePako(link: str):
    if "#pako:" not in link:
        raise ValueError("Invalid pako link")

    b64_data = link.split("#pako:")[1].replace("-", "+").replace("_", "/")
    return json.loads(zlib.decompress(base64.b64decode(b64_data)).decode("utf-8"))


if __name__ == "__main__":
    mermaid_chart = """graph TD
    A[Start] --> B{Is it working?}
    B -- Yes --> C[Great!]
    B -- No --> D[Fix it]
    D --> B
    """
    mermaid_chart_link = encodePako(mermaid_chart)
    link = "https://www.mermaidchart.com/play" + mermaid_chart_link
    link2 = "https://www.mermaid.live/edit" + mermaid_chart_link
    print("Link:", "\n", link)
    print("Link:", "\n", link2)
    decoded_data = decodePako(link)
    print("Decoded data:", decoded_data)
    print("Matches original:", decoded_data["code"] == mermaid_chart)

[zeekoe]

While I'm doing Java now I tried to copy this implementation and it doesn't seem to work. Does mermaid still use Base64? It looks like some other character encoding.

[dvulanov]

Mermaid does use base64. If you are only encoding base64 and not then zipping, you need to switch the url encoding variable from #pako to #base64

[zeekoe]

#base64 should be good enough for me, thanks for the suggestion!

This doesn't work though:

https://mermaid.live/view#base64:Z3JhcGggTFIKRU5EX0VYRUNVVEVEKCgiZW5kIikpClBST0NFU1NfU1RBUlQoKCJzdGFydCIpKQpQUk9DRVNTX1NUQVJUIC0tPiBFTkRfRVhFQ1VURUQ=

An online base64 decoder does seem to make correct text out of it. Could you spot the error?

[zeekoe]

Got it! There were two problems:

• The main one is that the encoded data should be a json string containing more than just the graph definition
• The second one (where my confusion came from) is that a different Base64 character set is being used than the most used one; see https://stackoverflow.com/questions/13195143/range-of-valid-character-for-a-base-64-encoding

A snippet of Java code that works for me:

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Base64;
import java.util.zip.Deflater;


class MermaidCodeThingy {
  public static void main(String[] args) throws IOException {
    String simpl = "{\"code\":\"graph LR\\nEND_EXECUTED((\\\"end\\\"))\\nPROCESS_START((\\\"start\\\"))\\nPROCESS_START --> END_EXECUTED\",\"mermaid\":\"{\\n  \\\"theme\\\": \\\"dark\\\"\\n}\",\"autoSync\":true,\"updateDiagram\":false,\"editorMode\":\"code\"}";

    String base64String = Base64.getEncoder().encodeToString(simpl.getBytes(StandardCharsets.UTF_8));
    String url = "https://mermaid.live/view#base64:" + base64String.replace("/", "_").replace("+", "-");
    System.out.println(url);

    // And now compressed
    byte[] output = new byte[10000];
    Deflater compresser = new Deflater();
    compresser.setInput(simpl.getBytes(StandardCharsets.UTF_8));
    compresser.finish();
    int compressedDataLength = compresser.deflate(output);
    compresser.end();

    String compressedBase64String = Base64.getEncoder().encodeToString(Arrays.copyOfRange(output, 0, compressedDataLength));
    String url2 = "https://mermaid.live/view#pako:" + compressedBase64String.replace("/", "_").replace("+", "-");
    System.out.println(url2);
  }
}

[gclotault]

Hello,

Here is the Python script by @dvulanov converted to JavaScript :

const pako = require('pako');

function jsStringToByte(data) {
  return new TextEncoder().encode(data);
}

function jsBtoa(data) {
  let binary = '';
  const bytes = new Uint8Array(data);
  for (let i = 0; i < bytes.byteLength; i++) {
    binary += String.fromCharCode(bytes[i]);
  }
  return btoa(binary);
}

function genPakoLink(graphMarkdown, editMode) {
  const jGraph = {
    code: graphMarkdown,
    mermaid: { theme: 'default' },
  };

  const byteStr = jsStringToByte(JSON.stringify(jGraph));

  const deflated = pako.deflate(byteStr);

  const dEncode = jsBtoa(deflated);

  const link =
    `http://mermaid.live/${editMode ? 'edit' : 'view'}#pako:` + dEncode.replace('+', '-').replace('/', '_');

  return link;
}

StackBlitz example : https://stackblitz.com/edit/stackblitz-starters-5bzieps1

[dvulanov]

I'm still a big fan of mermaid, and it looks like this thread is still helping people. Note that if you're like me and posting your mermaid links to Azure Devops wiki documents (markdown), these links will break (at least my python implementation does) because they produce pluses and slashes in the URL (which devops sanitizes). You can get around this by replacing + for - and / for _

i.e. link = 'http://mermaid.live/view#pako:' + js_bytes_to_string(dEncode).replace('+', '-').replace('/', '_')

(I've updated my original code, and @zeekoe already addressed this issue)

[Chi-Iroh]

In case anybody needs it, here's an implementation in free form ILE RPG for IBMi :

**free

ctl-opt dftactgrp(*no) bnddir('ZLIB/ZLIB');

/include ZLIB/H,ZLIBFREE

dcl-s code varchar(8192) ccsid(*utf8) inz;
dcl-s json char(8192) ccsid(*utf8) inz;
dcl-s jsonHex char(8192) ccsid(*hex) inz;
dcl-s jsonHex2 char(8192) ccsid(*hex) inz;
dcl-s jsonInflatedLen uns(10) inz;
dcl-s jsonDeflatedLen uns(10) inz;

dcl-s zlibStatus int(10) inz(Z_OK);
dcl-ds zstream likeds(z_stream);

code = 'graph TD' + '\n';
code += '\t' + 'A[hello]-->B[world]' + '\n';

code = '{"code": "' + code + '", "mermaid": {"theme": "default"}}';
json = code;
jsonInflatedLen = %len(code);
longsndmsg('JSON chars (' + %char(jsonInflatedLen) + ' bytes): ' + json);
longsndmsg('JSON bytes: ' + bytes_of(%addr(json) : jsonInflatedLen));
jsonDeflatedLen = compressBound(jsonInflatedLen); // max deflated len

jsonHex = json;
bzero(%addr(zstream) : %size(zstream));
zstream.zs_next_in = %addr(jsonHex);
zstream.zs_avail_in = jsonInflatedLen;
zstream.zs_next_out = %addr(jsonHex2);
zstream.zs_avail_out = jsonDeflatedLen;


zlibStatus = deflateInit2(zstream : Z_BEST_COMPRESSION : Z_DEFLATED : 15 : 8 : Z_DEFAULT_STRATEGY : ZLIB_VERSION : %size(z_stream));
if zlibStatus <> Z_OK;
    snd-msg 'deflateInit2 error';
    snd-msg %str(zError(zlibStatus));
    return;
endif;

zlibStatus = deflate(zstream : Z_FINISH);
if zlibStatus <> Z_STREAM_END;
    snd-msg 'deflate error !';
    snd-msg %str(zError(zlibStatus));
    snd-msg 'deflate partial bytes: ' + bytes_of(%addr(jsonHex2) : jsonDeflatedLen);
    return;
endif;
if deflateEnd(zstream) <> Z_OK;
    snd-msg 'deflateEnd error !';
    snd-msg %str(zError(zlibStatus));
    return;
endif;

longsndmsg('deflated bytes: ' + bytes_of(%addr(jsonHex2) : zstream.zs_total_out));

json = %subst(jsonHex2 : 1 : zstream.zs_total_out);
jsonDeflatedLen = zstream.zs_total_out;
exec sql values qsys2.base64_encode(substr(:jsonHex2, 1, cast(:jsonDeflatedLen as integer))) into :json;
json = %scanrpl(%char('+' : *utf8) : %char('-' : *utf8) : json : *stdcharsize); // *stdcharsize as base64 uses simple unaccented characters
json = %scanrpl(%char('/' : *utf8) : %char('_' : *utf8) : json : *stdcharsize); // *stdcharsize as base64 uses simple unaccented characters

longsndmsg(%trim(json));
return;

dcl-proc bytes_of;
    dcl-pi *n varchar(65536);
        strptr pointer value;
        n_bytes uns(10) value;
    end-pi;

    dcl-s i uns(10) inz;
    dcl-s byte uns(3) based(strptr);
    dcl-s res varchar(65536) inz;

    for i = 1 to n_bytes;
        if i > 1;
            res += ' ';
        endif;
        res += hexdigit(%div(byte : 16));
        res += hexdigit(%rem(byte : 16));
        strptr += 1;
    endfor;
    return res;
end-proc;

dcl-proc hexdigit;
    dcl-pi *n varchar(1);
        n uns(3) const;
    end-pi;

    return %subst('0123456789ABCDEF' : n + 1 : 1);
end-proc;

dcl-proc bzero;
    dcl-pi *n;
        ptr pointer value;
        size uns(10) value;
    end-pi;

    dcl-s byte uns(3) based(ptr);

    dou size = 0;
        byte = 0;
        ptr += 1;
        size -= 1;
    enddo;
    return;
end-proc;

dcl-proc longsndmsg;
    dcl-pi *n;
        str varchar(65536) const;
    end-pi;

    dcl-s start uns(20) inz(1);
    dcl-s len uns(20) inz;

    len = %len(str);

    dou start > len;
        snd-msg %subst(str : start : %min(512 : len - start) : *natural);
        start += 512;
    enddo;
end-proc;

It's pretty long as I added some debug procedures at the end, and I advise a record length of about 140 for the physical file storing this snippet, as some lines are long too.
It expects that you built zlib inside ZLIB library, which should contain a *BNDDIR named ZLIB too.
To build zlib, you can clone its repo in the IFS, then go to <zlib_repo_root>/os400/ and run make.sh, after having configured some settings at the beginning of the very same file.
Only CCSIDs 1208 (UTF-8) and 65535 (hexadecimal mode, no conversion is performed) have been used, so it should work whatever your system/job CCSID is (mine are both 1147, French with euro symbol).
The strings have arbitrary sizes, so put whatever you want as their size.
Last, the header ZLIBFREE I'm referring too is introduced by a PR of mine (at the time I'm writing, not accepted), but zlib already has a header (in fixed form though), but it needs small adjustments to compile --> madler/zlib#1095