TL;DR

1. Connect to the database using one of the VS Code extensions

   • SQL Server (mssql) extension
   • PostgreSQL extension by Microsoft
   • PostgreSQL extension by Database Client (supports other databases like Oracle, DuckDB etc.)

2. Install JupySQL

   pip install jupysql

3. Create SQLAlchemy engine and assign it to engine variable

4. Load JupySQL magics and connect using engine

   %load_ext sql
   %sql engine --alias my_db

5. In the cell you want to write SQL, change cell language to SQL (Ctrl + K, M by default)

6. Make sure the cell is connected to the database (active connection is visible at the top of the SQL cell)

7. Put %%sql magic at the top of the SQL cell

8. Write your SQL and enjoy SQL IntelliSense (context aware autocomplete), SQL formatting and work with Pandas DataFrames

9. ???

10. Profit!

Requirements

Connect to the database

import getpass
import sqlalchemy as sa
import pyodbc

driver = [driver for driver in pyodbc.drivers() if "ODBC" in driver][0] # take available ODBC driver

connection_url = sa.engine.URL.create(
    drivername="mssql+pyodbc", # dialect+driver
    username="sa",
    password=getpass.getpass(), # you can write password here in plain text or pass environment variable
    host="localhost",
    port="1433",
    database="AdventureWorks",
    query={ # different key: value pairs for each type of database
        "driver": driver,
        "Encrypted": "yes",
        "TrustServerCertificate": "yes",
    },
)

engine = sa.create_engine(connection_url)

import getpass
import sqlalchemy as sa

connection_url = sa.engine.URL.create(
    drivername="postgresql+psycopg", # dialect+driver
    username="postgres",
    password=getpass.getpass(), # you can write password here in plain text or pass environment variable
    host="localhost",
    port="5432",
    database="postgres",
)

engine = sa.create_engine(connection_url)

import getpass
import sqlalchemy as sa

connection_url = sa.engine.URL.create(
    drivername="oracle+oracledb", # dialect+driver
    username="my_user",
    password=getpass.getpass(), # you can write password here in plain text or pass environment variable
    host="localhost",
    port="1521",
    query={ # different key: value pairs for each type of database
        "service_name": "FREEPDB1"
    }
)

engine = sa.create_engine(connection_url)

%load_ext sql
%sql engine --alias my_db

Write SQL in the cell

Put %%sql magic at the top of the cell. You can now write SQL queries in the SQL cell and enjoy IntelliSense (context aware autocomplete) for SQL syntax and database objects as well as SQL formatting³.

I must admit the setup is kind of fragile. In case IntelliSense for database objects doesn’t work try one or all of these:

• using command palette (Ctrl+Shift+P) select MS SQL: Refresh IntelliSense Cache (or equivalent in other extension)
• delete cell and create new one (change it’s language to SQL and connect it to database)
• restart VS Code
• (this one is weird) type your first SQL queries slowly, once it gets going you can type faster (maybe it has something to do with the fact that SQL IntelliSense in VS Code is kinda slow?)

Work with pandas

In order to pass the result of the query to Pandas DataFrame there is a two step process

1. Assign result to a variable e.g. sql_result modyfying the %%sql magic

   %%sql sql_result <<
   SELECT
   *
   FROM Production.Product

2. In next Python cell extract it to DataFrame

   df_result = sql_result.DataFrame()

Alternatively, you can setup Pandas output globally (this allows to skip second step):

%config SqlMagic.autopanda = True

To save an existing DataFrame df to database you can use flag --persist or --persist-replace (table will have the same name as the DataFrame variable, schema is simply schema in the database)

%sql --persist schema.df
%sql --persist-replace schema.df 

Close connection

To close connection use --close flag with connection alias

%sql --close my_db

Other features

There are a lot of other features in JupySQL like

• parametric queries using Jinja
• working with multiple connections
• query .csv and Excel files
• plotting results of the queries

and much more. Everything is cleanly explained in the JupySQL documentation

Dyck path

Maybe it is not common convention but let’s call Dyck path from to Dyck path of order

import matplotlib.pyplot as plt
from matplotlib.axes import Axes
from itertools import accumulate
import numpy as np
from scipy.special import comb

plt.rcParams["text.usetex"] = True


def seq_to_path(
    seq: list[int], starting_point: tuple[int, int] = (0, 0)
) -> tuple[list, list]:
    """Map list of steps to a path on a square lattice
    - 1 == UP
    - 0 == RIGHT

    Parameters
    ----------
    seq : list[int]
        Sequence of steps
    starting_point : tuple[int, int], optional
        Point at which to start a path, by default (0, 0)

    Returns
    -------
    tuple[list, list]
        List of x and list of y coordinates of points on a path
    """
    steps = np.array(seq)
    # Concatenate [0] at the start to represent the initial starting point shift
    x = starting_point[0] + np.concatenate(([0], np.cumsum(steps == 0)))
    y = starting_point[1] + np.concatenate(([0], np.cumsum(steps == 1)))
    return x, y


ax_good: Axes
ax_bad: Axes
fig, (ax_good, ax_bad) = plt.subplots(1, 2)

ax_good.set_title("Dyck path")
dyck_path = seq_to_path([1, 1, 0, 0, 1, 0, 1, 0])

ax_good.set_aspect(1)
ax_good.plot(dyck_path[0], dyck_path[1], marker=".", lw=2, ms=10)
ax_good.plot([0, 4], [0, 4], zorder=0)
ax_good.grid()
ax_good.set_yticks([0, 1, 2, 3, 4])
ax_good.set_xticks([0, 1, 2, 3, 4])

ax_bad.set_title("Non-Dyck path")
non_dyck_path = seq_to_path([1, 0, 0, 1, 1, 0, 0, 1])
ax_bad.set_aspect(1)
ax_bad.plot(non_dyck_path[0], non_dyck_path[1], marker=".", lw=2, ms=10)
ax_bad.plot([0, 4], [0, 4], zorder=0)
ax_bad.grid()
ax_bad.set_yticks([0, 1, 2, 3, 4])
ax_bad.set_xticks([0, 1, 2, 3, 4])

plt.show()

Sometimes you can find different but equivalent versions of the definition e.g.:

• path is always below
• path is from to , using only diagonal moves and never dips below line (kind of like ASCII mountain range built using symbols \ and /)
• path from to using UP and RIGHT steps, where number of RIGHT steps at any point is less than or equal to number of UP steps

General UP/RIGHT path from to consists of UP steps and RIGHT steps - in total steps. To count how many unique paths (not only Dyck paths) there are in total we need to determine in how many ways we can place UP steps in places (the rest will be taken by RIGHT steps). This is exactly unordered sampling without replacement, so we have:

Counting Dyck paths - graphical method

The problem is the following:

Imagine a non-Dyck path - at some point it crosses the line. Let’s call first step that makes the path a non-Dyck path the bad step - it causes the line to cross the line. Now reflect the part of the path from start up to the bad step with respect to line .

fig, ax = plt.subplots(1, 1)
ax.axes.set_aspect(1)
reflected_part = seq_to_path([0, 1, 1], starting_point=(1, -1))
ax.plot(non_dyck_path[0], non_dyck_path[1], marker=".", lw=2, ms=10)
ax.plot([0, 4], [0, 4], zorder=0)
ax.plot([0, 4], [-1, 3], zorder=0)
ax.plot(reflected_part[0], reflected_part[1], marker=".", lw=2, ms=10, label="reflected part")

ax.text(x=1.1, y=1.1, s="bad step", size=12)
ax.text(x=2.5, y=1.7, s=r"y=x-1", rotation=45)
ax.legend()
ax.grid()
ax.set_yticks([-1, 0, 1, 2, 3, 4])
ax.set_xticks([0, 1, 2, 3, 4])

When we connect reflected part and the rest of the path we get a path from to . In fact, each non-Dyck path will be mapped to different path from to and vice versa. To count paths from to (so non-Dyck paths equivalently) we can use same argument as counting all paths. This means that we have steps in total of which are RIGHT steps:

Together with Eq. 1 this gives us total number of Dyck paths of order as:

As promised, we got result that is famous in counting problems and is called Catalan number (in the Wikipedia article you will also find this proof in section Second proof)

Counting Dyck paths - cyclic permutations

Equipped with Raney’s lemma we can move to second method of counting Dyck paths. But first we need to change representation of paths from sequence of UP and RIGHT steps to sequence of height of steps at each position:

dyck_path = [[0, 0, 1, 2, 2, 3, 3, 4], [0, 2, 2, 2, 3, 3, 4, 4]]
non_dyck_path = [[0, 0, 1, 2, 2, 3, 4, 4], [0, 1, 1, 1, 3, 3, 3, 4]]

fig, (ax_good, ax_bad) = plt.subplots(1, 2)

ax_good.plot(dyck_path[0], dyck_path[1], marker=".", lw=2, ms=10)
ax_good.plot([0, 4], [0, 4], zorder=0)

ax_good_top = ax_good.twiny()
ax_good_top.set_xlim(ax_good.get_xlim())
ax_good_top.set_ylim(ax_good.get_ylim())
ax_good.set_box_aspect(1)
ax_good.set_aspect(1)
ax_good_top.set_aspect(1)
ax_good.grid()
ax_good_top.set_xticks([i for i in range(5)])
ax_good_top.set_xticklabels(["[2", 0, 1, 1, "0]"])
ax_good_top.set_title("Dyck path, step heights")

ax_bad.plot(non_dyck_path[0], non_dyck_path[1], marker=".", lw=2, ms=10)
ax_bad.plot([0, 4], [0, 4], zorder=0)

ax_bad_top = ax_bad.twiny()
ax_bad_top.set_xlim(ax_bad.get_xlim())
ax_bad_top.set_ylim(ax_bad.get_ylim())
ax_bad.set_box_aspect(1)
ax_bad.set_aspect(1)
ax_bad_top.set_aspect(1)
ax_bad.grid()
ax_bad_top.set_xticks([i for i in range(5)])
ax_bad_top.set_xticklabels(["[1", 0, 2, 0, "1]"])
ax_bad_top.set_title("Non-Dyck path, step heights")

plt.show()

In this representation general UP/RIGHT paths from to can be defined as:

Sequence of non-negative integers that sums to :

and Dyck paths additionally satisfy constraint:

We would like to group all UP/RIGHT paths from to into groups containing only cyclic permutations e.g.:

[2, 0, 0, 1]
[1, 2, 0, 0]
[0, 1, 2, 0]
[0, 0, 1, 2]

Since we have constraint Eq. 5 and all there is no possibility to make subcycles. In order to show this, we can map this problem to placing balls into boxes. To make subcycles we would have to split the sequence into equal parts, each containing balls. Finding size of subcycles is equivalent to finding greatest common denominator of and but this is:

Therefore, there are no subcycles. Subcycles would lead to repeated elements within group of cyclic permutations. This means that:

Each group of cyclic permutations has unique elements.

Using this fact we can count total number of cyclic groups:

This is suspicious - the number of cyclic groups of UP/RIGHT paths is equal to number of Dyck paths Eq. 2. We can follow this trail and potentially find more direct connection.

Let’s transform our sequence of step heights in the following way:

The minus in front of index means that sequence is running in the reverse order compared to sequence . However, sum from index to include all elements from both sequences Constraint Eq. 5 for sequence takes the following form:

For sequences of integer that sum up to we know from Raney’s lemma that there is only one cyclic permutation that satisfies:

Let’s translate this statement back to original sequence of step heights reversing the formula Eq. 7 ():

To transform this further we need to split the sum of all elements of in Eq. 5 into two parts (again - if we sum all elements order in which summing index runs doesn’t matter):

Inserting this form into inequality Eq. 8 gives us:

We can now rename indices in the following way:

It is important to note here the range of possible values o - since and , this means that:

Putting everything into inequality Eq. 9 we get:

For we sum empty sequence, so this is trivially satisfied (we have LHS: and RHS:, so ). Since we are dealing with integers we can change to by getting rid of . This gives us:

From the above calculation we conclude that:

There can be only one cyclic permutation within each cyclic group that is Dyck path.

Therefore, since we’ve established the number of cyclic groups Eq. 6 we’ve also counted the number of Dyck paths

which is the same as in graphical method Eq. 2 - Success!

Although, this method was more elaborate than graphical method it highlights very important fact:

Dyck paths correspond to unique generators of all cyclic groups for sequences defined by:

Generating Dyck paths

Sometimes counting Dyck paths isn’t enough. We might want to generate all Dyck paths of given order. To achieve this we can use following recursive algorithm (inspired by algorithm on scipython blog):

1. Start from empty path
2. Run all following instructions for the path so far:
   1. If the path reached return it
   2. If the number of UP steps is less than send to instruction 2. path so far with added UP step
   3. If the number of RIGHT steps is less than number of UP steps send to instruction 2. path so far with added RIGHT step

In Python, this can be implemented using generators (they allow to iterate over some sequence similarilry to range or various generators from itertools module):

def dyck_paths(n: int):
    """Generator producing Dyck paths of order `n`.

    - 1 == UP step
    - 0 == RIGHT step

    Parameters
    ----------
    n : int
        Order of Dyck paths
    """

    def make_path(path: tuple, n_up: int, n_right: int):
        # if path reached the end, yield it
        if len(path) == 2 * n:
            yield path

        # add UP step if you can
        if n_up < n:
            yield from make_path(path + (1,), n_up=n_up + 1, n_right=n_right)

        # add RIGHT step if you can
        if n_right < n_up:
            yield from make_path(path + (0,), n_up=n_up, n_right=n_right + 1)

    yield from make_path((1,), n_up=1, n_right=0)  # start from single UP step

Slightly modified version returns Dyck paths in step height representation:

def dyck_path_heights(n: int):
    """Generator producing Dyck paths of order `n` in heights representation

    Path has `n+1` heights.

    Parameters
    ----------
    n : int
        Order of Dyck paths
    """

    def make_heights(path: tuple, n_up: int, n_right: int):
        # if path reached the end, yield it
        if n_up + n_right == 2 * n:
            yield path

        # add UP step if you can
        if n_up < n:
            yield from make_heights(
                path[:-1] + (path[-1] + 1,), n_up=n_up + 1, n_right=n_right
            )

        # add RIGHT step if you can
        if n_right < n_up:
            yield from make_heights(path + (0,), n_up=n_up, n_right=n_right + 1)

    yield from make_heights((1,), n_up=1, n_right=0)

Let’s test it in action for order

n = 4
catalan_number = comb(2 * n, n, exact=True, repetition=False) // (n + 1)

fig, axes = plt.subplots(nrows=3, ncols=5)
ax: Axes

for ax, seq, height_seq in zip(axes.reshape(-1), dyck_paths(n), dyck_path_heights(n)):
    path = seq_to_path(seq)
    ax.plot(path[0], path[1])
    ax.plot([0,n], [0,n], zorder=0)
    ax.set_aspect(1)
    ax.set_xticks(list(range(n+1)), height_seq)
    ax.set_yticks([])
    ax.xaxis.set_ticks_position("top")

fig.delaxes(axes.reshape(-1)[-1])
fig.suptitle(f"{catalan_number} Dyck paths of order {n}")
plt.show()