Edit (not just read) pandas Dataframe from ipython REPL

[weichm]

Following this discussion (#2026), I can view a pandas Dataframe in visidata using

import pandas as pd
from visidata import vd
vd.view_pandas(pd.DataFrame(range(10)))

I wonder whether visidata can also be used to edit the dataframe. That is, I want something like df = vd.view_pandas(pd.DataFrame(range(10))) to get a Dataframe out of visidata.

https://www.visidata.org/docs/loading/ suggests that this is not possible for R: "Please note, that this tool opens the data frame in readonly mode."

However, 10 Ways to use Visidata (https://www.visidata.org/blog/2020/ten/) gives the example

mv $(ls | vd) /trash

for the shell. Does anybody know if there is something similar for Pandas dataframes in python REPL?

[paddymul]

One thing to keep in mind for the notebook environment, the cells are very ephemeral and many workflows involve re-executing a cell. So you re-execute a cell where you have visidata state, what happens to that visidata state?

I'm not familiar with this integration. If you are just shelling out to visidata and still have a terminal running visidata terminal running somewhere, then you re-run the same cell and you get a new terminal window, you should be fine. If the terminal runs in place, you are left with a tricky situation.

One thing you could do is keep track of which dataframe was sent in, then you could display a dialogue saying "you already made edits to this dataframe, do you want to start from there?" Tracking to see if the same dataframe is sent as an argument to a function is a bit tricky.

Curious to see how visidata approaches this.

[weichm]

I do not know if I understand your comment, but maybe I am to naive: I just want visidata act as some python function in the sense that it simply returns a DataFrame object, much like df = pd.DataFrame(range(10)). I think in this case there should be no side effects because mv $(ls | vd) /trash also does not seem to have any.

I like the astonishing simplicity of mv $(ls | vd) /trash, which makes me think I will tackle the problem this way, i.e. use python subprocess and call vd from the shell. I have to write pd.DataFrame(range(10)) to STDIN and call subprocess then parse STDOUT into df. Perhaps make this into a custom IPython magic command. In case I succeed, I will post my solution.

What a great piece of software.

[midichef]

Because the visidata PandasSheet object uses a dataframe internally, as self.df, I think there is a simple way to do what you want. In your REPL setup, define edit_pandas, a slightly modified version of view_pandas().

import visidata
def edit_pandas(df):
    vd = visidata.vd
    sheet = visidata.PandasSheet('dataframe', source=df)
    vd.run(sheet)
    return sheet.df   #return the sheet's internal dataframe object

Then in your REPL, when you want to view the dataframe and keep the edited result, do:

import pandas as pd
df = edit_pandas(pd.DataFrame(range(10)))

However, I don't use Pandas or IPython, so I can't be sure this works in depth. Check the results carefully to make sure it's keeping all the changes to your edited dataframe!

[saulpw]

You're becoming quite the VisiData master, @midichef! Thanks for helping people out.

[weichm]

Right now, I was struggling with the implementation of my plan when I saw your answer.

Thanks a lot! This indeed is much better than the solution I suggested.

[midichef]

Also loaders/_pandas.py has a couple of comments about implementation details that you may want to be aware of.

visidata/visidata/loaders/_pandas.py

Lines 82 to 90 in 9c6e901

	Warning:
	The index of the pandas.DataFrame input must be unique.
	Otherwise the selection functionality, which relies on
	looking up selected rows via the index, will break.
	This can be done by calling reset_index().
	Note:
	Columns starting with "__vd_" are reserved for internal usage
	by the VisiData loader.

visidata/visidata/loaders/_pandas.py

Lines 181 to 183 in 9c6e901

	# VisiData assumes string column names but pandas does not. Forcing string
	# columns at load-time avoids various errors later.
	df.columns = df.columns.astype(str)

I don't think they matter for your use case, but keep in mind that you can refer to the details there if you get surprising results later.