Add storage API to open a file for appending, without truncating it's content first #5176

mbaklor · 2024-09-30T21:40:46Z

Implemented in both memory and desktop, need to implement for mobile still, setting this as a draft for now, I don't know who can do mobile and how but for now this is what I could pull off.

Description:

Needed an option to add to a file in the storage without truncating it first, specifically to write long lasting log files.

Fixes #(issue)

Checklist:

Tests included.
Lint and formatter run with no errors.
Tests all pass.

Where applicable:

Public APIs match existing style and have Since: line.

implemented in both memory and desktop, need to implement for mobile still

because I forgot how io reader works

andydotxyz

Clear and clean, thanks for looking at this - a good addition.

I think it will need the mobile code added to land though.
Would tips on how to progress be sufficient, or are you handing this on to someone else?

mbaklor · 2024-10-17T22:33:50Z

Thank you that was super encouraging and got me the drive to try the mobile code myself!

Let me know if this mobile code makes sense to you, and if someone with an android or ios setup - well - set up could check that this code works I'd super appreciate it as well!

thanks to https://lists.apple.com/archives/cocoa-dev/2002/Aug/msg00754.html for giving me the necessary shape for the function

andydotxyz · 2024-10-21T16:08:31Z

Looking great. The code compiles and runs on my iOS test devices.
But there is no test for the file repository on mobile.
Is there a snippet of code that could be run, or an example we could drop into fyne_demo that would verify?

Thanks!

mbaklor · 2024-10-23T11:02:30Z

I have discovered in writing this test snippet that I in fact forgot to implement the last part of the puzzle, the storage interface itself. Fixed and pushed, and here's the snippet I used to check it on my PC

func main() {
	app := app.NewWithID("io.fyne.appendtest")
	win := app.NewWindow("Append Test")
	win.Resize(fyne.NewSize(400, 300))

	app.Storage().Create("test.txt")

	btnAppend := widget.NewButton("append to file", func() {
		doc, err := app.Storage().Append("test.txt")
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		_, err = doc.Write([]byte("test line\n"))
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		err = doc.Close()
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		dialog.ShowInformation("Success", "Wrote to file", win)
	})
	btnRead := widget.NewButton("read file", func() {
		doc, err := app.Storage().Open("test.txt")
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		b := make([]byte, 64)
		_, err = doc.Read(b)
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		dialog.ShowInformation("Success", "Successfully read from file\n\ncontents:\n"+string(b), win)
		println(string(b))
	})

	content := container.NewGridWithRows(2, btnAppend, btnRead)
	win.SetContent(container.NewCenter(content))
	win.ShowAndRun()
}

storage.go

coveralls · 2024-10-26T22:33:07Z

coverage: 59.999% (+0.04%) from 59.961%
when pulling 5afc06f on mbaklor:storage-append
into de361f6 on fyne-io:develop.

mbaklor · 2024-10-26T22:47:40Z

the new "it works on my system" with the storage interface reverted to how it was is:

func main() {
	app := app.NewWithID("io.fyne.appendtest")
	win := app.NewWindow("Append Test")
	win.Resize(fyne.NewSize(400, 300))
	root := app.Storage().RootURI()
	uri, err := storage.Child(root, "test.txt")
	if err != nil {
		dialog.ShowError(err, win)
		return
	}

	btnAppend := widget.NewButton("append to file", func() {
		doc, err := storage.Appender(uri)
		defer doc.Close()
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		_, err = doc.Write([]byte("test line\n"))
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		dialog.ShowInformation("Success", "Wrote to file", win)
	})
	btnRead := widget.NewButton("read file", func() {
		doc, err := storage.Reader(uri)
		defer doc.Close()
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		b := make([]byte, 64)
		_, err = doc.Read(b)
		if err != nil {
			dialog.ShowError(err, win)
			return
		}
		dialog.ShowInformation("Success", "Successfully read from file\n\ncontents:\n"+string(b), win)
		println(string(b))
	})

	content := container.NewGridWithRows(2, btnAppend, btnRead)
	win.SetContent(container.NewCenter(content))
	win.ShowAndRun()
}

andydotxyz · 2024-10-28T10:48:48Z

internal/driver/mobile/repository.go

+	return fileWriterForURI(u, true)
+}
+
+// TODO: need someone who understands native code to write this, I'm not sure


Seems like this TODO could go?

OOPS! deleted

andydotxyz

Thanks for working on this. I think there is still a breaking change in the Repository interface API, but we have discussed how to resolve it. I left notes here as well which I hope help.

andydotxyz · 2024-10-28T10:52:48Z

storage/repository/repository.go

+	// file if it exists
+	//
+	// Since: 2.6
+	Appender(u fyne.URI) (fyne.URIWriteCloser, error)


This is the remaining breaking change where a new (optional) interface will be needed, perhaps AppendableRepository?

andydotxyz · 2024-10-28T10:53:17Z

storage/uri.go

+		return nil, err
+	}
+
+	wrepo, ok := repo.(repository.WritableRepository)


Corresponding to the note above this should be a check for supporting the Appender

…itory

andydotxyz · 2024-11-07T09:32:12Z

This doesn't work work on Android - I finally tracked it down to the modes always being NULL which caused all write operations to fail.

This patch fixes Android!

diff --git a/internal/driver/mobile/android.c b/internal/driver/mobile/android.c
index 14e72b66b..db4f55c06 100644
--- a/internal/driver/mobile/android.c
+++ b/internal/driver/mobile/android.c
@@ -154,9 +154,9 @@ void* saveStream(uintptr_t jni_env, uintptr_t ctx, char* uriCstr, bool truncate)
        jobject uri = parseURI(jni_env, ctx, uriCstr);
        jstring modes = NULL;
        if (truncate) {
-               jstring modes = (*env)->NewStringUTF(env, "wt"); // truncate before write
+               modes = (*env)->NewStringUTF(env, "wt"); // truncate before write
        } else {
-               jstring modes = (*env)->NewStringUTF(env, "wa");
+               modes = (*env)->NewStringUTF(env, "wa");
        }
        jobject stream = (jobject)(*env)->CallObjectMethod(env, resolver, saveOutputStream, uri, modes);
        jthrowable loadErr = (*env)->ExceptionOccurred(env);

andydotxyz · 2024-11-07T19:35:25Z

Unfortunately on iOS it reports writing successfully but the read back just gets EOF not the file content

andydotxyz · 2024-11-07T20:08:07Z

The following change made it work on iOS:

diff --git a/internal/driver/mobile/file_ios.m b/internal/driver/mobile/file_ios.m
index 18b08de24..9ff83770d 100644
--- a/internal/driver/mobile/file_ios.m
+++ b/internal/driver/mobile/file_ios.m
@@ -39,7 +39,7 @@ bool iosExistsPath(const char* path) {
     NSFileHandle* handle = [NSFileHandle fileHandleForWritingToURL:url error:&err];
 
     if (!truncate) {
-        [handle truncateFileAtOffset:[handle seekToEndOfFile]];
+        [handle seekToEndOfFile];
     }
 
     return handle;

but in part it was because the read operation can return EOF error which we were treating as a problem, so try this:

              _, err = doc.Read(b)
              if err != nil && err != io.EOF {
                      dialog.ShowError(err, win)
                      return
              }

andydotxyz

Looking great. With those small changes for Android and iOS I think this is a winner

andydotxyz

Thanks so much for this

mbaklor added 3 commits September 30, 2024 23:43

implemented appender to open a file without truncating

9856a9d

implemented in both memory and desktop, need to implement for mobile still

fixed failing tests in internal/docs

9e868e3

because I forgot how io reader works

set openFile to os.O_WRONLY assuming that's why it fails?

276adfe

andydotxyz reviewed Oct 12, 2024

View reviewed changes

first draft for append on mobile drivers

2ba9612

mbaklor marked this pull request as ready for review October 17, 2024 22:34

made real ios append without loading entire file to memory

eeb8511

thanks to https://lists.apple.com/archives/cocoa-dev/2002/Aug/msg00754.html for giving me the necessary shape for the function

andydotxyz requested changes Oct 24, 2024

View reviewed changes

storage.go Outdated Show resolved Hide resolved

mbaklor force-pushed the storage-append branch from 1ad36f8 to eeb8511 Compare October 26, 2024 22:27

andydotxyz reviewed Oct 28, 2024

View reviewed changes

andydotxyz requested changes Oct 28, 2024

View reviewed changes

mbaklor added 2 commits November 2, 2024 23:50

added optional interface AppendableRepository extending WritableRepos…

ed1fd11

…itory

removed Append method from internal.Docs

504b502

andydotxyz requested changes Nov 8, 2024

View reviewed changes

mbaklor added 2 commits November 23, 2024 22:58

file_ios seek to end without truncating at offset

2d1ffe5

android file modes were always NULL because oops

5afc06f

andydotxyz approved these changes Nov 23, 2024

View reviewed changes

andydotxyz merged commit a6f745f into fyne-io:develop Nov 23, 2024
11 checks passed

Uh oh!

Add storage API to open a file for appending, without truncating it's content first #5176

Add storage API to open a file for appending, without truncating it's content first #5176

Uh oh!

Conversation

mbaklor commented Sep 30, 2024

Description:

Checklist:

Where applicable:

Uh oh!

andydotxyz left a comment

Choose a reason for hiding this comment

Uh oh!

mbaklor commented Oct 17, 2024

Uh oh!

andydotxyz commented Oct 21, 2024

Uh oh!

mbaklor commented Oct 23, 2024

Uh oh!

Uh oh!

coveralls commented Oct 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbaklor commented Oct 26, 2024

Uh oh!

andydotxyz Oct 28, 2024

Choose a reason for hiding this comment

Uh oh!

mbaklor Nov 2, 2024

Choose a reason for hiding this comment

Uh oh!

andydotxyz left a comment

Choose a reason for hiding this comment

Uh oh!

andydotxyz Oct 28, 2024

Choose a reason for hiding this comment

Uh oh!

andydotxyz Oct 28, 2024

Choose a reason for hiding this comment

Uh oh!

andydotxyz commented Nov 7, 2024

Uh oh!

andydotxyz commented Nov 7, 2024

Uh oh!

andydotxyz commented Nov 7, 2024

Uh oh!

andydotxyz left a comment

Choose a reason for hiding this comment

Uh oh!

andydotxyz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Oct 26, 2024 •

edited

Loading