Devlog #13: UI/UX Implementation (WIP)

With the backend functionality (mostly) complete, I’ve been working on implementing a smooth and easy-to-use user interface.

Added system tray icon

I added a system tray icon to allow the user to easily relaunch the UI while the backend continuously runs in the background.

Cloud-folder assignment combobox + two-stage assignment system

Instead of one-shotting everything, there’s now a two-stage system to cloud assignment. When a user adds a cloud to a folder, it first is added to UserAssignedClouds. During the next synchronization, the service will:

1. check UserAssignedClouds
2. upload the folder to the remote locations in those clouds
3. move the clouds from UserAssignedClouds to AssignedClouds (this way, we know where the folder currently exists at)

Add an IgnoreRules editor for WatchedFolders

I had previously coded an ignore rules popup with a visual and raw rules editor, and I implemented it today specifically for WatchedFolders. The user can now set ignore rule globs (according to VS Code glob rules: https://code.visualstudio.com/docs/editor/glob-patterns) to give themselves more control over what the service syncs and what it doesn’t.

Miscellaneous UI improvements + functionalities

• completed folders.tsx page
• added web hook listener in WebView2 to natively open File Explorer for folders
• redid the styling for the sync.tsx page (still a WIP)

Fix Dropbox/OneDrive OAuth triggers and revocations

There were previously issues with the Dropbox (wrong redirectURI) and OneDrive (wrong OAuth consumer link) flows that I fixed after some testing. Everything regarding different accounts should work 100%!

Devlog #12: Architectural Cleanup

Dropbox and OneDrive providers are complete

I reused the same overall logic for the Google Drive provider to write code for the Dropbox and OneDrive providers. I’m using the .NET REST APIs for both to handle file chunking and uploads automatically.

Implement PKCE for all provider OAuth2 flows for better security

To enhance the security of the OAuth2 process from a local desktop app like mine, I’ve implemented the PKCE security extension for all providers, to prevent authorization code interception and injection.

Resolve Architectural Drift

Over the development of this app, I’ve been continually implementing new ideas and removing what hasn’t worked. Eventually, it got to a point where I had thousands of lines of redundant or mismatched code for the sync engine, redundant features laying in my scripts that were no longer planned or used, and many redundant properties for the database schemas as well. Overall I was losing track of my original vision and goals, and development was becoming increasingly slow and stressful.

To help me get out of this development rut, I turned to AI. Specifically, here was a portion of my prompt to Github Copilot:

Please do the following:
1. Analyze my code to figure out the current architecture
2. Identify: 
   a) areas of weakness
   b) inconsistencies
   c) what doesn't work, dead code, etc.
3. Ask me necessary questions needed to clear up any design (or for us to figure out the system architecture together)

Finally, only help me fix any issues in the code once we have a clear idea of everything (and I give you the go ahead).

Over the next day, I went over each of the identified errors or weaknesses one-by-one with the AI. It helped me pinpoint exactly the areas of my code that needed work, and I used it to successfully reintegrated the code with my original vision.

Note: I use ‘cloud’ to refer to a provider-account pair. For example, a Google Drive provider with a specific email is one cloud. A different cloud would be the same provider but a different email, or a different provider altogether.

Here’s a breakdown of what I did:

1. Remove architectural drift, abandoned code, and unused features (e.g. GlobalIgnoreRules, automatic destination selection, FileSyncRecord and FolderSyncRecord)
2. Implement a new-cloud bootstrap (when the user uploads stuff to a new cloud, perform a bootstrap operation to create all existing files and folders)
3. Implement destructive cloud removal for when the user removes a cloud assignment
4. Add pre-sync validation (reject missing local roots, reject invalid assignments)
5. Make Watcher queuing reliable (detect unmonitored changes, etc.)

Devlog #11: Cloud Provider File Actions 2 (WIP)

Google Drive Cloud Provider Actions Complete

Since the last devlog, I’ve successfully completed the logic required to sync files and folders to Google Drive accounts using the Google Drive .NET REST API.
Using the .NET REST API involves first building a drive client service object, through which you’ll use to interact with the REST API for actions like uploading files or deleting them.
The logic is written to keep an exact carbon copy of the local version of any watched folders on the cloud. It does so by storing a queue of all of the changes a user makes to a folder and its children, and automatically ignores any conflicting changes from FileSystemWatcher that could be simplified (e.g. if a user changes an item before deleting it, we can just delete it instead of first updating the changed file AND THEN deleting it).
As the REST API does not support uploading multiple items or entire folders, every file is uploaded one-by-one via a stream. All cloud items are tracked not by their cloud IDs, but rather through their hierarchical relationship with the overarching folder parent (WatchedFolder).

Google OAuth2 Flows

Since the drive client service is created via access tokens (which expire), there’s a very real possibility of our cached access token expiring in the middle of an operation. Thus, I decided to use the Google OAuth2 Flows library to automatically refresh the access token when it expires.

Microsoft Graph and Dropbox File APIs (WIP)

The Microsoft Graph and Dropbox File Action APIs should follow mostly the same high-level structure as before, with minor changes made to accommodate differences in how each of the APIs work.

Microsoft OAuth2 access token expiry handling

With the Microsoft API, I instead implemented a custom class TokenManagerAccessTokenProvider which inherits from IAccessTokenprovider, before passing this into building the GraphServiceClient.

Devlog #10: Cloud Provider File Actions (WIP)

Currently working on the cloud provider services API of my backend, making file actions such as uploading files from the local filesystem to a cloud provider (Google Drive, OneDrive, Dropbox) possible.

Cloud Provider Services (WIP)

I recently just implemented the OAuth process for Microsoft Entra/Graph in order to access the Microsoft OneDrive API, which differs quite significantly from the Google Cloud OAuth process for Google Drive API, despite the fact that both services use the OAuth2.0 protocol. For one, Google uses centralized endpoints and the unified Cloud dashboard to manage desktop apps that require access to Google user information. Microsoft uses their identity platform (Microsoft Entra UID) with endpoints that can be multi- or single-tenant. The actual scopes where the REST APIs for Microsoft OneDrive and other services resides in Microsoft Graph.

Implemented Glob Processing + Ignore Rules for FileSystemWatcher

In my application, the user can implement ignore rules for specific folders, allowing the application to prevent any matching files or folders from being monitored and uploaded to the cloud. The specific glob rules I referenced come from the VS Code docs: https://code.visualstudio.com/docs/editor/glob-patterns

Essentially, I have a collection of methods that parse the glob and turn it into a regex, which I can confirm is a match with a specific path through Regex.IsMatch().

Devlog #9: FileSystem Events

I’ve been spending some time reworking and polishing the logic and database schemas that allows the OS to monitor changes made to WatchedFolders through the FileSystemWatcher class.

Handling FileSystem Events

Here’s the current rundown of how my program monitors for filesystem events (e.g. file created, deleted, renamed, moved, or changed)

filesystem event
    -> record the change in a database queue
    -> return

When a scheduled backup runs, the program reads the queue and the associated changes made to each file/folder, and then updates the corresponding item in the cloud.

scheduled backup
    -> read queued changes
    -> check whether the file still exists
    -> wait/retry if it is locked
    -> upload the latest version
    -> remove the queue item/mark as complete

I’ve also made edits to the OnFileRenamedEvent() to differentiate between renames and moves, which both use the “Renamed” event.

PendingSyncItem

This will be a SQLite table that stores pending sync actions upon being handled by the event handler.

The PendingSyncItem works in a simple way for both files and folders. It tracks five types of boolean events props in terms of cascading importance. Multiple event props can be true.

1. Deleted - the most important event prop. If this prop is true, all other event props are skipped (as they matter not if the file was deleted). A deleted API request will be sent to the corresponding REST API.
2. Moved - creates the file if the file did not previously exist. References the string OriginalLocalPath.
3. Renamed - sends a rename API request.
4. Changed - finds the existing file in the cloud and updates the file.
5. Created - creates a new file. If a file was Created, Deleted, and then Created, two PendingSyncItems will be created. The first one will be overlooked as the Deleted prop is true.

Devlog #8

Database schemas + API planning

I’ve mostly been working on database schemas and models since the last devlog, and I’ve created APIs to allow the frontend to access this information from the database as well. A lot of systems design work has been going on in the background. As I’m still a beginner to such concepts, a lot of rewriting code has also been going on in the background!

To be honest, progress has slowed since I’ve been splitting myself between working on the frontend and backend. I often find myself getting a bit stuck or lost in what’s happening. I think I’ll work on completely finishing one thing before moving onto the next for the next devlog.

Below are snippets of what I’ve been working on, including API methods for the frontend, the API endpoints for the backend, and some peeks at some of the changes I’ve made to existing models/new models that I’ve created.

FileSystemWatcher

I’ll be using the built-in C# FileSystemWatcher in order to track which files and folders the user has made changes to. The alternative is polling for these changes on-demand (e.g. checking the folder’s “Last-Modified” date). However, the problem with such an approach is that Windows does not automatically update a folder’s “Last Modified” date exactly when a file inside that folder is changed. Thus, to confirm whether any changes actually occurred to a folder or not, the program would have to poll ALL subdirectories and files, which could be performance-intensive. On the other hand, FileSystemWatcher is an abstraction over OS-level kernel APIs. The kernel itself registers a listener on an attached folder when FileSystemWatcher is used. Thus it is very efficient and much more accurate than polling a folder’s properties.

Devlog #7

API errors fixed - OAuth now works 100%

In my specific architecture, API calls from frontend to backend work like so:

fetch() call from Vite frontend --> intercepted by WPF WebService2 middleware shell --> forwards down NamedPipe --> received by Kestrel (cross-platform web server) on the ASP.NET Core C# backend. 

For the longest time, I just couldn’t get API calls from the frontend to reach the backend. I would always end up with a TypeError: failed to fetch(), which, as I found out, could mean a lot of things. Initially, I thought it was a CORS issue with the backend. I added builder.Services.AddCors() to allow cross-origin requests, and also adjusted the content-headers included with the fetch() request on the frontend, but I still experienced the same issue.

Eventually, I discovered that it was a CORS issue with the MIDDLEWARE. It turns out after the WPF WebView2 shell forwards the fetch() request down the NamedPipe and retrieves a response from the backend, it actually creates a new web request to return it to the frontend with createWebResourceRequest(). This new response DOESN’T COPY the CORS headers from the backend, meaning the Access-Control-Allow-Origin header was missing from the response despite the backend returning a 200 OK status code. This blocks the frontend from reading the response, and a generic TypeError: Failed to fetch is thrown.

Since the desktop app is essentially a proxy/interceptor - it’s creating its own HTTP response, so it needs to handle CORS headers itself.

I was also missing an OPTIONS preflight responder on the backend, so I added that in as well.

Listener callback dynamic port access

When OAuth is requested from the frontend to the backend, the backend adds a listener to the provider’s OAuth function call. This way, say when the user completes Google’s OAuth process, Google redirects to our listener url, and we can display a Success page to the user.

I had a minor logic error in my listener callback implementation where the same port was always used, even if it was unavailable. For this reason, if the user were to leave the OAuth page and retry the OAuth process, they would encounter an error if the listener was still not cleaned up. I implemented a dynamic port searcher to automatically serve the user a port that was free and unused.

Implementing EF Core Backend System and FileSystem schemas

I’m currently planning to implement the core of the backend functionality, which is to monitor file changes occuring to “watched” folders and upload them asynchronously and autonomously to providers the user has signed into.

A great way I’ve found to map out future app functionality is to map out schemas for anything that needs to be stored locally. Currently, I’m working on schemas for SystemSettings, SyncEvents, and FileSystemWatcherWorker.

Below is a short overview of the current basic features of the app, including OAuth account management + a notification/error-logging system.

Devlog #6

These past days I’ve been focused on cleaning up the auth code and setting up frontend-backend communication.

IScopeFactory - Singletons and Scoped Objects

.NET (modern C#) features native dependency injection, where abstractions can be mapped directly to a class, in order to allow that class to receive its required objects from an external source in a more efficient and cleaner manner.

In ASP.NET Core C#, you can register classes as a couple of different types of services to be injected into other classes. I mainly use singletons and scoped objects. Singletons, like their name suggests, are long-lived classes, of which a single instance exists throughout the application’s lifecycle. By contrast, scoped objects are created once per client request before being destroyed.

The problem is that when you use dependency injection, you cannot inject a shorter-lifespan class into a longer-lifespan class (e.g. a scoped into a singleton). This necessitates the use of IServiceScopeFactory to automatically manage the lifetimes of these child classes.

Thread-Safe Singletons

By default, the System.Collections.Generic library in C# is NOT threadsafe. That means that, if multiple scoped objects or singletons access the same Dictionary, that could throw an error. Thus, I switched them out for ConcurrentDictionaries throughout my app.

Multi-Account Functionality Integration

I further integrated functionality that allows my app to store MULTIPLE accounts per provider (e.g. 3 OneDrive accounts, 4 Google Drive accounts). This required refactoring the CloudToken object to include a new AccountId prop, retrieving that prop from the OAuth2.0 protocol, and rewriting some logic in TokenManager to include AccountId validity checks.

RevokeOAuthSync

I added a method to revoke OAuth by terminating the refresh and access tokens using the revoke OAuth2.0 endpoint.

Frontend-Backend Communication

With the basics of the auth protocol API and TokenManager API setup, I’ve begun attempting to facilitate API calls between the React Vite frontend and the ASP.NET Core C# backend. It’s still a WIP, but things are moving! I also added some additional UI to allow the user to select and manage their accounts on different providers.

Devlog #5

I’m currently on vacation, but I’m trying to stay productive by working on some software projects.

OAuth2.0 Token Management + EF Core SQLite Setup + HTTPClient/DPAPI

The past few days, I’ve been working on setting up some basic OAuth functionality for Google and Google Drive. I thought it would be very simple at first, but perhaps since I’m new to C# and on vacation, it took me a bit to get used to the C#/ASP.NET features and development patterns, such as dependency injection.

Additionally, there’s plenty more to using OAuth 2.0 that I was not aware of at first. Step 1) involved acquiring ClientId and Clientsecret values from the Google Dev Console page. These were stored in User Secrets in Visual Studio along with the project.

For desktop clients, clients use the provider’s OAuth API link to apply for a long-lived Refresh Token. This Refresh Token, once encrypted and stored securely, is used to continuously request for short-lived Access Tokens, which is what is actually used for Google’s API services, like the Google Drive API.

Encryption was setup using the .NET data protection API, DPAPI. I created two classes: a GoogleDriveProvider to manage Google-specific OAuth setup and Drive API calls, and a generic TokenManager that handles the refresh-access token exchange for all OAuth-related providers, including ones I will add in the future such as OneDrive and Dropbox.

For the HTTPClients used to make the web requests, I decided to add an HTTPClient to GoogleDriveProvider using builder.Services.AddHttpClient().

Finally, all pertinent information to the app is stored within an SQLite database using EF Core. Refresh Tokens are stored within an object called CloudTokens.

I have the basic functionality for the OAuth setup, so my next steps will just be to test it out and debug for any errors.

Below are some snapshots from the code I wrote.

Devlog #4

WPF WebView2 Setup

Succeeded in porting (?) the Vite frontend to Wpf WebView2! I’m just happy it works now. There was an issue with the Configuration Manager/build setup, where WebView2 turned out to be incompatible with the Any CPU build, and so I had to convert the solution to an x64 platform build, which works out since that’s what I intended to do in the first place.

For now, the source of the WebView2 WPF frame is just a localhost url. When prod version comes out, I’ll be bundling a folder containing the Vite frontend along with the app, which will then become the new source.

Devlog #3

React Vite UI work

I’ve finished the skeleton and main UI features of my frontend Vite app (unfortunately took longer than expected). Nothing actually works cause it’s just a frontend for now.

I added…

1. A dashboard page (for quickly viewing information about cloud providers and storage quotas)
2. A folders & files page (for “staging” folders and files for autonomous backups in the background)
3. A sync page (for pausing/starting sync)
4. A timeline page (for viewing sync events and chronology)
5. A link services page (for linking accounts from different cloud providers)
6. An about page

No functionality actually exists yet as it’s just for planning.

I’ll be implementing the backend next and the WebView2 functionality, before establishing communication between the frontend and backend via Windows pipes.

Devlog #2

React Vite + WebView2 + ASP.NET Core C# transition

I’m in the middle of transitioning from a Winforms + Windows Service backend architecture to a decoupled C# ASP.NET Web Service + React Vite frontend with WebView2.

WebView2 allows developers to embed web content (HTML/CSS/JS) directly into native Windows apps. I considered using Electron, but WebView2 is widely seen as more performant as it uses the Chromium runtime already built into Windows, as opposed to Electron which bundles a SEPARATE instance of Chromium and Node.js into every app, resulting in a large footprint.

I’ve redone the UI in Vite and setup the basic ASP.NET endpoints using pipes. The next steps would be to finalize a good-looking UI and create the WPF project that would host the WebView capabilities.

Devlog #1

Basic Winforms UI

Setup a basic UI in Winforms, which took longer than I thought. Unforunately, at the time I started this project, I didn’t know about the possibilities of a ASP.NET + React setup. I wanted to avoid Electron due to the heavy resource consumption, but since the possibility of using WebView2 exists for me now, I might transition to that soon.
I just don’t mesh very well with Winforms…