Ooopppss I exposed jwt tokens in github

Shelf-fish Devlog #8

Yepp I exposed my jwt tokens in my github, but trust me I knew what I was doing, those tokens are long expired, my backend isn’t published yet and i will change the exposed passwords, which was 12345678, cuz why not

Anyways I have done a lot things, in the last devlog, i had connected the frontend with the backend and made the detection endpoint work with the frontend. i had to manually type the authentication tokens cuz i had not set up auth in the frontend, but I have now fixed this

Here’s what I have done:

• Created tokenStorage.js - it handles getting the tokens, saving them and clearing them using expo-secure-store, i used expo-secure-store cuz it saves tokens in keychains rather than files that makes them much less likely to leak or read by anyone/app/malwares
• Created authService.js- this one handles send the get and post requests for registering, logging in and getting current user data
• Added interceptors to Api, i am using a single api instance using axios.create() across the app, so i have added interceptors to attach auth tokens and check if they are valid or expired
• Created the login and register screens, they don’t look good now but will work on the ui after i get everything set up, I will make everything exist first, the designs can be taken care of afterwards
• I also created a new api route at ‘/users’, this one is responsible for getting user data, right now it just gets the user info but I think i would also make it update it…
• I also added navigation using react-navigation

What’s next ?

• I haven’t finished the home screen yet, well nothing is finished yet, but the home screen just greets the user, i will add buttons to navigate to camera and upload components
• I will also create the item related pages, that will show all items and let the user create items after detecting the product

The images:

They look soo bad but as I said I will make everything exist first and then make the ui good

1. The register page
2. The login page, it’s not even filled yet, cuz i want to add text and a logo, i will create a logo too lol i forgot about that
3. The home page, empty page with a greeting and a sign out button, it was made to check if everything is working or not

Frontend Connected

Shelf-fish devlog # 7

In the last devlog, I had finished creating the detection logic using gemini api, after that I have done a lot of things and suffered a lot of headaches too

What have I done??

• Created api endpoints for detections at /detections and added it to the fastapi app.
• Added authentication dependency to the detection api so that only authenticated users can upload images and we can know who detects which item.
• Created the Camera and Upload image logic, I have used expo-camera and expo-imagepicker for that, it was pretty easy, expo’s docs are straight forward and I have used them before
• Created the uploading function that will send a post request containing the image to the backend.
• I also added storage fn to upload the captured/picked image to cloud using cloudinary.

What’s next?

• Now I will work on the ui and create login and register screens cuz manually typing the authentication token to the uploading fn is really frustrating
• I also have an idea for this project which would make it very unique cuz at this point it just looks like just any other inventory management systems.

The images-

1. console log of the item after it was detected by gemini and saved to database,
2. the image being displayed after clicking the url,
3. the images in cloudinary, the other one is stock image and was there automatically.

Detection Working!!!

Shelf-fish Devlog #6

TL;DR

What have done?

• Created a basic jwt auth
• Used Gemini Api and Gemini 3.1 Flash Lite to create the product detection logic

What’s next?

• I will now make api routes for detection that would output the brand, item name and category of the item

Got lots of time?

Sooo the auth and detection logic is complete, we are now close to ending the backend, earlier after dropping the plan of using a classification model, I had decided to use an ocr to detect text from the input photo and then use that text as an input for openai api and ask it to identify the brand name, item name and category from the image, but since I am broke , I couldn’t use openai api as it costs money and then learned that gemini has free api plans that include multi-modal inputs like images and give out text, so I swited to that and it works flawlessly, the rate limits are also generous, at 500 requests per day and 15 per minute. I tested it with 6 text images and it gave 100% acc but the sample size is pretty small to say that it is 100% accurate, but it does work.

The detection logic, at this point, is just a function, i need to make api routes for that and add a detection log table, the log table would be useful at providing data about rate limits and is also nice to have if the project get bigger, I also have to make a list of categories out of which gemini would choose the category of the product cuz at this point, it outputs anything it likes as a category

Backend is Up

Shelf-fish Devlog #5

What have I done??

• Set up FastAPI and SQLModel - I have successfully connected and set up the backend with the database, I am using neon’s postgres db

• Created Item table - I also created the first table of item, it has the following columns- Name, id, quantity, category and price

• Set up CRUD - set up basic crud end points for the item table

What’s next??

• I will now create auth using pyjwt and passlib
• Next I will work on the core mechanism for extracting name from the product image using ocr

Dropping the plan

Shelf-fish Devlog #4

Heyy gng, since the past few days I have been just editing and reviewing images and they don’t seem to finish any time soon but stardance is finishing soon and I want to finish this project before that so I am dropping the workplan of using a classification model for identifying items

That approach was flawed, I had not thought about that earlier instead I was biased by my previous successful project using the classification approach to identify pokemon

I think a better approach would be to use an ocr and then ask an llm to parse that into brand name, item name and weight and other things

Despite the my orignal plan failing, I want to highlight that engineering, in all forms, is itterative, no one finds the solution to the problem in their first try and the things I have made before, ie- my scraper, is not obselete at all, it works so good, like it scraped 150k images in 6 hours, that’s really insane and very useful too, I will make it as another seperate project in the future cuz I think it deserves some more recognition and will improve it too

first 3 images- images of clean dataset, products are cropped and all noises are deleted
last image - for comparison, an unclean dataset, a lot of noises in it like people and some images dont even have the product

Who is Who - Shelf-Fish devlog #3

There were so many duplicate images

So in the last devlog, I told my scraper scraped nearly 150k images, but when I began cleaning them I noticed a lot of images were exact duplicates, I thought it was just for a class or two but It was for every single one of them , except Oreo

After manually cleaning 14 classes with duplicates, I gave up, and decided to use python to delete duplicates, I used hashlib and md5 hash to detect exact duplicate images inside all folders of my root train images folder which each represent one class, and I used os.remove to delete those duplicates.

I ended up deleting around 34k images which brings my dataset from 150k to 100k images, but this would change too cuz the 100k maybe unique but they still have a lot of discrepencies, like random products and all.

I also used matplotlib to plot a graph of the images remaining, and the results dont look good for a few classes which have now less than 300 images, others are in range 500 - 800, remember they were around 1100 images earlier, so around 800 images were duplicates of the remaining 300.

I know what’s causing this, multiple search queries, since google images only displays a fixed 400 images per search, I made my scraper in a way that it searches for the same product multiple times but with a different query which causes popular images to be duplicates

One more thing, the dataset would not only decrease in number, but after cleaning the images, I would augment them which would drastically increase the number of images and also make the model less likely to overfit.

Also the graph is really ugly cuz there are like 130 classes on the x axis and they dont really fit well and I am not an expert in pyplot so I dont know how to make it look good

Shelf-Fish Devlog #2

My Scraper is on steroids !!!

I scraped 150,000 images

Yooooo guys the scraper worked too good now, it scraped roughly 150,000 images in just 6 hours, that is insane, the speed was around 7 images per second or 420 images per minute, that is really crazy, my old scraper took a full day for 75k images, this is like a million times faster!!!

In the last devlog, I acheived a speed of around 1079 images in 10 minutes or around 100 images per hour, so since then I have made it 4x times faster.

Here’s what I did:

• Added premature scrolling - earlier the scraper would miss a lot of images or would wait for a long period of time for the images to load and then scroll, but now as soon as the main div appears on the screen, the scraper scrolls down a bit triggering lazy load instantly and then scrolls down to the bottom triggering te lazy load 2 times in a cycle, this made the whole scrolling thing so much quicker.

• Fixed downloads not cancelling - I am storing some extra urls so that if one or more of the urls failed to download, I have backup for completing the target, but I am using a ThreadPoolExecutor to download images concurrently, when it reaches the target the executor used to just break but this didnt prevent it from downloading all other images stored in my array even tho they aren’t saved on the disc, this caused the scraper to wait indefinitely for a long time, since it now has to fetch double or sometimes more the amount, but I have now fixed that

Next steps?

I will now clean the dataset, and since by cleaning the dataset I am actively contributing to the completion of the project, I have got permission to use lapse to record that time

Shelf-Fish Devlog #1

What is Shelf-Fish?

It is a computer vision based cross platform inventory management and billing system that I am making with react native as frontend and probably a fastapi backend.

How will the project work?

The project would be made in several phases that are listed below:

• Phase 01 - Scraper

  In this phase I will make a fast scraper with selenium that would scrape 1000-1500 images for over 130 grocery items that I have listed.

• Phase 02 - Training

  In this phase I would clean the dataset and train the model!!

• Phase 03 - Backend

  I would build a basic cors backend in fastapi and add some user relations

• Phase 04 - Frontend

  This phase involves building a cross platform fronend with react native.

• Phase 05 - ESP 32

  This one is not confirmed yet, but if i get the time, instead of using a phone, I would use the esp 32 and esp32 cam to register objects and create bills, or perhaps I should make this the main focus of the project and make this a hardware project, I am still not sure.

What have I done till now?

I am modifying my old scraper that I used for my previous pokedex project, which was very very slow, I have changed it completely, It now directly goes to the image search url instead of going to google.images.com and then manually typing the words. It now also downloads images concurrently using the ThreadPoolExecutor, earlier this part was the slowest but in just 2 seconds, it downloads 125 images,

But it is still pretty slow, It took over 10 min to scrape 1079 images, and it was a bit short of the 1125 target images which is due to slow network that caused the images to not load. For the next devlog I will try to make it sub 8 min or faster