@Kanga2050

I have made the attempt to start sorting and generating insights for the entire OpenAlex Database. I have developed code to create and sort CSVs for every single topic (of which there are about 4000). As it turns out, there are about 400 million papers with abstracts, which means every topic has about 100 thousand papers, just enough for effective analysis. The calculations should take about a month, but can be paused and continued, etc. I have implemented efficiencies to make the project feasible. I am excited to see what I learn.

Doing this I hope to:

Identify cross disciplinary topic gaps
Generate insights on the structure of the entire database through heighrarchical clustering
Identify per term reductions across fields to analyze confidence in sentiment.

Ok so I made some big changes. First of all, I updated the topic classification to a dedicated model, which I later hope to integrate with the visual representation. Additionally, I cleaned up code and explanations.

In the future I hope to:

Create a single python executable to run all the code and generation
Fully leaf topic generate on the research paper repository itself
Integrate new data sources
Improve data cleaning to recognize unknowns semantically
Also make topic representation fully semantic as well with a dot product distance.

Also, check out this topic hierarchy:!!!

Ok so I made some big changes. First of all, I updated the topic classification to a dedicated model, which I later hope to integrate with the visual representation. Additionally, I cleaned up code and explanations.

In the future I hope to:

Create a single python executable to run all the code and generation
Fully leaf topic generate on the research paper repository itself
Integrate new data sources
Improve data cleaning to recognize unknowns semantically
Also make topic representation fully semantic as well with a dot product distance.

Also, check out this topic hierarchy:!!!

Consolidated scraper code into the python journal for ease of use. If anyone could test on their end to see how it runs on a different computer, that would be great. Additionally, I cleaned up other sections of code, and I shortened some explanations and comments that were previously needed but now not. In the future, I plan to ensure that the 3rd phase of the analysis always works properly (it seems to currently work about a third of the time, depending on the cleanliness of the data from the api for whatever reason?) I tested the code with other prompts, and it seems to work well ig. I don’t even know. Som1 help pls lol.🙃

Consolidated scraper code into the python journal for ease of use. If anyone could test on their end to see how it runs on a different computer, that would be great. Additionally, I cleaned up other sections of code, and I shortened some explanations and comments that were previously needed but now not. In the future, I plan to ensure that the 3rd phase of the analysis always works properly (it seems to currently work about a third of the time, depending on the cleanliness of the data from the api for whatever reason?) I tested the code with other prompts, and it seems to work well ig. I don’t even know. Som1 help pls lol.🙃

Began code, inspired by a research paper I read. I have created a preliminary graph based on the research area “Space Biology”. Using BERTopic on sentences identified to have words with intent similar to “Unkown” or “To be researched”. Out of 200,000 research papers processed, 25520 were used in the final training.

Began code, inspired by a research paper I read. I have created a preliminary graph based on the research area “Space Biology”. Using BERTopic on sentences identified to have words with intent similar to “Unkown” or “To be researched”. Out of 200,000 research papers processed, 25520 were used in the final training.