Tech

I love tinkering and working on tech projects (mostly software). All my public projects are on my GitHub, but here are the ones I’m most proud of:

Projects

Ohana - a search engine for the living web (source)

With most human web traffic today being consolidated into a handful of centralized, proprietary platforms, I wanted to create an easy way to find cool indie¹ websites². With how much of the web is AI-generated slop and/or malicious, filter this cream of the crop has been no easy feat.

I used Docker Compose to orchestrate a list of concurrent core services to achieve this:

• Database: Postgresql. I chose Postgres mainly because I planned for most of the services to be simultaneously interacting with it in some way, and because of its extensibility should I need to introduce in-memory cacheing or vector embeddings.

• Crawler: written in Go, it leverages goroutines to crawl dozens of pages concurrently, with each routine handling a single domain to ensure a polite request rate. This approach maximizes concurrent requests while ensuring respect for sites’ server resources. It writes the content of every page it finds to the database server.

• Indexer: written in Java, the native language of the Apache Lucene library, which it uses to create a searchable index based on the web pages written to the database by the crawler. (This service is currently being rewritten in Rust to remove the memory overhead of the JVM, using tantivy instead of Lucene.)

• Query Engine: also in Java, it RESTfully accepts user queries from the client, forwarding them to Lucene’s search methods to return search results from the index back to the client.

• Client: written in PHP and still bare-bones while I iron out the other core services.