Truck Inventory Scraper

TRUCK INVENTORY SCRAPER Go-based Web Scraping & Email Matching System	Language	Go
Portfolio	Code

Code in this repository

Path

Code lives in this repo at code/inventory/. Clone the portfolio repository to get the source.

How to run

Clone this repository, then open a terminal in the-monospace-web/code/inventory.
Install Go if needed. Ensure PostgreSQL is running and configured (connection string or env).
Configure Microsoft Graph API credentials (OAuth2 client credentials) for Outlook email matching.
Run go build then execute the binary. The app starts the scraper coordinator and email matcher in the background; it will run an initial scrape and then periodic scrapes (about every 45–75 minutes) and poll Outlook for new emails to match.
For production, use the deployment/ systemd service config.

code/inventory/

Overview

Production-ready Go application that scrapes truck listings from 8 major marketplaces and automatically matches them to client email requests via Outlook integration using Microsoft Graph API.

Architecture

Data Flow

Marketplace websites are scraped via Rod browser automation into the scraper coordinator, which saves listings to PostgreSQL. The email matcher queries the database for matches and fetches emails from Microsoft Graph API (Outlook); it runs in the background alongside the scraper.

Key Components

Scraper Coordinator - Orchestrates scraping from 8 marketplaces using Rod (Go-native browser automation)
Database Layer - PostgreSQL with pgx/v5, auto-creates schema, implements upsert logic
Email Integration - Microsoft Graph SDK for Outlook, OAuth2 client credentials flow
Intelligent Matching - Parses email content to extract search criteria and matches to listings

Project Structure

inventory/
├── main.go              # Main orchestration
├── scraper.go           # Scraper coordinator
├── database.go          # PostgreSQL layer
├── email.go             # Outlook integration
├── ready_deals.go       # Deal processing
├── agent/
│   ├── extract.go       # Listing extraction
│   ├── listing.go       # Data structures
│   └── client.go        # LLM client
├── scrapers/
│   ├── arrowtruck/      # JSON API scraper
│   ├── selectrucks/     # JSON API scraper
│   ├── truckplanet/     # JSON+HTML scraper
│   ├── ironconnect/     # HTML scraper
│   └── ...              # Additional scrapers
└── deployment/          # Systemd service config

Marketplace Support

Supports 8 major truck marketplaces with different data extraction strategies:

Pure JSON APIs - ArrowTruck, SelecTrucks (direct API calls)
JSON with HTML - TruckPlanet, TruckTractorTrailer, HeavyTruckDealers
HTML-only - IronConnect, IronPlanet, TruckertoTrucker (browser automation)

All scrapers produce unified []*agent.ListingExtract output for consistent processing.

Dependencies

Key dependencies from go.mod:

go-rod/rod - Go-native browser automation (no Node.js dependency)
jackc/pgx/v5 - PostgreSQL driver (direct SQL, no ORM)
microsoftgraph/msgraph-sdk-go - Official Microsoft Graph SDK
PuerkitoBio/goquery - HTML parsing

Features

Multi-marketplace scraping with configurable CSS selectors
PostgreSQL database with automatic schema creation
Outlook email integration via Microsoft Graph API
Intelligent email parsing to extract search criteria
Automated matching and response emails
Production-ready systemd service configuration
Graceful error handling and logging