Produce Inflation Tracker vs. HTML Madness: A Journey Through the Chaotic Web Layouts of Grocery Websites

Jokes aside: how price parser works

Web scraping and data extraction system is designed to monitor product prices across multiple international retail websites. The system employs a template-based approach for HTML parsing, utilizing both Selenium WebDriver for controlling real browser for dynamic content handling and BeautifulSoup for HTML parsing. The architecture supports multi-store, multi-country price tracking with built-in inflation rate calculations and standardized unit conversions.

The system's flexibility stems from its configuration-driven approach. The store_config.txt file defines scraping templates for each retail store, containing structured data blocks that specify how to extract product information from different website layouts.

System workflow

The main execution flow orchestrates all system components in a coordinated sequence, processing each store configuration and extracting data for both cheapest and most expensive product variants.

Processing Loop Structure

The system iterates through each store configuration, processing both cheapest and most expensive variants when URLs are available. For each variant, the system:

1. Fetches the webpage content using Selenium WebDriver
2. Applies template matching to extract product title and price information
3. Processes the extracted data through the standardization pipeline
4. Calculates derived values such as price per unit and inflation rates
5. Saves the processed data to the database with appropriate metadata

Configuration format structure

Each store configuration follows a standardized format with five primary sections. Let's examine the Walmart configuration as an example:

STORE = Walmart
COUNTRY = USA
PRODUCT = Bread

TITLE = [
<span data-automation-id="product-title" class="normal dark-gray mb0 mt1 lh-title f6 f5-l lh-copy">FFF</span>
]

PRICE = [
<span class="w_iUH7">current price FFF</span>
]

CURRENCY_MAP = ["$": "USD"]

URLS = [
cheapest: https://www.walmart.com/search?q=bread&sort=price_low
most_expensive: 
]
        

The "FFF" placeholder acts as a dynamic content marker, indicating where the actual product data should be extracted from the webpage. This templating approach allows the system to handle diverse HTML structures across different retailers.

Configuration components

The configuration parser processes each store block by identifying key sections and extracting the relevant information:

• Store sdentification: STORE, COUNTRY, and PRODUCT fields establish the basic metadata for each scraping target
• HTML templates: TITLE and PRICE sections contain HTML snippets with "FFF" placeholders that guide the extraction process
• Currency mapping: CURRENCY_MAP translates currency symbols to standardized currency codes
• URL variants: URLS section provides endpoints for both cheapest and most expensive product sorting

Database schema and relationships

To efficiently store price tracking data across multiple dimensions the system implements a normalized relational database structure using SQLite.

The PriceSample table serves as the central repository, linking stores and product types while maintaining historical price data with calculated inflation rates.

Web scraping implementation

The system employs a dual-approach web scraping strategy, utilizing both Selenium WebDriver for JavaScript-heavy sites and falling back to HTML-based scraping for simplicity.

Selenium configuration

The Selenium implementation uses Firefox WebDriver with extensive customization to handle modern web security measures and anti-bot detection systems:

def fetch_page_selenium(url):
    firefox_options = FirefoxOptions()
    firefox_options.binary_location = r"d:\...firefox.exe"
    firefox_options.set_capability("moz:webdriverClick", False)
    firefox_options.set_preference("dom.webdriver.enabled", False)
    firefox_options.set_preference("webdriver_accept_untrusted_certs", True)

Template-based data extraction

The core extraction mechanism operates through a template matching algorithm that compares HTML elements from the configuration templates with actual webpage content.

The template matching process involves several steps:

1. Element identification: The system parses template HTML to identify elements containing "FFF" placeholders
2. Attribute matching: For each template element, the system searches for corresponding elements in the scraped page with matching tag names and attributes
3. Content extraction: Once matched, the system extracts the actual content replacing the "FFF" placeholder
4. Duplicate handling: The algorithm prevents duplicate extraction from nested elements through relationship tracking

Data processing pipeline

The extracted raw data undergoes comprehensive processing to standardize formats, calculate derived values, and ensure data quality before database insertion.

Price information processing

The extract_price_info function handles the complex task of extracting numerical price values from diverse string formats across different locales and currencies:

def extract_price_info(price_string, currency_map):
    # Handle European decimal format (comma as decimal separator)
    if ',' in price_clean and '.' not in price_clean:
        price_clean = price_clean.replace(',', '.')
    elif ',' in price_clean and '.' in price_clean:
        # Handle format like "1.234,56" (European thousands separator)
        if price_clean.rfind(',') > price_clean.rfind('.'):
            price_clean = price_clean.replace('.', '').replace(',', '.')
        

The system processes different decimal and thousands separators used across various countries, ensuring accurate price extraction regardless of regional formatting conventions.

Package size and unit processing

Product package information extraction employs regular expressions to identify and parse size and unit data from product titles:

def extract_package_info(title_string):
    patterns = [
        r'(\d+(?:[,\.]\d+)?)\s*(oz|lb|g|kg|ml|l|fl oz|г|мл|л|кг)\b',
        r'(\d+(?:[,\.]\d+)?)\s*(ounce|pound|gram|kilogram|liter|litre)\b'
    ]
        

The system maintains unit conversion tables to standardize measurements across different regional systems:

UNIT_PATTERNS = {
    "oz": 0.0283495,
    "fl oz": 0.0295735,
    "lb": 0.453592,
    "kg": 1.0,
    "g": 0.001,
    "l": 1.0,
    "ml": 0.001,
    "г": 0.001,    # Cyrillic gram
    "мл": 0.001,   # Cyrillic milliliter
    "л": 1.0       # Cyrillic liter
}
        

Price per unit calculations

The system calculates standardized price-per-unit values to enable meaningful price comparisons across different package sizes and measurement systems:

            
calculate_price_per_unit(price_number, package_size, package_unit, currency)
            
Converts package sizes to standard units (kg for weight, liter for volume) and calculates normalized price-per-unit values for comparison purposes.
        

Inflation rate calculation

The system incorporates temporal price change tracking by calculating inflation rates based on historical price data stored in the database. This feature enables tracking of price changes over time for each product-store combination.

def calculate_inflation_rate(current_price, previous_price):
    if previous_price is None or previous_price == 0:
        return 0.0
    inflation_rate = ((current_price - previous_price) / previous_price) * 100
    return inflation_rate
        

The inflation calculation retrieves the most recent price record for each product-store-variant combination and computes the percentage change from the previous measurement. This enables tracking of price trends and identifying price fluctuations.

Error handling and data validation

The system implements comprehensive error handling and data validation to ensure data integrity and system reliability:

The system validates that both product title and price information are present before attempting database insertion, preventing incomplete records from corrupting the dataset.

Validation mechanisms

• Empty data checking: Validates that extracted product titles and prices contain meaningful content
• Database constraint enforcement: Utilizes SQLite constraints to maintain referential integrity and prevent duplicate entries
• Exception handling: Try-catch blocks protect against network failures, parsing errors, and database connection issues
• Selenium timeout management: Implements waiting strategies to handle dynamic content loading

Database operations and data management

The system implements a data management strategy using SQLite with proper transaction handling and data integrity constraints.

Database helper functions

The database operations use INSERT OR REPLACE statements to handle potential duplicate entries gracefully, ensuring that the latest price information overwrites previous entries for the same date and product combination.

Performance considerations and optimizations

The system incorporates several performance optimizations to handle price monitoring efficiently:

• Connection pooling: Database connections are managed efficiently with proper opening and closing patterns
• Batch processing: Configuration parsing and database operations are optimized for batch execution
• Memory management: BeautifulSoup objects are created and destroyed appropriately to prevent memory leaks
• Selenium lifecycle: WebDriver instances are properly initialized and terminated for each scraping session

Extensibility

The system architecture facilitates easy extension and maintenance through several design patterns:

Configuration-driven design

Adding new stores requires only configuration file updates without code modifications. The template-based approach allows the system to adapt to different HTML structures through configuration changes.

Modular function design

Each major operation is encapsulated in dedicated functions with clear responsibilities, making the system easier to debug, test, and extend. The separation of concerns between web scraping, data processing, and database operations enables independent modification of each component.

Scaper limitations

While the system provides robust price monitoring capabilities, several limitations should be considered:

• Modern websites employ sophisticated anti-bot detection systems that may require ongoing adaptation of the Selenium configuration and request patterns.
• Website layout changes require corresponding template updates in the configuration file
• Network failures and temporary website unavailability require designing of respective mechanisms

Conclusion

The Product Price Parser represents an approach to automated price monitoring, combining flexible configuration management with robust data processing capabilities. The system's architecture balances complexity with maintainability, providing a scalable foundation for international price tracking. Through its template-based extraction engine, comprehensive data processing pipeline, and normalized database structure, the system enables efficient monitoring of product prices across diverse retail environments.

The implementation demonstrates advanced web scraping techniques, proper database design principles, and thoughtful error handling strategies. The system's modular design and configuration-driven approach ensure that it can adapt to changing requirements and expanding monitoring scope without requiring significant architectural modifications.