bulk-questionnaire-upload/backend/README.md

mForm Bulk Upload Backend

A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities.

---

Table of Contents

• Overview
• Architecture
• API Endpoints
• Database Schema
• Excel File Requirements
• Setup & Installation
• Error Handling & Logging
• Performance Metrics
• Development & Maintenance Tips
• Extending the Backend

---

Overview

This backend provides:

• Validation of Excel files for correct structure and content
• Parse-only functionality for schema preview without database storage
• Parsing and storage of forms, questions, and answer options in MongoDB
• RESTful API endpoints for form management, including in-place form updates
• Comprehensive error handling with detailed error types and suggestions
• Performance metrics logging for monitoring and optimization
• CORS support for seamless frontend integration

---

Architecture

Main Components:

• main.py: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
• services/xlsform_parser.py: Business logic for validating and parsing Excel files
• services/database_service.py: Async database operations for forms, questions, and options
• database.py: MongoDB connection management and collection handles
• models/: Pydantic models for validation, API responses, and internal data structures

Startup Flow:

• Loads environment variables from .env
• Connects to MongoDB on startup, closes on shutdown
• Exposes API endpoints under /api/

---

API Endpoints

File Validation

• POST /api/validate
  • Description: Validate Excel file structure and content
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "valid": true,
      "message": "File format is valid.",
      "sheets": [...],
      "form_metadata": {...},
      "questions_count": 10,
      "options_count": 30,
      "errors": [],
      "warnings": []
    }
    

File Parsing (Parse Only)

• POST /api/forms/parse
  • Description: Parse Excel file and return JSON schema without saving to database
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "id": null,
      "title": {"default": "Form Title"},
      "language": "en",
      "version": "1.0",
      "groups": [...],
      "settings": {...},
      "metadata": {
        "questions_count": 10,
        "options_count": 30,
        "parse_time": 0.082,
        "created_at": "2025-07-30T00:00:00Z",
        "sheets_found": ["Forms", "Questions Info", "Answer Options"],
        "file_name": "example.xlsx"
      },
      "raw_data": {...}
    }
    

  • Error Response:

    {
      "detail": {
        "error": "Parsing failed",
        "message": "Missing required sheet: Forms",
        "error_type": "MISSING_SHEET",
        "suggestions": [
          "Ensure your Excel file contains a sheet named 'Forms'",
          "Check sheet names for typos or extra spaces"
        ]
      }
    }
    

File Upload

• POST /api/upload
  • Description: Parse and store one or more Excel files
  • Request: multipart/form-data with one or more files
  • Response: List of parsed form objects or error details

Update Form

• PUT /api/forms/{form_id}/update
  • Description: Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

Forms Management

• GET /api/forms

  • Description: List all forms
  • Response: { "forms": [...], "count": 2 }

• GET /api/forms/{form_id}

  • Description: Get a form with its questions and options
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

• DELETE /api/forms/{form_id}

  • Description: Delete a form and all related data
  • Response: { "message": "Form deleted successfully" }

• DELETE /api/forms

  • Description: Delete all forms and related data
  • Response: { "message": "All forms deleted successfully" }

---

Database Schema

Forms Collection

{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}

Questions Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}

Options Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}

---

Excel File Requirements

The backend expects Excel files with three sheets:

1. Forms
   • Columns: Language, Title
2. Questions Info
   • Columns: Order, Title, View Sequence, Input Type
3. Answer Options
   • Columns: Order, Id, Label

Validation will fail if required sheets or columns are missing.

---

Setup & Installation

Prerequisites

• Python 3.8+
• MongoDB (local or cloud instance)
• pip

Installation Steps

1. Install dependencies:

   pip install -r requirements.txt
   

2. Set up MongoDB:
   • Install locally or use MongoDB Atlas
   • Create a database named mform_bulk_upload
3. Environment Configuration:
   • Create a .env file in the backend directory:

     MONGODB_URL=mongodb://localhost:27017
     DATABASE_NAME=mform_bulk_upload
     API_HOST=0.0.0.0
     API_PORT=8000
     

4. Run the application:

   uvicorn main:app --reload
   

   The API will be available at http://localhost:8000

---

Error Handling & Logging

• All API endpoints use structured error handling with FastAPI's HTTPException.
• Enhanced error responses include specific error types and actionable suggestions.
• Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more.
• Errors and warnings during file validation are returned in the API response.
• Application-level errors are logged using Python's logging module (see main.py, services/).
• Metrics (e.g., processing times, counts) are logged to metrics.txt for performance monitoring.
• Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.

---

Performance Metrics

The application logs detailed performance metrics to metrics.txt including:

• Validation times: File validation performance per form
• Parse-only times: Schema parsing without database operations
• Upload processing times: Complete form processing with database storage
• Question/Option processing: Individual item processing performance
• Cold startup tracking: Application initialization times

Development & Maintenance Tips

• Centralize logic: All business logic is in services/, and all DB access in database_service.py.
• Environment variables: Use .env for DB config; never hardcode secrets.
• Testing: Use tools like httpie or Postman to test endpoints.
• Extending: Add new endpoints in main.py and corresponding logic in services/.
• Logging: Check metrics.txt and logs for troubleshooting and performance analysis.
• CORS: Update allowed origins in main.py if frontend URL changes.
• Error handling: Use structured error responses with specific error types for better user experience.

Extending the Backend

• Add new endpoints: Define in main.py, implement logic in services/, and update models as needed.
• Add new collections: Update database.py and database_service.py for new MongoDB collections.
• Validation: Extend XLSFormParser for new validation rules or file formats.
• Error types: Add new error classifications in the parsing service for specific failure cases.
• Metrics: Extend metrics logging for new operations or performance measurements.
• Documentation: Update this README and docstrings in code for any new features or changes.

Why you don't do compression with XLS/XLSX files

XLS and XLSX files are already compressed formats (especially XLSX, which is a ZIP archive of XML files). Applying additional compression (like gzip) typically results in minimal size reduction—often less than 3%. This extra step adds processing overhead without significant storage or transfer benefits. In most cases, it's more efficient to transfer these files as-is.

Compression Metrics for XLSX Files

File Name	Compressed Size (bytes)	Decompressed Size (bytes)	Compression Ratio	Bytes Saved
valid_form_7.xlsx.gz	37,009	37,973	2.5%	964
valid_form_8.xlsx.gz	38,975	40,021	2.6%	1,046
valid_form_9.xlsx.gz	37,341	38,420	2.8%	1,079
valid_form_10.xlsx.gz	37,269	38,208	2.5%	939

Observation: The compression ratios are very low (2.5%–2.8%), saving only about 939–1,079 bytes per file. This demonstrates that compressing XLSX files provides negligible space savings.

To put this in perspective, this required refactoring half the codebase with over 5,000 changed lines of code.

License

MIT