bulk-questionnaire-upload/backend/README.md

mForm Bulk Upload Backend

A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities.

---

Table of Contents

• Overview
• Architecture
• API Endpoints
• Database Schema
• Excel File Requirements
• Setup & Installation
• Error Handling & Logging
• Performance Metrics
• Development & Maintenance Tips
• Extending the Backend

---

Overview

This backend provides:

• Validation of Excel files for correct structure and content
• Parse-only functionality for schema preview without database storage
• Parsing and storage of forms, questions, and answer options in MongoDB
• RESTful API endpoints for form management, including in-place form updates
• Comprehensive error handling with detailed error types and suggestions
• Performance metrics logging for monitoring and optimization
• CORS support for seamless frontend integration

---

Architecture

Main Components:

• main.py: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
• services/xlsform_parser.py: Business logic for validating and parsing Excel files
• services/database_service.py: Async database operations for forms, questions, and options
• database.py: MongoDB connection management and collection handles
• models/: Pydantic models for validation, API responses, and internal data structures

Startup Flow:

• Loads environment variables from .env
• Connects to MongoDB on startup, closes on shutdown
• Exposes API endpoints under /api/

---

API Endpoints

File Validation

• POST /api/validate
  • Description: Validate Excel file structure and content
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "valid": true,
      "message": "File format is valid.",
      "sheets": [...],
      "form_metadata": {...},
      "questions_count": 10,
      "options_count": 30,
      "errors": [],
      "warnings": []
    }
    

File Parsing (Parse Only)

• POST /api/forms/parse
  • Description: Parse Excel file and return JSON schema without saving to database
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "id": null,
      "title": {"default": "Form Title"},
      "language": "en",
      "version": "1.0",
      "groups": [...],
      "settings": {...},
      "metadata": {
        "questions_count": 10,
        "options_count": 30,
        "parse_time": 0.082,
        "created_at": "2025-07-30T00:00:00Z",
        "sheets_found": ["Forms", "Questions Info", "Answer Options"],
        "file_name": "example.xlsx"
      },
      "raw_data": {...}
    }
    

  • Error Response:

    {
      "detail": {
        "error": "Parsing failed",
        "message": "Missing required sheet: Forms",
        "error_type": "MISSING_SHEET",
        "suggestions": [
          "Ensure your Excel file contains a sheet named 'Forms'",
          "Check sheet names for typos or extra spaces"
        ]
      }
    }
    

File Upload

• POST /api/upload
  • Description: Parse and store one or more Excel files
  • Request: multipart/form-data with one or more files
  • Response: List of parsed form objects or error details

Update Form

• PUT /api/forms/{form_id}/update
  • Description: Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

Forms Management

• GET /api/forms

  • Description: List all forms
  • Response: { "forms": [...], "count": 2 }

• GET /api/forms/{form_id}

  • Description: Get a form with its questions and options
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

• DELETE /api/forms/{form_id}

  • Description: Delete a form and all related data
  • Response: { "message": "Form deleted successfully" }

• DELETE /api/forms

  • Description: Delete all forms and related data
  • Response: { "message": "All forms deleted successfully" }

---

Database Schema

Forms Collection

{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}

Questions Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}

Options Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}

---

Excel File Requirements

The backend expects Excel files with three sheets:

1. Forms
   • Columns: Language, Title
2. Questions Info
   • Columns: Order, Title, View Sequence, Input Type
3. Answer Options
   • Columns: Order, Id, Label

Validation will fail if required sheets or columns are missing.

---

Setup & Installation

Prerequisites

• Python 3.8+
• MongoDB (local or cloud instance)
• pip

Installation Steps

1. Install dependencies:

   pip install -r requirements.txt
   

2. Set up MongoDB:
   • Install locally or use MongoDB Atlas
   • Create a database named mform_bulk_upload
3. Environment Configuration:
   • Create a .env file in the backend directory:

     MONGODB_URL=mongodb://localhost:27017
     DATABASE_NAME=mform_bulk_upload
     API_HOST=0.0.0.0
     API_PORT=8000
     

4. Run the application:

   uvicorn main:app --reload
   

   The API will be available at http://localhost:8000

---

Error Handling & Logging

• All API endpoints use structured error handling with FastAPI's HTTPException.
• Enhanced error responses include specific error types and actionable suggestions.
• Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more.
• Errors and warnings during file validation are returned in the API response.
• Application-level errors are logged using Python's logging module (see main.py, services/).
• Metrics (e.g., processing times, counts) are logged to metrics.txt for performance monitoring.
• Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.

---

Performance Metrics

The application logs detailed performance metrics to metrics.txt including:

• Validation times: File validation performance per form
• Parse-only times: Schema parsing without database operations
• Upload processing times: Complete form processing with database storage
• Question/Option processing: Individual item processing performance
• Cold startup tracking: Application initialization times

Typical Performance (M3 Pro MacBook Pro):

• Validation: 1.76-106.22ms (4.65ms average)
• Parse-only: 3.80-156.92ms (84.21ms average)
• Full upload: 222.08-281.11ms (247.50ms average)
• Question processing: 0.14-0.19ms per question
• Option processing: 0.14-0.16ms per option

---

Development & Maintenance Tips

• Centralize logic: All business logic is in services/, and all DB access in database_service.py.
• Environment variables: Use .env for DB config; never hardcode secrets.
• Testing: Use tools like httpie or Postman to test endpoints.
• Extending: Add new endpoints in main.py and corresponding logic in services/.
• Logging: Check metrics.txt and logs for troubleshooting and performance analysis.
• CORS: Update allowed origins in main.py if frontend URL changes.
• Error handling: Use structured error responses with specific error types for better user experience.

---

Extending the Backend

• Add new endpoints: Define in main.py, implement logic in services/, and update models as needed.
• Add new collections: Update database.py and database_service.py for new MongoDB collections.
• Validation: Extend XLSFormParser for new validation rules or file formats.
• Error types: Add new error classifications in the parsing service for specific failure cases.
• Metrics: Extend metrics logging for new operations or performance measurements.
• Documentation: Update this README and docstrings in code for any new features or changes.

---

License

MIT