vee1e/bulk-questionnaire-upload
1
0
Fork 0
mirror of https://github.com/vee1e/bulk-questionnaire-upload.git synced 2026-09-01 09:50:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
bulk-questionnaire-upload/backend/README.md

8.7 KiB
Raw Blame History

mForm Bulk Upload Backend

A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities.

Table of Contents

Overview

This backend provides:

  • Validation of Excel files for correct structure and content
  • Parse-only functionality for schema preview without database storage
  • Parsing and storage of forms, questions, and answer options in MongoDB
  • RESTful API endpoints for form management, including in-place form updates
  • Comprehensive error handling with detailed error types and suggestions
  • Performance metrics logging for monitoring and optimization
  • CORS support for seamless frontend integration

Architecture

Main Components:

  • main.py: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
  • services/xlsform_parser.py: Business logic for validating and parsing Excel files
  • services/database_service.py: Async database operations for forms, questions, and options
  • database.py: MongoDB connection management and collection handles
  • models/: Pydantic models for validation, API responses, and internal data structures

Startup Flow:

  • Loads environment variables from .env
  • Connects to MongoDB on startup, closes on shutdown
  • Exposes API endpoints under /api/

API Endpoints

File Validation

  • POST /api/validate
    • Description: Validate Excel file structure and content
    • Request: multipart/form-data with a single file field
    • Response: 
      {
  "valid": true,
  "message": "File format is valid.",
  "sheets": [...],
  "form_metadata": {...},
  "questions_count": 10,
  "options_count": 30,
  "errors": [],
  "warnings": []
}

File Parsing (Parse Only)

  • POST /api/forms/parse
    • Description: Parse Excel file and return JSON schema without saving to database
    • Request: multipart/form-data with a single file field
    • Response: 
      {
  "id": null,
  "title": {"default": "Form Title"},
  "language": "en",
  "version": "1.0",
  "groups": [...],
  "settings": {...},
  "metadata": {
    "questions_count": 10,
    "options_count": 30,
    "parse_time": 0.082,
    "created_at": "2025-07-30T00:00:00Z",
    "sheets_found": ["Forms", "Questions Info", "Answer Options"],
    "file_name": "example.xlsx"
  },
  "raw_data": {...}
}
    • Error Response: 
      {
  "detail": {
    "error": "Parsing failed",
    "message": "Missing required sheet: Forms",
    "error_type": "MISSING_SHEET",
    "suggestions": [
      "Ensure your Excel file contains a sheet named 'Forms'",
      "Check sheet names for typos or extra spaces"
    ]
  }
}

File Upload

  • POST /api/upload
    • Description: Parse and store one or more Excel files
    • Request: multipart/form-data with one or more files
    • Response: List of parsed form objects or error details

Update Form

  • PUT /api/forms/{form_id}/update
    • Description: Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
    • Request: multipart/form-data with a single file field
    • Response: 
      {
  "form": {...},
  "questions": [...],
  "options": [...],
  "questions_count": 10,
  "options_count": 30
}

Forms Management

  • GET /api/forms

    • Description: List all forms
    • Response: { "forms": [...], "count": 2 }

  • GET /api/forms/{form_id}

    • Description: Get a form with its questions and options
    • Response: 
      {
  "form": {...},
  "questions": [...],
  "options": [...],
  "questions_count": 10,
  "options_count": 30
}

  • DELETE /api/forms/{form_id}

    • Description: Delete a form and all related data
    • Response: { "message": "Form deleted successfully" }

  • DELETE /api/forms

    • Description: Delete all forms and related data
    • Response: { "message": "All forms deleted successfully" }

Database Schema

Forms Collection

{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}

Questions Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}

Options Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}

Excel File Requirements

The backend expects Excel files with three sheets:

  1. Forms
    • Columns: Language, Title
  2. Questions Info
    • Columns: Order, Title, View Sequence, Input Type
  3. Answer Options
    • Columns: Order, Id, Label

Validation will fail if required sheets or columns are missing.

Setup & Installation

Prerequisites

  • Python 3.8+
  • MongoDB (local or cloud instance)
  • pip

Installation Steps

  1. Install dependencies: 
    pip install -r requirements.txt
  2. Set up MongoDB:
    • Install locally or use MongoDB Atlas
    • Create a database named mform_bulk_upload
  3. Environment Configuration:
    • Create a .env file in the backend directory: 
      MONGODB_URL=mongodb://localhost:27017
DATABASE_NAME=mform_bulk_upload
API_HOST=0.0.0.0
API_PORT=8000
  4. Run the application: 
    uvicorn main:app --reload
    The API will be available at http://localhost:8000

Error Handling & Logging

  • All API endpoints use structured error handling with FastAPI's HTTPException.
  • Enhanced error responses include specific error types and actionable suggestions.
  • Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more.
  • Errors and warnings during file validation are returned in the API response.
  • Application-level errors are logged using Python's logging module (see main.py, services/).
  • Metrics (e.g., processing times, counts) are logged to metrics.txt for performance monitoring.
  • Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.

Performance Metrics

The application logs detailed performance metrics to metrics.txt including:

  • Validation times: File validation performance per form
  • Parse-only times: Schema parsing without database operations
  • Upload processing times: Complete form processing with database storage
  • Question/Option processing: Individual item processing performance
  • Cold startup tracking: Application initialization times

Development & Maintenance Tips

  • Centralize logic: All business logic is in services/, and all DB access in database_service.py.
  • Environment variables: Use .env for DB config; never hardcode secrets.
  • Testing: Use tools like httpie or Postman to test endpoints.
  • Extending: Add new endpoints in main.py and corresponding logic in services/.
  • Logging: Check metrics.txt and logs for troubleshooting and performance analysis.
  • CORS: Update allowed origins in main.py if frontend URL changes.
  • Error handling: Use structured error responses with specific error types for better user experience.

Extending the Backend

  • Add new endpoints: Define in main.py, implement logic in services/, and update models as needed.
  • Add new collections: Update database.py and database_service.py for new MongoDB collections.
  • Validation: Extend XLSFormParser for new validation rules or file formats.
  • Error types: Add new error classifications in the parsing service for specific failure cases.
  • Metrics: Extend metrics logging for new operations or performance measurements.
  • Documentation: Update this README and docstrings in code for any new features or changes.

License

MIT