bulk-questionnaire-upload/backend

mForm Bulk Upload Backend

A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility.

---

Table of Contents

• Overview
• Architecture
• API Endpoints
• Database Schema
• Excel File Requirements
• Setup & Installation
• Error Handling & Logging
• Development & Maintenance Tips
• Extending the Backend

---

Overview

This backend provides:

• Validation of Excel files for correct structure and content
• Parsing and storage of forms, questions, and answer options in MongoDB
• RESTful API endpoints for form management, including in-place form updates
• Detailed error handling and logging
• CORS support for seamless frontend integration

---

Architecture

Main Components:

• main.py: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
• services/xlsform_parser.py: Business logic for validating and parsing Excel files
• services/database_service.py: Async database operations for forms, questions, and options
• database.py: MongoDB connection management and collection handles
• models/: Pydantic models for validation, API responses, and internal data structures

Startup Flow:

• Loads environment variables from .env
• Connects to MongoDB on startup, closes on shutdown
• Exposes API endpoints under /api/

---

API Endpoints

File Validation

• POST /api/validate
  • Description: Validate Excel file structure and content
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "valid": true,
      "message": "File format is valid.",
      "sheets": [...],
      "form_metadata": {...},
      "questions_count": 10,
      "options_count": 30,
      "errors": [],
      "warnings": []
    }
    

File Upload

• POST /api/upload
  • Description: Parse and store one or more Excel files
  • Request: multipart/form-data with one or more files
  • Response: List of parsed form objects or error details

Update Form

• PUT /api/forms/{form_id}/update
  • Description: Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
  • Request: multipart/form-data with a single file field
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

Forms Management

• GET /api/forms

  • Description: List all forms
  • Response: { "forms": [...], "count": 2 }

• GET /api/forms/{form_id}

  • Description: Get a form with its questions and options
  • Response:

    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    

• DELETE /api/forms/{form_id}

  • Description: Delete a form and all related data
  • Response: { "message": "Form deleted successfully" }

• DELETE /api/forms

  • Description: Delete all forms and related data
  • Response: { "message": "All forms deleted successfully" }

---

Database Schema

Forms Collection

{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}

Questions Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}

Options Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}

---

Excel File Requirements

The backend expects Excel files with three sheets:

1. Forms
   • Columns: Language, Title
2. Questions Info
   • Columns: Order, Title, View Sequence, Input Type
3. Answer Options
   • Columns: Order, Id, Label

Validation will fail if required sheets or columns are missing.

---

Setup & Installation

Prerequisites

• Python 3.8+
• MongoDB (local or cloud instance)
• pip

Installation Steps

1. Install dependencies:

   pip install -r requirements.txt
   

2. Set up MongoDB:
   • Install locally or use MongoDB Atlas
   • Create a database named mform_bulk_upload
3. Environment Configuration:
   • Create a .env file in the backend directory:

     MONGODB_URL=mongodb://localhost:27017
     DATABASE_NAME=mform_bulk_upload
     

4. Run the application:

   python main.py
   

   The API will be available at http://localhost:8000

---

Error Handling & Logging

• All API endpoints use structured error handling with FastAPI's HTTPException.
• Errors and warnings during file validation are returned in the API response.
• Application-level errors are logged using Python's logging module (see main.py, services/).
• Metrics (e.g., processing times, counts) are logged to metrics.txt for performance monitoring.
• Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.

---

Development & Maintenance Tips

• Centralize logic: All business logic is in services/, and all DB access in database_service.py.
• Environment variables: Use .env for DB config; never hardcode secrets.
• Testing: Use tools like httpie or Postman to test endpoints.
• Extending: Add new endpoints in main.py and corresponding logic in services/.
• Logging: Check metrics.txt and logs for troubleshooting and performance analysis.
• CORS: Update allowed origins in main.py if frontend URL changes.

---

Extending the Backend

• Add new endpoints: Define in main.py, implement logic in services/, and update models as needed.
• Add new collections: Update database.py and database_service.py for new MongoDB collections.
• Validation: Extend XLSFormParser for new validation rules or file formats.
• Documentation: Update this README and docstrings in code for any new features or changes.

---

License

MIT