bulk-questionnaire-upload/backend/README.md

# mForm Bulk Upload Backend

A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities.

---

## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [API Endpoints](#api-endpoints)
- [Database Schema](#database-schema)
- [Excel File Requirements](#excel-file-requirements)
- [Setup & Installation](#setup--installation)
- [Error Handling & Logging](#error-handling--logging)
- [Performance Metrics](#performance-metrics)
- [Development & Maintenance Tips](#development--maintenance-tips)
- [Extending the Backend](#extending-the-backend)

---

## Overview
This backend provides:
- Validation of Excel files for correct structure and content
- Parse-only functionality for schema preview without database storage
- Parsing and storage of forms, questions, and answer options in MongoDB
- RESTful API endpoints for form management, including in-place form updates
- Comprehensive error handling with detailed error types and suggestions
- Performance metrics logging for monitoring and optimization
- CORS support for seamless frontend integration

---

## Architecture

**Main Components:**
- `main.py`: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
- `services/xlsform_parser.py`: Business logic for validating and parsing Excel files
- `services/database_service.py`: Async database operations for forms, questions, and options
- `database.py`: MongoDB connection management and collection handles
- `models/`: Pydantic models for validation, API responses, and internal data structures

**Startup Flow:**
- Loads environment variables from `.env`
- Connects to MongoDB on startup, closes on shutdown
- Exposes API endpoints under `/api/`

---

## API Endpoints

### File Validation
- **POST** `/api/validate`
  - **Description:** Validate Excel file structure and content
  - **Request:** `multipart/form-data` with a single file field
  - **Response:**
    ```json
    {
      "valid": true,
      "message": "File format is valid.",
      "sheets": [...],
      "form_metadata": {...},
      "questions_count": 10,
      "options_count": 30,
      "errors": [],
      "warnings": []
    }
    ```

### File Parsing (Parse Only)
- **POST** `/api/forms/parse`
  - **Description:** Parse Excel file and return JSON schema without saving to database
  - **Request:** `multipart/form-data` with a single file field
  - **Response:**
    ```json
    {
      "id": null,
      "title": {"default": "Form Title"},
      "language": "en",
      "version": "1.0",
      "groups": [...],
      "settings": {...},
      "metadata": {
        "questions_count": 10,
        "options_count": 30,
        "parse_time": 0.082,
        "created_at": "2025-07-30T00:00:00Z",
        "sheets_found": ["Forms", "Questions Info", "Answer Options"],
        "file_name": "example.xlsx"
      },
      "raw_data": {...}
    }
    ```
  - **Error Response:**
    ```json
    {
      "detail": {
        "error": "Parsing failed",
        "message": "Missing required sheet: Forms",
        "error_type": "MISSING_SHEET",
        "suggestions": [
          "Ensure your Excel file contains a sheet named 'Forms'",
          "Check sheet names for typos or extra spaces"
        ]
      }
    }
    ```

### File Upload
- **POST** `/api/upload`
  - **Description:** Parse and store one or more Excel files
  - **Request:** `multipart/form-data` with one or more files
  - **Response:** List of parsed form objects or error details

### Update Form
- **PUT** `/api/forms/{form_id}/update`
  - **Description:** Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
  - **Request:** `multipart/form-data` with a single file field
  - **Response:**
    ```json
    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    ```

### Forms Management
- **GET** `/api/forms`
  - **Description:** List all forms
  - **Response:** `{ "forms": [...], "count": 2 }`

- **GET** `/api/forms/{form_id}`
  - **Description:** Get a form with its questions and options
  - **Response:**
    ```json
    {
      "form": {...},
      "questions": [...],
      "options": [...],
      "questions_count": 10,
      "options_count": 30
    }
    ```

- **DELETE** `/api/forms/{form_id}`
  - **Description:** Delete a form and all related data
  - **Response:** `{ "message": "Form deleted successfully" }`

- **DELETE** `/api/forms`
  - **Description:** Delete all forms and related data
  - **Response:** `{ "message": "All forms deleted successfully" }`

---

## Database Schema

### Forms Collection
```json
{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}
```

### Questions Collection
```json
{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}
```

### Options Collection
```json
{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}
```

---

## Excel File Requirements

The backend expects Excel files with **three sheets**:
1. **Forms**
   - Columns: `Language`, `Title`
2. **Questions Info**
   - Columns: `Order`, `Title`, `View Sequence`, `Input Type`
3. **Answer Options**
   - Columns: `Order`, `Id`, `Label`

Validation will fail if required sheets or columns are missing.

---

## Setup & Installation

### Prerequisites
- Python 3.8+
- MongoDB (local or cloud instance)
- pip

### Installation Steps
1. **Install dependencies:**
   ```bash
   pip install -r requirements.txt
   ```
2. **Set up MongoDB:**
   - Install locally or use MongoDB Atlas
   - Create a database named `mform_bulk_upload`
3. **Environment Configuration:**
   - Create a `.env` file in the backend directory:
     ```
     MONGODB_URL=mongodb://localhost:27017
     DATABASE_NAME=mform_bulk_upload
     API_HOST=0.0.0.0
     API_PORT=8000
     ```
4. **Run the application:**
   ```bash
   uvicorn main:app --reload
   ```
   The API will be available at `http://localhost:8000`

---

## Error Handling & Logging
- All API endpoints use structured error handling with FastAPI's HTTPException.
- Enhanced error responses include specific error types and actionable suggestions.
- Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more.
- Errors and warnings during file validation are returned in the API response.
- Application-level errors are logged using Python's logging module (see `main.py`, `services/`).
- Metrics (e.g., processing times, counts) are logged to `metrics.txt` for performance monitoring.
- Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.

---

## Performance Metrics

The application logs detailed performance metrics to `metrics.txt` including:
- **Validation times**: File validation performance per form
- **Parse-only times**: Schema parsing without database operations
- **Upload processing times**: Complete form processing with database storage
- **Question/Option processing**: Individual item processing performance
- **Cold startup tracking**: Application initialization times

## Development & Maintenance Tips
- **Centralize logic:** All business logic is in `services/`, and all DB access in `database_service.py`.
- **Environment variables:** Use `.env` for DB config; never hardcode secrets.
- **Testing:** Use tools like `httpie` or Postman to test endpoints.
- **Extending:** Add new endpoints in `main.py` and corresponding logic in `services/`.
- **Logging:** Check `metrics.txt` and logs for troubleshooting and performance analysis.
- **CORS:** Update allowed origins in `main.py` if frontend URL changes.
- **Error handling:** Use structured error responses with specific error types for better user experience.

## Extending the Backend
- **Add new endpoints:** Define in `main.py`, implement logic in `services/`, and update models as needed.
- **Add new collections:** Update `database.py` and `database_service.py` for new MongoDB collections.
- **Validation:** Extend `XLSFormParser` for new validation rules or file formats.
- **Error types:** Add new error classifications in the parsing service for specific failure cases.
- **Metrics:** Extend metrics logging for new operations or performance measurements.
- **Documentation:** Update this README and docstrings in code for any new features or changes.

# Why you don't do compression with XLS/XLSX files

XLS and XLSX files are already compressed formats (especially XLSX, which is a ZIP archive of XML files). Applying additional compression (like gzip) typically results in minimal size reduction—often less than 3%. This extra step adds processing overhead without significant storage or transfer benefits. In most cases, it's more efficient to transfer these files as-is.

# Compression Metrics for XLSX Files

| File Name              | Compressed Size (bytes) | Decompressed Size (bytes) | Compression Ratio | Bytes Saved |
|------------------------|------------------------|---------------------------|-------------------|-------------|
| valid_form_7.xlsx.gz   | 37,009                 | 37,973                    | 2.5%              | 964         |
| valid_form_8.xlsx.gz   | 38,975                 | 40,021                    | 2.6%              | 1,046       |
| valid_form_9.xlsx.gz   | 37,341                 | 38,420                    | 2.8%              | 1,079       |
| valid_form_10.xlsx.gz  | 37,269                 | 38,208                    | 2.5%              | 939         |

**Observation:**
The compression ratios are very low (2.5%–2.8%), saving only about 939–1,079 bytes per file. This demonstrates that compressing XLSX files provides negligible space savings.

To put this in perspective, this required refactoring half the codebase with over 5,000 changed lines of code.

## License
MIT