bulk-questionnaire-upload/README.md

Bulk Questionnaire Upload

A web application for uploading and parsing Excel-based questionnaires in XLSForm-like format with MongoDB integration.

Project Structure

│
├── frontend/          # Angular frontend application
├── backend/           # FastAPI backend with MongoDB
└── README.md

Stack

Frontend

• Angular 16+
• Material UI
• SCSS

Backend

• Python FastAPI
• MongoDB for data persistence
• openpyxl/pandas for Excel parsing
• motor/pymongo for MongoDB integration

Prerequisites

• Python 3.8+ (recommend using a virtual environment)
• Node.js 16+ and npm
• MongoDB (local or cloud instance)
• pip

Installation & Setup

1. Install MongoDB (Artix/Arch Linux)

Start MongoDB service:

sudo rc-service mongodb start      # OpenRC (Artix default)
# or
sudo systemctl start mongodb      # If using systemd
mongosh --eval "db.runCommand('ping')"
# Should output: { ok: 1 }

2. Backend Setup

1. Navigate to the backend directory:

   cd backend/
   

2. Create and activate virtual environment:

   python -m venv .venv
   source .venv/bin/activate
   

3. Install Python dependencies:

   pip install --upgrade pip
   pip install --break-system-packages -r requirements.txt
   

   If you see an "externally-managed-environment" error, use the --break-system-packages flag as above.

4. Create environment configuration: Create a .env file in the backend/ directory:

   MONGODB_URL=mongodb://localhost:27017
   DATABASE_NAME=mform_bulk_upload
   API_HOST=0.0.0.0
   API_PORT=8000
   

5. Start the FastAPI server:

   uvicorn main:app --reload
   

   The API will be available at http://localhost:8000

3. Frontend Setup

1. Navigate to the frontend directory:

   cd frontend/
   

2. Install Node.js dependencies:

   npm install
   

3. Start the development server:

   ng serve
   

   The frontend will be available at http://localhost:4200

API Endpoints

File Validation

• POST /api/validate Validate Excel file structure. Returns detailed validation information including sheet status, metadata, and counts.

File Upload

• POST /api/upload Parse and store Excel file in MongoDB. Saves form metadata, questions, and answer options to separate collections.

Forms Management

• GET /api/forms Get all forms from database.
• GET /api/forms/{form_id} Get specific form with questions and options.
• DELETE /api/forms/{form_id} Delete form and all related data.

Database Schema

Forms Collection

{
  "_id": "ObjectId",
  "title": "string",
  "language": "string",
  "version": "string",
  "created_at": "ISO timestamp"
}

Questions Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "title": "string",
  "view_sequence": "number",
  "input_type": "number",
  "created_at": "ISO timestamp"
}

Options Collection

{
  "_id": "ObjectId",
  "form_id": "string",
  "order": "number",
  "option_id": "number",
  "label": "string",
  "created_at": "ISO timestamp"
}

Sample Performance Metrics Output

Below is a real example of metrics collected for uploading 9 forms (each with ~400 questions and 3-10 options per question):

Metric	Time	Description
delete_all_forms_time	460ms	Time to delete all forms
deleted_forms	9	Number of forms deleted
deleted_questions	3570	Number of questions deleted
deleted_options	23154	Number of options deleted
validation_time_per_form	120-260ms	Time to validate each form file
form_process_time	240-1080ms	Time to process and save one form
questions_process_time	1.6-2.92s	Time to process and save all questions in a form
avg_one_question_process_time	4-7ms	Average time to process one question
options_process_time	9.23-11.8s	Time to process and save all options in a form
avg_one_option_process_time	3.6-4.5ms	Average time to process one option
total_form_upload_time	12.04-15.62s	Total time to process and upload a form
all_forms_batch_process_time	15.63s	Time to process all forms in the batch
total_forms	9	Number of forms processed in the batch
avg_one_form_process_time	1.74s	Average time to process one form in the batch