vee1e/bulk-questionnaire-upload
1
0
Fork 0
mirror of https://github.com/vee1e/bulk-questionnaire-upload.git synced 2026-09-01 09:50:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

refactor: update README files for backend and frontend with enhanced documentation and structure

Browse source 
- Renamed backend README to reflect "Bulk Questionnaire Upload" focus.
- Expanded backend documentation to include detailed architecture, API references, and error handling strategies.
- Updated frontend README to include new features, installation instructions, and API integration details.
- Improved project structure descriptions and added sections for testing, deployment, and future enhancements.
This commit is contained in:
vee1e 2025-08-26 00:56:05 +05:30
parent 63307cab65
commit 69047a1bdd
2 changed files with 817 additions and 311 deletions
Download patch file Download diff file Expand all files Collapse all files

678
backend/README.md
View file

@ -1,298 +1,530 @@
# mForm Bulk Upload Backend
# mForm Bulk Questionnaire Upload Backend
A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities.
A comprehensive FastAPI backend for bulk Excel-based questionnaire uploads, featuring advanced validation, parsing, and storage capabilities. Designed for robust integration with the Angular frontend and built for extensibility.
---
## Architecture Overview
## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [API Endpoints](#api-endpoints)
- [Database Schema](#database-schema)
- [Excel File Requirements](#excel-file-requirements)
- [Setup & Installation](#setup--installation)
- [Error Handling & Logging](#error-handling--logging)
- [Performance Metrics](#performance-metrics)
- [Development & Maintenance Tips](#development--maintenance-tips)
- [Extending the Backend](#extending-the-backend)
### Core Components
---
#### **Main Application (`main.py`)**
- **FastAPI Application**: Central application instance with comprehensive middleware
- **CORS Configuration**: Configured for Angular frontend integration
- **API Routes**: RESTful endpoints for file processing and form management
- **Lifecycle Management**: Startup/shutdown events for MongoDB connection handling
- **Metrics Logging**: Performance tracking and cold start monitoring
- **Error Handling**: Structured error responses with detailed suggestions
## Overview
This backend provides:
- Validation of Excel files for correct structure and content
- Parse-only functionality for schema preview without database storage
- Parsing and storage of forms, questions, and answer options in MongoDB
- RESTful API endpoints for form management, including in-place form updates
- Comprehensive error handling with detailed error types and suggestions
- Performance metrics logging for monitoring and optimization
- CORS support for seamless frontend integration
#### **Business Logic (`services/xlsform_parser.py`)**
- **Excel File Validation**: Comprehensive structural and content validation
- **Data Parsing**: XLSForm-compliant parsing with multiple question types
- **Cross-Reference Validation**: Ensures data consistency between sheets
- **Error Classification**: Detailed error categorization with actionable suggestions
- **Performance Optimization**: Concurrent processing for bulk uploads
---
#### **Database Layer (`services/database_service.py`)**
- **Async MongoDB Operations**: Full CRUD operations with error handling
- **Data Integrity**: Atomic operations and cascade deletions
- **Performance Tracking**: Database operation timing and metrics
- **Connection Management**: Proper resource handling and cleanup
## Architecture
#### **Data Models (`models/`)**
- **Pydantic Validation**: Request/response models with type safety
- **API Schemas**: Structured data contracts for frontend integration
- **Validation Models**: Comprehensive validation result structures
**Main Components:**
- `main.py`: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging
- `services/xlsform_parser.py`: Business logic for validating and parsing Excel files
- `services/database_service.py`: Async database operations for forms, questions, and options
- `database.py`: MongoDB connection management and collection handles
- `models/`: Pydantic models for validation, API responses, and internal data structures
## API Reference
**Startup Flow:**
- Loads environment variables from `.env`
- Connects to MongoDB on startup, closes on shutdown
- Exposes API endpoints under `/api/`
### File Processing Endpoints
---
#### **POST `/api/validate`**
Validates Excel file structure and content without processing.
## API Endpoints
**Request**: `multipart/form-data`
```python
file: UploadFile # .xls or .xlsx file
```
### File Validation
- **POST** `/api/validate`
- **Description:** Validate Excel file structure and content
- **Request:** `multipart/form-data` with a single file field
- **Response:**
**Response**:
```json
{
"valid": true,
"message": "File format is valid.",
"sheets": [...],
"form_metadata": {...},
"questions_count": 10,
"options_count": 30,
"sheets": [
{
"name": "Forms",
"exists": true,
"columns": ["Language", "Title"],
"required_columns": ["Language", "Title"],
"missing_columns": [],
"row_count": 1
}
],
"form_metadata": {
"language": "en",
"title": "Sample Form"
},
"questions_count": 5,
"options_count": 15,
"errors": [],
"warnings": []
}
```
### File Parsing (Parse Only)
- **POST** `/api/forms/parse`
- **Description:** Parse Excel file and return JSON schema without saving to database
- **Request:** `multipart/form-data` with a single file field
- **Response:**
#### **POST `/api/forms/parse`**
Parses Excel file and returns structured JSON schema without database storage.
**Request**: `multipart/form-data`
```python
file: UploadFile # .xls or .xlsx file
```
**Response**:
```json
{
"id": null,
"title": {"default": "Form Title"},
"title": {"default": "Sample Questionnaire"},
"version": "1.0.0",
"language": "en",
"version": "1.0",
"groups": [...],
"settings": {...},
"metadata": {
"questions_count": 10,
"options_count": 30,
"parse_time": 0.082,
"created_at": "2025-07-30T00:00:00Z",
"sheets_found": ["Forms", "Questions Info", "Answer Options"],
"file_name": "example.xlsx"
},
"raw_data": {...}
}
```
- **Error Response:**
```json
"groups": [
{
"detail": {
"error": "Parsing failed",
"message": "Missing required sheet: Forms",
"error_type": "MISSING_SHEET",
"suggestions": [
"Ensure your Excel file contains a sheet named 'Forms'",
"Check sheet names for typos or extra spaces"
]
"name": "default",
"label": {"default": "Default Group"},
"questions": [
{
"type": "text",
"name": "1",
"label": {"default": "What is your name?"},
"required": false,
"choices": null
}
]
}
],
"metadata": {
"questions_count": 5,
"options_count": 15,
"parse_time": 0.082,
"sheets_found": ["Forms", "Questions Info", "Answer Options"],
"file_name": "questionnaire.xlsx",
"validation_warnings": []
}
}
```
### File Upload
- **POST** `/api/upload`
- **Description:** Parse and store one or more Excel files
- **Request:** `multipart/form-data` with one or more files
- **Response:** List of parsed form objects or error details
#### **POST `/api/upload`**
Processes and stores multiple Excel files concurrently.
### Update Form
- **PUT** `/api/forms/{form_id}/update`
- **Description:** Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID.
- **Request:** `multipart/form-data` with a single file field
- **Response:**
**Request**: `multipart/form-data`
```python
files: List[UploadFile] # Multiple .xls or .xlsx files
```
**Response**: Array of parsed form objects with database IDs and metadata.
### Form Management Endpoints
#### **GET `/api/forms`**
Retrieves all forms with summary information.
**Response**:
```json
{
"forms": [
{
"form": {...},
"questions": [...],
"options": [...],
"questions_count": 10,
"options_count": 30
"id": "507f1f77bcf86cd799439011",
"title": "Sample Form",
"language": "en",
"version": "1.0.0",
"created_at": "2024-01-15T10:30:00Z"
}
```
],
"count": 1
}
```
### Forms Management
- **GET** `/api/forms`
- **Description:** List all forms
- **Response:** `{ "forms": [...], "count": 2 }`
#### **GET `/api/forms/{form_id}`**
Retrieves complete form data including questions and options.
- **GET** `/api/forms/{form_id}`
- **Description:** Get a form with its questions and options
- **Response:**
**Response**:
```json
{
"form": {
"id": "507f1f77bcf86cd799439011",
"title": "Sample Form",
"language": "en",
"version": "1.0.0",
"created_at": "2024-01-15T10:30:00Z"
},
"questions": [
{
"form": {...},
"questions": [...],
"options": [...],
"questions_count": 10,
"options_count": 30
"id": "507f1f77bcf86cd799439012",
"form_id": "507f1f77bcf86cd799439011",
"order": 1,
"title": "What is your name?",
"view_sequence": 1,
"input_type": 1,
"created_at": "2024-01-15T10:30:00Z"
}
```
],
"options": [...],
"questions_count": 5,
"options_count": 15
}
```
- **DELETE** `/api/forms/{form_id}`
- **Description:** Delete a form and all related data
- **Response:** `{ "message": "Form deleted successfully" }`
#### **PUT `/api/forms/{form_id}/update`**
Updates an existing form with new Excel file data.
- **DELETE** `/api/forms`
- **Description:** Delete all forms and related data
- **Response:** `{ "message": "All forms deleted successfully" }`
**Request**: `multipart/form-data`
```python
file: UploadFile # Updated .xls or .xlsx file
```
---
#### **DELETE `/api/forms/{form_id}`**
Deletes a form and all related questions and options.
#### **DELETE `/api/forms`**
Deletes all forms and related data (bulk operation).
## Data Models & Validation
### Core Data Structures
#### **Supported Question Types**
```python
SUPPORTED_QUESTION_TYPES = {
1: 'text', # Text input
2: 'select_one', # Single choice
3: 'select_multiple', # Multiple choice
4: 'integer', # Whole numbers
5: 'decimal', # Decimal numbers
6: 'date', # Date picker
7: 'time', # Time picker
8: 'datetime', # Date and time
9: 'note', # Display text
10: 'calculate' # Computed value
}
```
#### **Validation Rules**
**Forms Sheet Requirements**:
- Required columns: `Language`, `Title`
- Supported languages: en, fr, es, de, it, pt, ar, zh, ja, ko, hi, ru
- Title length: ≤ 255 characters
- Only first row is processed
**Questions Info Sheet Requirements**:
- Required columns: `Order`, `Title`, `View Sequence`, `Input Type`
- Order: Positive integers, unique
- View Sequence: Positive integers
- Input Type: 1-10 (see supported types above)
- Title length: ≤ 1000 characters
**Answer Options Sheet Requirements**:
- Required columns: `Order`, `Id`, `Label`
- Order: Positive integers
- Id: Positive integers, unique per Order
- Label length: ≤ 500 characters
### Cross-Reference Validation
- Choice questions (types 2, 3) must have corresponding options
- All option orders must map to existing question orders
- No orphaned options without corresponding questions
## Database Schema
### Forms Collection
```json
### Collections Overview
#### **forms**
```javascript
{
"_id": "ObjectId",
"title": "string",
"language": "string",
"version": "string",
"created_at": "ISO timestamp"
_id: ObjectId,
title: String,
language: String,
version: String,
created_at: ISODate
}
```
### Questions Collection
```json
#### **questions**
```javascript
{
"_id": "ObjectId",
"form_id": "string",
"order": "number",
"title": "string",
"view_sequence": "number",
"input_type": "number",
"created_at": "ISO timestamp"
_id: ObjectId,
form_id: String,
order: Number,
title: String,
view_sequence: Number,
input_type: Number,
created_at: ISODate
}
```
### Options Collection
```json
#### **options**
```javascript
{
"_id": "ObjectId",
"form_id": "string",
"order": "number",
"option_id": "number",
"label": "string",
"created_at": "ISO timestamp"
_id: ObjectId,
form_id: String,
order: Number,
option_id: Number,
label: String,
created_at: ISODate
}
```
---
### Indexes & Performance
- Forms: `{created_at: -1}` (recent forms first)
- Questions: `{form_id: 1}` (efficient form retrieval)
- Options: `{form_id: 1}` (efficient form retrieval)
## Excel File Requirements
## Configuration & Setup
The backend expects Excel files with **three sheets**:
1. **Forms**
- Columns: `Language`, `Title`
2. **Questions Info**
- Columns: `Order`, `Title`, `View Sequence`, `Input Type`
3. **Answer Options**
- Columns: `Order`, `Id`, `Label`
Validation will fail if required sheets or columns are missing.
---
## Setup & Installation
### Prerequisites
- Python 3.8+
- MongoDB (local or cloud instance)
- pip
### Installation Steps
1. **Install dependencies:**
### Environment Variables
```bash
pip install -r requirements.txt
```
2. **Set up MongoDB:**
- Install locally or use MongoDB Atlas
- Create a database named `mform_bulk_upload`
3. **Environment Configuration:**
- Create a `.env` file in the backend directory:
```
# MongoDB Configuration
MONGODB_URL=mongodb://localhost:27017
DATABASE_NAME=mform_bulk_upload
# API Configuration
API_HOST=0.0.0.0
API_PORT=8000
```
4. **Run the application:**
# CORS Configuration
FRONTEND_URL=http://localhost:4200
```
### Installation
```bash
# Create virtual environment
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Create .env file
cp .env.example .env
# Start the application
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
## Error Handling & Validation
### Error Classification System
#### **File-Level Errors**
- `MISSING_FILE`: No file uploaded
- `INVALID_FILE_FORMAT`: Wrong file extension or format
- `EMPTY_FILE`: Zero-byte file
- `FILE_ACCESS_ERROR`: Cannot read file content
- `CORRUPTED_FILE`: Invalid Excel structure
#### **Structure-Level Errors**
- `MISSING_SHEET`: Required sheet not found
- `MISSING_COLUMNS`: Required columns missing
- `EMPTY_SHEET`: Sheet contains no data
#### **Content-Level Errors**
- `INVALID_DATA_TYPE`: Wrong data type in cells
- `MISSING_VALUE`: Required field is empty
- `INVALID_VALUE`: Value outside acceptable range
- `DUPLICATE_VALUE`: Non-unique value where uniqueness required
#### **Cross-Reference Errors**
- `MISSING_REFERENCE`: Choice question without options
- `ORPHANED_REFERENCE`: Options without corresponding question
### Enhanced Error Responses
```json
{
"detail": {
"error": "Validation failed",
"message": "Found 3 error(s) and 2 warning(s).",
"error_type": "VALIDATION_ERROR",
"file_name": "questionnaire.xlsx",
"errors": [
{
"type": "missing_column",
"message": "Required column 'Title' is missing",
"location": "Forms sheet",
"row": null,
"column": "Title"
}
],
"warnings": [...],
"suggestions": [
"Ensure your Excel file contains sheets named: 'Forms', 'Questions Info', 'Answer Options'",
"Check that all required columns are present in each sheet"
]
}
}
```
## Performance & Monitoring
### Metrics Tracked
- **Validation Performance**: File validation times
- **Parse Performance**: Schema parsing without DB operations
- **Upload Performance**: Complete form processing with storage
- **Database Operations**: Individual CRUD operation times
- **Cold Start Time**: Application initialization duration
- **Batch Processing**: Multi-file upload performance
### Performance Optimizations
- **Concurrent Processing**: Async file processing with `asyncio.gather()`
- **Connection Pooling**: MongoDB connection reuse
- **Efficient Parsing**: Pandas DataFrame operations for large datasets
- **Memory Management**: File stream handling to prevent memory leaks
## Development & Extension Guide
### Adding New Features
#### **1. New API Endpoint**
```python
# main.py
@app.post("/api/forms/export/{form_id}")
async def export_form(form_id: str):
# Implementation
pass
```
#### **2. New Business Logic**
```python
# services/xlsform_parser.py
class XLSFormParser:
def export_to_format(self, form_id: str, format: str) -> Dict[str, Any]:
# Implementation
pass
```
#### **3. New Database Operations**
```python
# services/database_service.py
class DatabaseService:
async def export_form_data(self, form_id: str) -> Dict[str, Any]:
# Implementation
pass
```
#### **4. New Data Models**
```python
# models/export.py
from pydantic import BaseModel
class ExportRequest(BaseModel):
format: str # 'json', 'csv', 'xml'
include_metadata: bool = True
```
### Testing Strategy
#### **Unit Tests**
```python
# Test individual components
def test_xlsform_parser_validation():
parser = XLSFormParser()
# Test validation logic
def test_database_service_operations():
service = DatabaseService()
# Test database operations
```
#### **Integration Tests**
```python
# Test complete workflows
def test_file_upload_workflow():
# Test end-to-end file processing
pass
```
### Code Quality Guidelines
#### **Error Handling**
- Use structured error responses with specific error types
- Include actionable suggestions in error messages
- Log errors with appropriate context
- Never expose sensitive information in error responses
#### **Performance**
- Use async/await for I/O operations
- Implement proper connection pooling
- Monitor and log performance metrics
- Optimize database queries with appropriate indexes
#### **Security**
- Validate all input data
- Use parameterized queries
- Implement proper CORS configuration
- Never log sensitive information
## Troubleshooting
### Common Issues
#### **MongoDB Connection Issues**
```python
# Check connection
from database import connect_to_mongo
await connect_to_mongo()
```
#### **File Processing Errors**
```python
# Enable debug logging
import logging
logging.basicConfig(level=logging.DEBUG)
```
#### **Performance Issues**
```python
# Check metrics
with open('metrics.txt', 'r') as f:
print(f.read())
```
### Debug Commands
```bash
uvicorn main:app --reload
```
The API will be available at `http://localhost:8000`
# Check MongoDB collections
mongo mform_bulk_upload --eval "db.forms.count()"
---
# Test API endpoints
curl -X POST "http://localhost:8000/api/validate" -F "file=@test.xlsx"
## Error Handling & Logging
- All API endpoints use structured error handling with FastAPI's HTTPException.
- Enhanced error responses include specific error types and actionable suggestions.
- Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more.
- Errors and warnings during file validation are returned in the API response.
- Application-level errors are logged using Python's logging module (see `main.py`, `services/`).
- Metrics (e.g., processing times, counts) are logged to `metrics.txt` for performance monitoring.
- Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes.
# Monitor logs
tail -f logs/app.log
```
---
## Additional Resources
## Performance Metrics
### Excel File Format Standards
- [XLSForm Specification](https://xlsform.org/)
- [ODK XForm Standards](https://docs.getodk.org/xform/)
- [Excel File Format Documentation](https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-xlsx/)
The application logs detailed performance metrics to `metrics.txt` including:
- **Validation times**: File validation performance per form
- **Parse-only times**: Schema parsing without database operations
- **Upload processing times**: Complete form processing with database storage
- **Question/Option processing**: Individual item processing performance
- **Cold startup tracking**: Application initialization times
### API Documentation
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [Pydantic Models](https://pydantic-docs.helpmanual.io/)
- [MongoDB Python Driver](https://pymongo.readthedocs.io/)
## Development & Maintenance Tips
- **Centralize logic:** All business logic is in `services/`, and all DB access in `database_service.py`.
- **Environment variables:** Use `.env` for DB config; never hardcode secrets.
- **Testing:** Use tools like `httpie` or Postman to test endpoints.
- **Extending:** Add new endpoints in `main.py` and corresponding logic in `services/`.
- **Logging:** Check `metrics.txt` and logs for troubleshooting and performance analysis.
- **CORS:** Update allowed origins in `main.py` if frontend URL changes.
- **Error handling:** Use structured error responses with specific error types for better user experience.
### Development Tools
- [uvicorn](https://www.uvicorn.org/) - ASGI server
- [pandas](https://pandas.pydata.org/) - Data processing
- [openpyxl](https://openpyxl.readthedocs.io/) - Excel file handling
## Extending the Backend
- **Add new endpoints:** Define in `main.py`, implement logic in `services/`, and update models as needed.
- **Add new collections:** Update `database.py` and `database_service.py` for new MongoDB collections.
- **Validation:** Extend `XLSFormParser` for new validation rules or file formats.
- **Error types:** Add new error classifications in the parsing service for specific failure cases.
- **Metrics:** Extend metrics logging for new operations or performance measurements.
- **Documentation:** Update this README and docstrings in code for any new features or changes.
## Contributing
# Why you don't do compression with XLS/XLSX files
XLS and XLSX files are already compressed formats (especially XLSX, which is a ZIP archive of XML files). Applying additional compression (like gzip) typically results in minimal size reduction—often less than 3%. This extra step adds processing overhead without significant storage or transfer benefits. In most cases, it's more efficient to transfer these files as-is.
# Compression Metrics for XLSX Files
| File Name | Compressed Size (bytes) | Decompressed Size (bytes) | Compression Ratio | Bytes Saved |
|------------------------|------------------------|---------------------------|-------------------|-------------|
| valid_form_7.xlsx.gz | 37,009 | 37,973 | 2.5% | 964 |
| valid_form_8.xlsx.gz | 38,975 | 40,021 | 2.6% | 1,046 |
| valid_form_9.xlsx.gz | 37,341 | 38,420 | 2.8% | 1,079 |
| valid_form_10.xlsx.gz | 37,269 | 38,208 | 2.5% | 939 |
**Observation:**
The compression ratios are very low (2.5%–2.8%), saving only about 939–1,079 bytes per file. This demonstrates that compressing XLSX files provides negligible space savings.
To put this in perspective, this required refactoring half the codebase with over 5,000 changed lines of code.
1. Follow the established code structure
2. Add comprehensive error handling
3. Include performance metrics for new operations
4. Update documentation for any API changes
5. Add tests for new functionality
6. Use meaningful commit messages
## License
MIT
MIT License - see LICENSE file for details.
*For questions or support, please refer to the project documentation or create an issue in the repository.*