diff --git a/backend/README.md b/backend/README.md index 55c423e..5f15677 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,298 +1,530 @@ -# mForm Bulk Upload Backend +# mForm Bulk Questionnaire Upload Backend -A FastAPI backend for validating, parsing, and storing Excel-based form data in MongoDB. Designed for robust integration with the Angular frontend and easy extensibility. Features comprehensive error handling, performance metrics logging, and both parse-only and full upload capabilities. +A comprehensive FastAPI backend for bulk Excel-based questionnaire uploads, featuring advanced validation, parsing, and storage capabilities. Designed for robust integration with the Angular frontend and built for extensibility. ---- +## Architecture Overview -## Table of Contents -- [Overview](#overview) -- [Architecture](#architecture) -- [API Endpoints](#api-endpoints) -- [Database Schema](#database-schema) -- [Excel File Requirements](#excel-file-requirements) -- [Setup & Installation](#setup--installation) -- [Error Handling & Logging](#error-handling--logging) -- [Performance Metrics](#performance-metrics) -- [Development & Maintenance Tips](#development--maintenance-tips) -- [Extending the Backend](#extending-the-backend) +### Core Components ---- +#### **Main Application (`main.py`)** +- **FastAPI Application**: Central application instance with comprehensive middleware +- **CORS Configuration**: Configured for Angular frontend integration +- **API Routes**: RESTful endpoints for file processing and form management +- **Lifecycle Management**: Startup/shutdown events for MongoDB connection handling +- **Metrics Logging**: Performance tracking and cold start monitoring +- **Error Handling**: Structured error responses with detailed suggestions -## Overview -This backend provides: -- Validation of Excel files for correct structure and content -- Parse-only functionality for schema preview without database storage -- Parsing and storage of forms, questions, and answer options in MongoDB -- RESTful API endpoints for form management, including in-place form updates -- Comprehensive error handling with detailed error types and suggestions -- Performance metrics logging for monitoring and optimization -- CORS support for seamless frontend integration +#### **Business Logic (`services/xlsform_parser.py`)** +- **Excel File Validation**: Comprehensive structural and content validation +- **Data Parsing**: XLSForm-compliant parsing with multiple question types +- **Cross-Reference Validation**: Ensures data consistency between sheets +- **Error Classification**: Detailed error categorization with actionable suggestions +- **Performance Optimization**: Concurrent processing for bulk uploads ---- +#### **Database Layer (`services/database_service.py`)** +- **Async MongoDB Operations**: Full CRUD operations with error handling +- **Data Integrity**: Atomic operations and cascade deletions +- **Performance Tracking**: Database operation timing and metrics +- **Connection Management**: Proper resource handling and cleanup -## Architecture +#### **Data Models (`models/`)** +- **Pydantic Validation**: Request/response models with type safety +- **API Schemas**: Structured data contracts for frontend integration +- **Validation Models**: Comprehensive validation result structures -**Main Components:** -- `main.py`: FastAPI app, API routes, startup/shutdown events, CORS, and metrics logging -- `services/xlsform_parser.py`: Business logic for validating and parsing Excel files -- `services/database_service.py`: Async database operations for forms, questions, and options -- `database.py`: MongoDB connection management and collection handles -- `models/`: Pydantic models for validation, API responses, and internal data structures +## API Reference -**Startup Flow:** -- Loads environment variables from `.env` -- Connects to MongoDB on startup, closes on shutdown -- Exposes API endpoints under `/api/` +### File Processing Endpoints ---- +#### **POST `/api/validate`** +Validates Excel file structure and content without processing. -## API Endpoints +**Request**: `multipart/form-data` +```python +file: UploadFile # .xls or .xlsx file +``` -### File Validation -- **POST** `/api/validate` - - **Description:** Validate Excel file structure and content - - **Request:** `multipart/form-data` with a single file field - - **Response:** +**Response**: ```json { "valid": true, "message": "File format is valid.", - "sheets": [...], - "form_metadata": {...}, - "questions_count": 10, - "options_count": 30, + "sheets": [ + { + "name": "Forms", + "exists": true, + "columns": ["Language", "Title"], + "required_columns": ["Language", "Title"], + "missing_columns": [], + "row_count": 1 + } + ], + "form_metadata": { + "language": "en", + "title": "Sample Form" + }, + "questions_count": 5, + "options_count": 15, "errors": [], "warnings": [] } ``` -### File Parsing (Parse Only) -- **POST** `/api/forms/parse` - - **Description:** Parse Excel file and return JSON schema without saving to database - - **Request:** `multipart/form-data` with a single file field - - **Response:** +#### **POST `/api/forms/parse`** +Parses Excel file and returns structured JSON schema without database storage. + +**Request**: `multipart/form-data` +```python +file: UploadFile # .xls or .xlsx file +``` + +**Response**: ```json { "id": null, - "title": {"default": "Form Title"}, + "title": {"default": "Sample Questionnaire"}, + "version": "1.0.0", "language": "en", - "version": "1.0", - "groups": [...], - "settings": {...}, - "metadata": { - "questions_count": 10, - "options_count": 30, - "parse_time": 0.082, - "created_at": "2025-07-30T00:00:00Z", - "sheets_found": ["Forms", "Questions Info", "Answer Options"], - "file_name": "example.xlsx" - }, - "raw_data": {...} - } - ``` - - **Error Response:** - ```json + "groups": [ { - "detail": { - "error": "Parsing failed", - "message": "Missing required sheet: Forms", - "error_type": "MISSING_SHEET", - "suggestions": [ - "Ensure your Excel file contains a sheet named 'Forms'", - "Check sheet names for typos or extra spaces" - ] + "name": "default", + "label": {"default": "Default Group"}, + "questions": [ + { + "type": "text", + "name": "1", + "label": {"default": "What is your name?"}, + "required": false, + "choices": null + } + ] + } + ], + "metadata": { + "questions_count": 5, + "options_count": 15, + "parse_time": 0.082, + "sheets_found": ["Forms", "Questions Info", "Answer Options"], + "file_name": "questionnaire.xlsx", + "validation_warnings": [] } } ``` -### File Upload -- **POST** `/api/upload` - - **Description:** Parse and store one or more Excel files - - **Request:** `multipart/form-data` with one or more files - - **Response:** List of parsed form objects or error details +#### **POST `/api/upload`** +Processes and stores multiple Excel files concurrently. -### Update Form -- **PUT** `/api/forms/{form_id}/update` - - **Description:** Update an existing form with a new Excel file (XLS/XLSX). The form is updated in place, preserving its ID. - - **Request:** `multipart/form-data` with a single file field - - **Response:** +**Request**: `multipart/form-data` +```python +files: List[UploadFile] # Multiple .xls or .xlsx files +``` + +**Response**: Array of parsed form objects with database IDs and metadata. + +### Form Management Endpoints + +#### **GET `/api/forms`** +Retrieves all forms with summary information. + +**Response**: ```json +{ + "forms": [ { - "form": {...}, - "questions": [...], - "options": [...], - "questions_count": 10, - "options_count": 30 + "id": "507f1f77bcf86cd799439011", + "title": "Sample Form", + "language": "en", + "version": "1.0.0", + "created_at": "2024-01-15T10:30:00Z" } - ``` + ], + "count": 1 +} +``` -### Forms Management -- **GET** `/api/forms` - - **Description:** List all forms - - **Response:** `{ "forms": [...], "count": 2 }` +#### **GET `/api/forms/{form_id}`** +Retrieves complete form data including questions and options. -- **GET** `/api/forms/{form_id}` - - **Description:** Get a form with its questions and options - - **Response:** +**Response**: ```json +{ + "form": { + "id": "507f1f77bcf86cd799439011", + "title": "Sample Form", + "language": "en", + "version": "1.0.0", + "created_at": "2024-01-15T10:30:00Z" + }, + "questions": [ { - "form": {...}, - "questions": [...], - "options": [...], - "questions_count": 10, - "options_count": 30 + "id": "507f1f77bcf86cd799439012", + "form_id": "507f1f77bcf86cd799439011", + "order": 1, + "title": "What is your name?", + "view_sequence": 1, + "input_type": 1, + "created_at": "2024-01-15T10:30:00Z" } - ``` + ], + "options": [...], + "questions_count": 5, + "options_count": 15 +} +``` -- **DELETE** `/api/forms/{form_id}` - - **Description:** Delete a form and all related data - - **Response:** `{ "message": "Form deleted successfully" }` +#### **PUT `/api/forms/{form_id}/update`** +Updates an existing form with new Excel file data. -- **DELETE** `/api/forms` - - **Description:** Delete all forms and related data - - **Response:** `{ "message": "All forms deleted successfully" }` +**Request**: `multipart/form-data` +```python +file: UploadFile # Updated .xls or .xlsx file +``` ---- +#### **DELETE `/api/forms/{form_id}`** +Deletes a form and all related questions and options. + +#### **DELETE `/api/forms`** +Deletes all forms and related data (bulk operation). + +## Data Models & Validation + +### Core Data Structures + +#### **Supported Question Types** +```python +SUPPORTED_QUESTION_TYPES = { + 1: 'text', # Text input + 2: 'select_one', # Single choice + 3: 'select_multiple', # Multiple choice + 4: 'integer', # Whole numbers + 5: 'decimal', # Decimal numbers + 6: 'date', # Date picker + 7: 'time', # Time picker + 8: 'datetime', # Date and time + 9: 'note', # Display text + 10: 'calculate' # Computed value +} +``` + +#### **Validation Rules** + +**Forms Sheet Requirements**: +- Required columns: `Language`, `Title` +- Supported languages: en, fr, es, de, it, pt, ar, zh, ja, ko, hi, ru +- Title length: ≤ 255 characters +- Only first row is processed + +**Questions Info Sheet Requirements**: +- Required columns: `Order`, `Title`, `View Sequence`, `Input Type` +- Order: Positive integers, unique +- View Sequence: Positive integers +- Input Type: 1-10 (see supported types above) +- Title length: ≤ 1000 characters + +**Answer Options Sheet Requirements**: +- Required columns: `Order`, `Id`, `Label` +- Order: Positive integers +- Id: Positive integers, unique per Order +- Label length: ≤ 500 characters + +### Cross-Reference Validation +- Choice questions (types 2, 3) must have corresponding options +- All option orders must map to existing question orders +- No orphaned options without corresponding questions ## Database Schema -### Forms Collection -```json +### Collections Overview + +#### **forms** +```javascript { - "_id": "ObjectId", - "title": "string", - "language": "string", - "version": "string", - "created_at": "ISO timestamp" + _id: ObjectId, + title: String, + language: String, + version: String, + created_at: ISODate } ``` -### Questions Collection -```json +#### **questions** +```javascript { - "_id": "ObjectId", - "form_id": "string", - "order": "number", - "title": "string", - "view_sequence": "number", - "input_type": "number", - "created_at": "ISO timestamp" + _id: ObjectId, + form_id: String, + order: Number, + title: String, + view_sequence: Number, + input_type: Number, + created_at: ISODate } ``` -### Options Collection -```json +#### **options** +```javascript { - "_id": "ObjectId", - "form_id": "string", - "order": "number", - "option_id": "number", - "label": "string", - "created_at": "ISO timestamp" + _id: ObjectId, + form_id: String, + order: Number, + option_id: Number, + label: String, + created_at: ISODate } ``` ---- +### Indexes & Performance +- Forms: `{created_at: -1}` (recent forms first) +- Questions: `{form_id: 1}` (efficient form retrieval) +- Options: `{form_id: 1}` (efficient form retrieval) -## Excel File Requirements +## Configuration & Setup -The backend expects Excel files with **three sheets**: -1. **Forms** - - Columns: `Language`, `Title` -2. **Questions Info** - - Columns: `Order`, `Title`, `View Sequence`, `Input Type` -3. **Answer Options** - - Columns: `Order`, `Id`, `Label` - -Validation will fail if required sheets or columns are missing. - ---- - -## Setup & Installation - -### Prerequisites -- Python 3.8+ -- MongoDB (local or cloud instance) -- pip - -### Installation Steps -1. **Install dependencies:** +### Environment Variables ```bash - pip install -r requirements.txt - ``` -2. **Set up MongoDB:** - - Install locally or use MongoDB Atlas - - Create a database named `mform_bulk_upload` -3. **Environment Configuration:** - - Create a `.env` file in the backend directory: - ``` +# MongoDB Configuration MONGODB_URL=mongodb://localhost:27017 DATABASE_NAME=mform_bulk_upload + +# API Configuration API_HOST=0.0.0.0 API_PORT=8000 - ``` -4. **Run the application:** + +# CORS Configuration +FRONTEND_URL=http://localhost:4200 +``` + +### Installation +```bash +# Create virtual environment +python -m venv venv +source venv/bin/activate + +# Install dependencies +pip install -r requirements.txt + +# Create .env file +cp .env.example .env + +# Start the application +uvicorn main:app --reload --host 0.0.0.0 --port 8000 +``` + +## Error Handling & Validation + +### Error Classification System + +#### **File-Level Errors** +- `MISSING_FILE`: No file uploaded +- `INVALID_FILE_FORMAT`: Wrong file extension or format +- `EMPTY_FILE`: Zero-byte file +- `FILE_ACCESS_ERROR`: Cannot read file content +- `CORRUPTED_FILE`: Invalid Excel structure + +#### **Structure-Level Errors** +- `MISSING_SHEET`: Required sheet not found +- `MISSING_COLUMNS`: Required columns missing +- `EMPTY_SHEET`: Sheet contains no data + +#### **Content-Level Errors** +- `INVALID_DATA_TYPE`: Wrong data type in cells +- `MISSING_VALUE`: Required field is empty +- `INVALID_VALUE`: Value outside acceptable range +- `DUPLICATE_VALUE`: Non-unique value where uniqueness required + +#### **Cross-Reference Errors** +- `MISSING_REFERENCE`: Choice question without options +- `ORPHANED_REFERENCE`: Options without corresponding question + +### Enhanced Error Responses +```json +{ + "detail": { + "error": "Validation failed", + "message": "Found 3 error(s) and 2 warning(s).", + "error_type": "VALIDATION_ERROR", + "file_name": "questionnaire.xlsx", + "errors": [ + { + "type": "missing_column", + "message": "Required column 'Title' is missing", + "location": "Forms sheet", + "row": null, + "column": "Title" + } + ], + "warnings": [...], + "suggestions": [ + "Ensure your Excel file contains sheets named: 'Forms', 'Questions Info', 'Answer Options'", + "Check that all required columns are present in each sheet" + ] + } +} +``` + +## Performance & Monitoring + +### Metrics Tracked +- **Validation Performance**: File validation times +- **Parse Performance**: Schema parsing without DB operations +- **Upload Performance**: Complete form processing with storage +- **Database Operations**: Individual CRUD operation times +- **Cold Start Time**: Application initialization duration +- **Batch Processing**: Multi-file upload performance + +### Performance Optimizations +- **Concurrent Processing**: Async file processing with `asyncio.gather()` +- **Connection Pooling**: MongoDB connection reuse +- **Efficient Parsing**: Pandas DataFrame operations for large datasets +- **Memory Management**: File stream handling to prevent memory leaks + +## Development & Extension Guide + +### Adding New Features + +#### **1. New API Endpoint** +```python +# main.py +@app.post("/api/forms/export/{form_id}") +async def export_form(form_id: str): + # Implementation + pass +``` + +#### **2. New Business Logic** +```python +# services/xlsform_parser.py +class XLSFormParser: + def export_to_format(self, form_id: str, format: str) -> Dict[str, Any]: + # Implementation + pass +``` + +#### **3. New Database Operations** +```python +# services/database_service.py +class DatabaseService: + async def export_form_data(self, form_id: str) -> Dict[str, Any]: + # Implementation + pass +``` + +#### **4. New Data Models** +```python +# models/export.py +from pydantic import BaseModel + +class ExportRequest(BaseModel): + format: str # 'json', 'csv', 'xml' + include_metadata: bool = True +``` + +### Testing Strategy + +#### **Unit Tests** +```python +# Test individual components +def test_xlsform_parser_validation(): + parser = XLSFormParser() + # Test validation logic + +def test_database_service_operations(): + service = DatabaseService() + # Test database operations +``` + +#### **Integration Tests** +```python +# Test complete workflows +def test_file_upload_workflow(): + # Test end-to-end file processing + pass +``` + +### Code Quality Guidelines + +#### **Error Handling** +- Use structured error responses with specific error types +- Include actionable suggestions in error messages +- Log errors with appropriate context +- Never expose sensitive information in error responses + +#### **Performance** +- Use async/await for I/O operations +- Implement proper connection pooling +- Monitor and log performance metrics +- Optimize database queries with appropriate indexes + +#### **Security** +- Validate all input data +- Use parameterized queries +- Implement proper CORS configuration +- Never log sensitive information + +## Troubleshooting + +### Common Issues + +#### **MongoDB Connection Issues** +```python +# Check connection +from database import connect_to_mongo +await connect_to_mongo() +``` + +#### **File Processing Errors** +```python +# Enable debug logging +import logging +logging.basicConfig(level=logging.DEBUG) +``` + +#### **Performance Issues** +```python +# Check metrics +with open('metrics.txt', 'r') as f: + print(f.read()) +``` + +### Debug Commands ```bash - uvicorn main:app --reload - ``` - The API will be available at `http://localhost:8000` +# Check MongoDB collections +mongo mform_bulk_upload --eval "db.forms.count()" ---- +# Test API endpoints +curl -X POST "http://localhost:8000/api/validate" -F "file=@test.xlsx" -## Error Handling & Logging -- All API endpoints use structured error handling with FastAPI's HTTPException. -- Enhanced error responses include specific error types and actionable suggestions. -- Error types include: MISSING_FILE, INVALID_FILE_FORMAT, EMPTY_FILE, PARSING_ERROR, MISSING_SHEET, MISSING_COLUMNS, CORRUPTED_FILE, and more. -- Errors and warnings during file validation are returned in the API response. -- Application-level errors are logged using Python's logging module (see `main.py`, `services/`). -- Metrics (e.g., processing times, counts) are logged to `metrics.txt` for performance monitoring. -- Database errors are caught and logged; user-facing errors are returned with appropriate HTTP status codes. +# Monitor logs +tail -f logs/app.log +``` ---- +## Additional Resources -## Performance Metrics +### Excel File Format Standards +- [XLSForm Specification](https://xlsform.org/) +- [ODK XForm Standards](https://docs.getodk.org/xform/) +- [Excel File Format Documentation](https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-xlsx/) -The application logs detailed performance metrics to `metrics.txt` including: -- **Validation times**: File validation performance per form -- **Parse-only times**: Schema parsing without database operations -- **Upload processing times**: Complete form processing with database storage -- **Question/Option processing**: Individual item processing performance -- **Cold startup tracking**: Application initialization times +### API Documentation +- [FastAPI Documentation](https://fastapi.tiangolo.com/) +- [Pydantic Models](https://pydantic-docs.helpmanual.io/) +- [MongoDB Python Driver](https://pymongo.readthedocs.io/) -## Development & Maintenance Tips -- **Centralize logic:** All business logic is in `services/`, and all DB access in `database_service.py`. -- **Environment variables:** Use `.env` for DB config; never hardcode secrets. -- **Testing:** Use tools like `httpie` or Postman to test endpoints. -- **Extending:** Add new endpoints in `main.py` and corresponding logic in `services/`. -- **Logging:** Check `metrics.txt` and logs for troubleshooting and performance analysis. -- **CORS:** Update allowed origins in `main.py` if frontend URL changes. -- **Error handling:** Use structured error responses with specific error types for better user experience. +### Development Tools +- [uvicorn](https://www.uvicorn.org/) - ASGI server +- [pandas](https://pandas.pydata.org/) - Data processing +- [openpyxl](https://openpyxl.readthedocs.io/) - Excel file handling -## Extending the Backend -- **Add new endpoints:** Define in `main.py`, implement logic in `services/`, and update models as needed. -- **Add new collections:** Update `database.py` and `database_service.py` for new MongoDB collections. -- **Validation:** Extend `XLSFormParser` for new validation rules or file formats. -- **Error types:** Add new error classifications in the parsing service for specific failure cases. -- **Metrics:** Extend metrics logging for new operations or performance measurements. -- **Documentation:** Update this README and docstrings in code for any new features or changes. +## Contributing -# Why you don't do compression with XLS/XLSX files - -XLS and XLSX files are already compressed formats (especially XLSX, which is a ZIP archive of XML files). Applying additional compression (like gzip) typically results in minimal size reduction—often less than 3%. This extra step adds processing overhead without significant storage or transfer benefits. In most cases, it's more efficient to transfer these files as-is. - -# Compression Metrics for XLSX Files - -| File Name | Compressed Size (bytes) | Decompressed Size (bytes) | Compression Ratio | Bytes Saved | -|------------------------|------------------------|---------------------------|-------------------|-------------| -| valid_form_7.xlsx.gz | 37,009 | 37,973 | 2.5% | 964 | -| valid_form_8.xlsx.gz | 38,975 | 40,021 | 2.6% | 1,046 | -| valid_form_9.xlsx.gz | 37,341 | 38,420 | 2.8% | 1,079 | -| valid_form_10.xlsx.gz | 37,269 | 38,208 | 2.5% | 939 | - -**Observation:** -The compression ratios are very low (2.5%–2.8%), saving only about 939–1,079 bytes per file. This demonstrates that compressing XLSX files provides negligible space savings. - -To put this in perspective, this required refactoring half the codebase with over 5,000 changed lines of code. +1. Follow the established code structure +2. Add comprehensive error handling +3. Include performance metrics for new operations +4. Update documentation for any API changes +5. Add tests for new functionality +6. Use meaningful commit messages ## License -MIT + +MIT License - see LICENSE file for details. + +*For questions or support, please refer to the project documentation or create an issue in the repository.* diff --git a/frontend/README.md b/frontend/README.md index 1eddf82..103f84b 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,113 +1,387 @@ # mForm Bulk Upload Frontend -## About the Frontend +A modern, full-stack Angular application for bulk uploading, validating, parsing, and managing questionnaire forms. Built with Angular 19, featuring server-side rendering, comprehensive error handling, and a sleek dark theme interface. -This frontend is a modern Angular application for bulk uploading, validating, parsing, and managing questionnaire forms (in Excel format). It provides a user-friendly interface for users to drag-and-drop or browse files, validate them, parse them for preview, upload them to the backend, and view/manage parsed forms and their details. +## Quick Start -### Main Features -- **Bulk Upload**: Drag-and-drop or select multiple Excel files for upload. -- **Parse Only**: Parse Excel files to preview JSON schema without saving to database. -- **Validation**: Validate all selected files before uploading to ensure correct format and content. -- **Form Management**: View a list of all uploaded forms, search by title, and delete individual or all forms. -- **Form Details**: Click on a form to view its questions, options, and metadata in a modal dialog. -- **Schema Viewer**: Interactive modal for viewing parsed form schemas with structured display. -- **Update Form**: Update an existing form by selecting a new Excel file, preserving its ID and history. -- **Responsive UI**: Built with Angular Material for a clean, modern, and responsive user experience. -- **Progress Tracking**: Real-time progress bars for validation, parsing, uploading, and deletion operations. -- **Error Handling**: Comprehensive error messages with suggestions for fixing issues. +### Prerequisites +- Node.js 18+ +- Angular CLI 19+ +- Backend API server running -### Key Components -- **Navbar**: Displays the application title and navigation bar. -- **Search**: Allows searching forms by title. -- **Upload**: Handles file selection, validation, parsing, upload, and displays the list of forms and their details. +### Installation -### Project Structure -- `src/app/components/navbar/` - Navigation bar component -- `src/app/components/search/` - Search bar component -- `src/app/components/upload/` - Main upload and form management component -- `src/app/services/` - Services for form parsing, preview, and backend API communication -- `src/app/models/` - TypeScript interfaces for form and question data +```bash +# Navigate to frontend directory +cd frontend -# Backend Integration Points +# Install dependencies +npm install -This document outlines where and how the frontend communicates with the backend API for easy maintenance and future updates. +# Start development server +npm start -## Main Integration Service +# Build for production +npm run build -### FormService (`src/app/services/form.service.ts`) -This Angular service encapsulates all HTTP communication with the backend API. It is the single source of truth for API endpoints used by the frontend. +# Start SSR server +npm run serve:ssr:mform-upload +``` -**Endpoints Used:** -- `POST /api/validate` — Validate a form file before upload -- `POST /api/forms/parse` — Parse Excel file and return JSON schema without saving -- `POST /api/upload` — Upload a form file (single or multiple) -- `GET /api/forms` — Fetch all uploaded forms -- `GET /api/forms/:formId` — Fetch details for a specific form -- `PUT /api/forms/:formId/update` — Update a specific form with a new file -- `DELETE /api/forms/:formId` — Delete a specific form -- `DELETE /api/forms` — Delete all forms +### Development Commands -**Service Methods:** -- `validateFile(file: File)` -- `parseFile(file: File)` -- `uploadFile(file: File)` -- `uploadFiles(files: File[])` -- `getAllForms()` -- `getFormById(formId: string)` -- `updateForm(formId: string, file: File)` -- `deleteForm(formId: string)` -- `deleteAllForms()` +```bash +npm start # Development server on http://localhost:4200 +npm run build # Production build +npm run watch # Watch mode build +npm run test # Run tests with Vitest +npm run test:run # Run tests once +``` -## Components Using Backend Integration +## Architecture & Tech Stack -### UploadComponent (`src/app/components/upload/upload.component.ts`) -- Uses FormService to: - - Validate files before upload - - Parse files for schema preview - - Upload files - - Fetch the list of forms - - Update a form with a new file (update button, file picker, and update logic) - - Delete individual forms - - Delete all forms - - Fetch form details for preview +### Core Technologies +- **Angular 19** - Latest Angular with standalone components +- **TypeScript 5.7** - Full type safety and modern features +- **Angular Material** - UI component library with custom dark theme +- **Server-Side Rendering** - SEO-friendly and fast initial loads +- **Vitest** - Modern testing framework (replaces Jasmine/Karma) +- **Express.js** - SSR server backend -### Schema Viewer Modal -- Interactive modal component for viewing parsed schemas -- Displays form metadata, questions structure, and raw JSON -- Provides copy to clipboard and download functionality -- Keyboard shortcuts (ESC to close) +### Key Features -### NavbarComponent (`src/app/components/navbar/navbar.component.ts`) -- Indirectly uses backend data via FormPreviewService, which is populated by UploadComponent using FormService. +#### Bulk File Operations +- **Drag & Drop Interface** - Intuitive file selection with visual feedback +- **Multi-file Upload** - Process multiple Excel files simultaneously +- **Progress Tracking** - Real-time progress bars with color-coded operations +- **Batch Validation** - Validate all files before upload with detailed error reports -## How to Update Backend Endpoints -- All backend API URLs are defined in FormService as apiUrl. -- To change the backend base URL or endpoints, update FormService accordingly. -- If new endpoints are added to the backend, add corresponding methods to FormService and use them in components as needed. +#### Form Management +- **Excel Parsing** - Convert XLS/XLSX files to structured JSON schemas +- **Form Preview** - Interactive form details with question/option navigation +- **Search & Filter** - Real-time search across all uploaded forms +- **Form Updates** - In-place form updates preserving history and IDs -## Maintenance Tips -- Keep all backend API calls centralized in FormService for consistency and easy updates. -- Avoid direct HTTP calls in components; always use the service. -- Update this documentation whenever new integration points are added. +#### User Experience +- **Dark Theme** - Custom black/white Material Design theme +- **Keyboard Shortcuts** - Power user features for efficient navigation +- **Responsive Design** - Works seamlessly on desktop and mobile +- **Accessibility** - ARIA support and keyboard navigation -## Update Flow +#### Performance & Reliability +- **Server-Side Rendering** - Fast initial page loads and SEO benefits +- **Error Recovery** - Comprehensive error handling with user-friendly messages +- **Offline Support** - Graceful degradation and connection status +- **Memory Management** - Efficient file handling and cleanup -To update an existing form: -- Click the yellow update (refresh) icon next to a form in the list. -- Select a new Excel file (XLS/XLSX) from your computer. -- The file is sent to the backend, which updates the form in place (preserving its ID and history). -- The UI will reload the form list and show the updated details. +## Project Structure -This feature uses the backend endpoint `PUT /api/forms/:formId/update` and the `updateForm` method in FormService. +``` +frontend/ +├── src/ +│ ├── app/ +│ │ ├── components/ +│ │ │ ├── navbar/ # Navigation with form preview +│ │ │ ├── search/ # Search functionality +│ │ │ └── upload/ # Core upload/management logic +│ │ ├── models/ # TypeScript interfaces +│ │ ├── services/ # API and state management +│ │ ├── app.component.ts # Root component +│ │ ├── app.config.ts # Application configuration +│ │ ├── app.routes.ts # Routing (currently single-page) +│ │ └── main.ts # Bootstrap +│ ├── server.ts # SSR server configuration +│ ├── main.server.ts # SSR bootstrap +│ └── styles.scss # Global dark theme styles +├── public/ # Static assets +├── angular.json # Angular CLI configuration +├── vitest.config.ts # Testing configuration +└── package.json # Dependencies and scripts +``` -## Parse Only Flow +## Core Components -To preview a form schema without saving: -- Select Excel files using drag-and-drop or file browser. -- Click the "Parse Only" button to parse files without saving to database. -- View parsed results with structured information display. -- Use "View Schema" button to open detailed modal with interactive schema viewer. -- Download JSON schema or copy to clipboard from the modal. +### UploadComponent +**Location:** `src/app/components/upload/upload.component.ts` -This feature uses the backend endpoint `POST /api/forms/parse` and the `parseFile` method in FormService. +The main component handling all file operations and form management: +```typescript +// Key features implemented: +- Drag & drop file handling +- Multi-file validation and parsing +- Real-time progress tracking +- Form list management +- Schema preview modal +- Error handling and user feedback +``` + +**Key Methods:** +- `onDragOver/onDrop` - File drag & drop handling +- `validateFiles()` - Batch file validation +- `uploadFiles()` - Multi-file upload with progress +- `parseFilesOnly()` - Preview parsing without saving +- `showFormDetails()` - Form preview integration + +### NavbarComponent +**Location:** `src/app/components/navbar/navbar.component.ts` + +Navigation with integrated form preview system: + +```typescript +// Features: +- Form preview panel with slide animation +- Keyboard navigation (Ctrl+J/K, Shift+Ctrl+J/K) +- Question-by-question navigation +- Real-time form data display +``` + +### SearchComponent +**Location:** `src/app/components/search/search.component.ts` + +Intelligent search functionality: + +```typescript +// Capabilities: +- Real-time form title search +- Keyboard shortcut (Shift+K) focus +- Debounced search for performance +- Visual feedback and accessibility +``` + +## API Integration + +### FormService +**Location:** `src/app/services/form.service.ts` + +Centralized API communication layer: + +```typescript +@Injectable({ providedIn: 'root' }) +export class FormService { + private readonly apiUrl = 'http://localhost:8000/api'; + + // Core API methods + validateFile(file: File): Observable + parseFile(file: File): Observable + uploadFiles(files: File[]): Observable + getAllForms(): Observable + getFormById(formId: string): Observable + updateForm(formId: string, file: File): Observable + deleteForm(formId: string): Observable<{ message: string }> + deleteAllForms(): Observable<{ message: string }> +} +``` + +### Backend Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/validate` | Validate Excel file format and content | +| POST | `/api/forms/parse` | Parse Excel to JSON schema (no save) | +| POST | `/api/upload` | Upload and save form to database | +| GET | `/api/forms` | Retrieve all uploaded forms | +| GET | `/api/forms/:id` | Get specific form details | +| PUT | `/api/forms/:id/update` | Update existing form with new file | +| DELETE | `/api/forms/:id` | Delete specific form | +| DELETE | `/api/forms` | Delete all forms | + +## Styling & Theming + +### Dark Theme Implementation +**Location:** `src/styles.scss` + +Custom Material Design dark theme with: +- Black background with glassmorphism effects +- White text and borders for high contrast +- Custom color palette for buttons and states +- Responsive design with mobile-first approach + +### Component Styling Strategy +- **Inline styles** in component decorators for encapsulation +- **Global overrides** in `styles.scss` for consistency +- **CSS custom properties** for theme flexibility +- **SCSS nesting** for maintainable component styles + +## Testing Strategy + +### Vitest Configuration +**Location:** `vitest.config.ts` + +```typescript +export default defineConfig({ + test: { + environment: 'jsdom', + include: ['tests/frontend/**/*.spec.ts'], + globals: true + } +}) +``` + +### Test Structure +**Location:** `tests/frontend/` + +- **form.service.spec.ts** - API service testing +- Unit tests for components and services +- Integration tests for critical user flows +- E2E test coverage for upload workflows + +### Running Tests +```bash +npm run test # Watch mode +npm run test:run # Single run +``` + +## Deployment & Production + +### Build Configuration +**Location:** `angular.json` + +Key production settings: +- **SSR enabled** for better performance +- **Budget limits** for bundle size optimization +- **Asset optimization** and hashing +- **Source maps** for debugging + +### Production Build +```bash +npm run build +# Output: dist/mform-upload/ +``` + +### SSR Deployment +```bash +npm run serve:ssr:mform-upload +# Starts Express server on port 4000 +``` + +## Configuration & Environment + +### Angular Configuration +**Location:** `src/app/app.config.ts` + +```typescript +export const appConfig: ApplicationConfig = { + providers: [ + provideZoneChangeDetection({ eventCoalescing: true }), + provideRouter(routes), + provideClientHydration(withEventReplay()), + provideAnimations(), + provideHttpClient(withFetch()) + ] +}; +``` + +### Server-Side Rendering +**Location:** `src/server.ts` + +Express server with: +- Static file serving with caching +- Angular SSR integration +- Production-ready error handling + +## Keyboard Shortcuts + +| Shortcut | Action | Context | +|----------|--------|---------| +| `Shift + K` | Focus search | Global | +| `Ctrl + J` | Next form | Form list | +| `Ctrl + K` | Previous form | Form list | +| `Ctrl + Shift + J` | Next question | Form preview | +| `Ctrl + Shift + K` | Previous question | Form preview | +| `Esc` | Close modals/preview | Modal open | + +## Error Handling + +### Validation Error Types +- **File Structure Errors** - Missing sheets, columns, invalid formats +- **Data Validation Errors** - Type mismatches, missing values, duplicates +- **Network Errors** - Connection issues, server errors, timeouts +- **File Processing Errors** - Corrupted files, encoding issues + +### Error Recovery +- **User-friendly messages** with actionable suggestions +- **Automatic retry** for network failures +- **Graceful degradation** for non-critical features +- **Detailed error logs** for debugging + +## Future Enhancements + +### Potential Improvements +1. **File Type Support** - Add CSV, JSON import capabilities +2. **Real-time Collaboration** - Multi-user form editing +3. **Advanced Analytics** - Form usage statistics and insights +4. **Template System** - Pre-built form templates +5. **Export Options** - Additional export formats (PDF, XML) +6. **Offline Mode** - Full offline capability with sync +7. **Internationalization** - Multi-language support +8. **Performance Monitoring** - Application performance tracking + +### Scalability Considerations +- **Lazy Loading** - Implement route-based code splitting +- **Service Workers** - Add PWA capabilities +- **Caching Strategy** - Implement intelligent data caching +- **Bundle Optimization** - Code splitting and tree shaking +- **CDN Integration** - Static asset optimization + +## API Documentation + +### Form Validation Response +```typescript +interface FormValidation { + valid: boolean; + message: string; + errors?: ValidationError[]; + warnings?: ValidationWarning[]; + sheets?: SheetValidation[]; + form_metadata?: Record; +} +``` + +### Form Data Structure +```typescript +interface FormData { + id: string; + title: string; + language: string; + version: string; + created_at: string; +} +``` + +### Parsed Schema Structure +```typescript +interface ParsedSchema { + id: string | null; + title: { default: string }; + version: string; + language: string; + groups: any[]; + metadata: { + questions_count: number; + options_count: number; + parse_time: number; + }; +} +``` + +## Contributing + +### Development Guidelines +1. **Code Style** - Follow Angular style guide and TypeScript best practices +2. **Component Design** - Use standalone components with proper encapsulation +3. **State Management** - Centralize state in services, avoid component coupling +4. **Testing** - Write tests for new features and bug fixes +5. **Documentation** - Update README and add JSDoc comments + +### Code Quality +- **ESLint** integration with Angular CLI +- **Pre-commit hooks** for code quality checks +- **Type checking** with strict TypeScript configuration +- **Bundle analysis** for performance monitoring + +## License + +This project is licensed under the terms specified in the root LICENSE file. + +Note: This documentation is automatically generated and reflects the current state of the codebase. For the most up-to-date information, refer to the source code and tests.